Job Summary:
We are seeking a Senior Full Stack Engineer to transform a Data Insight proof of concept into a production-ready application. The product enables users to explore healthcare reporting data, ask follow-up questions, and understand findings supported by source evidence. The role involves building agentic workflows and developing the user experience around them, with particular ownership of evaluation and quality pipelines to ensure application reliability as it evolves. This is a senior-level contract position focused on full stack and cloud-native development.
Key Responsibilities:
• Build and operate agentic applications that interpret user intent, manage conversation context, call tools and APIs, and perform multi-step analysis with appropriate user review.
• Develop the Angular user experience, .NET APIs, and Python agent services, integrating with existing reporting systems, authentication, and data access controls.
• Own the evaluation pipeline by curating representative datasets with domain experts, defining scoring rubrics and acceptance criteria, and combining automated checks with human review.
• Evaluate answer correctness, supporting evidence, tool selection and execution, and end-to-end task success, including ambiguous requests, unsupported questions, and failure scenarios.
• Integrate evaluations and regression tests into CI/CD pipelines.
• Version prompts, evaluation datasets, and configurations; compare changes against baselines and enforce quality gates before release.
• Monitor production quality, latency, cost, and failures through traces and user feedback.
• Convert observed failures into reproducible test cases and measurable improvements.
• Implement reliable execution through timeouts, retries, cancellation, durable state, and recovery mechanisms.
• Protect sensitive data and enforce authorization at tool and data boundaries.
• Collaborate with product owners and domain experts to validate analytical findings, preserve source traceability, and communicate uncertainty and application limitations clearly.
Required Qualifications:
• 8+ years of professional software development experience, including the delivery and support of secure, scalable enterprise applications.
• Hands-on experience building agentic applications using large language models, tool calling, structured outputs, prompt and context design, and multi-step workflows.
• Strong full stack development skills with Angular, .NET Web APIs, and Python, including integration with existing enterprise services.
• Practical experience building evaluation and quality pipelines for AI applications, including test datasets, scoring criteria, regression analysis, and human review.
• Ability to explain how evaluation results have informed a release or quality improvement.
• Strong SQL, relational data modeling, and data analysis skills, with attention to numerical accuracy, missing data, and traceability of results.
• Experience with cloud-native application development on Azure.
• Experience with CI/CD, automated unit and integration testing, and end-to-end testing.
• Experience with authentication, authorization, secure API design, observability, and reliable asynchronous processing.
• Strong problem-solving and communication skills, with the ability to work independently and collaborate with product, engineering, and domain experts.
Preferred Qualifications:
• Experience with retrieval-augmented generation (RAG), including document ingestion, retrieval, grounding, and retrieval quality evaluation.
• Experience with healthcare analytics, enterprise reporting, or other applications requiring careful handling of sensitive data and verifiable analytical results.
• Experience taking AI prototypes into production and improving quality, latency, and operating costs using evaluation results and production evidence.
