Connsci is seeking a Senior Data Engineer to join our growing organization in support of one of our government customers. This individual will own the development and delivery of a cloud-based data anomaly detection application. This role will set up a hosted infrastructure and will architect, lead and manage development activities according to a formal project schedule and deliverables plan, while providing hands-on technical support and leadership across data engineering, software development, analytics, anomaly detection, reporting, and system delivery. The solution is built on the Databricks Medallion architecture, with complex business-rule logic used to identify, organize, and present anomalies in the data set through Power BI and related dashboard capabilities in an Azure Government virtual environment.
Key Responsibilities- Design and maintain secure data ingestion pipelines using Azure Data Factory, API Management, and SFTP
- Development and implementation of the data anomaly detection system, which will be hosted on Azure Gov. Cloud and based on Databricks Medallion architecture, Power BI reporting, and related application components.
- Create reports and technical documentation, mentor staff, resolve blockers, and ensure technical quality across Databricks, Power BI, and hosted web application components.
- Analyze source data from multiple systems with varying schemas.
- Develop and process data ingestion, rationalization, and analytic programming to support accurate anomaly detection.
- Manage data storage in Azure Data Lake Storage Gen2 (bronze/silver/gold architecture) with evidentiary controls (immutability, hashing, lineage).
- Optimize pipelines for performance, scalability, and cost efficiency across DEV, TEST, POC, PROD environments.
- Support development of business-rule logic used to classify, flag, summarize, and display anomalies in the hosted dashboard application.
- Maintain system security in compliance with NIST 800-53, 800-171, and related standards.
- Maintain technical work plans aligned to project milestones, contract deliverables, customer priorities, and formal schedule requirements.
- Maintain development, documentation, and functional quality standards.
- Track, communicate, and report on technical status, risks, dependencies, and schedule compliance to project leadership and customer stakeholders.
Our Ideal Candidate Will Have- Strong understanding of the software development lifecycle (SDLC), system architecture, application integration, and associated documentation requirements.
- Experience working with imperfect, inconsistent, incomplete, or high-volume datasets.
- Excellent written and verbal technical and professional communication skills.
- Strong leadership, organizational, planning, scheduling, documentation, and problem-solving abilities.
Required Qualifications- Bachelor's degree in IT, Computer Science, Engineering, or related field; or an equivalent combination of education and experience.
- At least 10 years of experience in software engineering, data engineering, or business intelligence.
- At least 5 years of experience designing, building, and deploying web-based applications.
- At least 5 years of experience managing software development projects with formal schedules, milestones, and phased customer deliverables.
- At least 5 years of experience with Power BI, ETL/ELT development, data modeling, dashboard and reporting solutions, and secure cloud environments.
- At least 2 years of experience with Databricks Medallion architecture (Bronze, Silver, Gold).
- At least 3 years of experience with data modeling and transformation using SQL and PySpark.
Preferred Qualifications- Prior experience supporting government, inspector general, audit, investigation, complaint, or other regulated case-management environments.
- Experience with Azure Government
- Relevant Microsoft Azure and Databrick certifications.
- Experience designing high-reliability intake solutions where incoming records must be traceable, recoverable, auditable, and processed without manual intervention.
