OverviewMicrosoft Security's
Getting Customers Ready for AI team is seeking a
Data Engineer II to help build the data foundations that power AI-native security systems.
In this role, you will design and develop scalable data platforms, pipelines, and telemetry systems that transform raw signals across identity, devices, data, applications, and infrastructure into high-quality, trusted datasets for analytics and AI-driven insights.
You will work in a collaborative engineering environment to deliver reliable, secure, and performant data systems, enabling downstream AI models, security detections, and customer-facing experiences. This role is ideal for engineers looking to deepen their expertise in large-scale data systems, distributed processing, and modern data platform architectures for AI and Agents.
ResponsibilitiesData Platform & Pipeline Engineering - Design and build scalable data pipelines (batch and streaming) to process large volumes of security and operational data.
- Develop and optimize ETL/ELT workflows that transform raw telemetry into structured, consumable datasets.
- Implement data ingestion frameworks to integrate multi-source data from services, APIs, and event streams.
- Improve pipeline performance, reliability, and efficiency through partitioning, indexing, and optimization techniques.
Data Modeling & Storage Systems - Design and evolve data models, schemas, and storage strategies for analytics and AI use cases.
- Work with distributed storage systems (e.g., data lakes, warehouses) to ensure scalability and cost efficiency.
- Maintain data partitioning, retention, and lifecycle strategies aligned to business and compliance needs.
- Ensure data is structured for downstream ML pipelines, feature engineering, and analytics workloads.
Data Quality, Governance & Security - Implement data validation, quality checks, and observability frameworks to ensure data accuracy and reliability.
- Apply best practices for data governance, lineage, and auditing across pipelines and datasets.
- Ensure compliance with security, privacy, and regulatory requirements when handling sensitive data.
- Contribute to standardization of data contracts and schemas across services.
AI Readiness & Intelligence Enablement - Enable high-quality datasets for AI/ML teams, including support for feature pipelines and training data preparation.
- Collaborate with AI engineers to ensure data is optimized for RAG pipelines, model training, and evaluation workflows.
- Build and maintain telemetry pipelines and metrics systems that provide insights into AI system performance and usage.
- Support development of data-driven signals and insights that improve customer security posture.
Collaboration & Execution - Partner with engineering, data science, and product teams to deliver end-to-end data solutions.
- Contribute to system design discussions, architecture reviews, and cross-team integration efforts.
- Work across teams to ensure consistent data definitions and interoperability across platforms.
Engineering Excellence - Write clean, maintainable, and well-tested code for data pipelines and platform services.
- Build monitoring and alerting mechanisms for pipeline health, latency, and data quality issues.
- Document data architecture, pipeline designs, and operational practices for knowledge sharing and scalability.
- Follow best practices for performance optimization, cost management, and reliability engineering.
- Embody our culture and values.
QualificationsRequired Qualifications - Bachelor's Degree in Computer Science or related technical field AND 2+ years of software, data, or related engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python
Other Requirements: Ability to meet Microsoft, customer, and/or government security screening requirements are required for this role. These requirements include but are not limited to the following specialized security screenings:
- Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years.
Preferred Qualifications:- Hands-on experience building or maintaining data pipelines and distributed data systems.
- Programming experience in Python, Scala, or SQL for data processing and pipeline development.
- Understanding of data modeling, ETL processes, and large-scale data processing concepts.
- Experience with big data frameworks (e.g., Spark, Flink) or streaming systems (e.g., Kafka/Event Hub).
- Familiarity with cloud-based data platforms (Azure preferred, e.g., ADLS, Synapse, Databricks).
- Exposure to data governance, security, and compliance practices in enterprise environments.
- Experience supporting AI/ML workloads through data pipelines, feature stores, or training datasets.
- Developed problem-solving skills with the ability to operate in ambiguity and deliver incrementally.
#MSFTSecurity
Software Engineering IC3 - The typical base pay range for this role across the U.S. is USD $102,100 - $202,200 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $133,800 - $219,200 per year.
Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here:
https://careers.microsoft.com/us/en/us-corporate-pay
This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.
