To get the best candidate experience, please consider applying for a maximum of 3 roles within 12 months to ensure you are not duplicating efforts.
Job Category
Software Engineering
Job Details
The Experience
We are seeking a Principal Software Engineer to join our Platform Trust Intelligence and Security (PTIS) team - the runtime substrate that detects, isolates, and contains threats across an enterprise pivoting to autonomous AI. As organizations deploy AI agents that execute multi-step reasoning, access sensitive data, and take real-world actions, a new class of security risk emerges. Agents operating outside their intended scope can exfiltrate data, probe permission boundaries, or trigger privileged operations at a speed and scale no human user can match - while appearing to function exactly as designed.
Product-layer guardrails define the operating envelope; this platform catches what slips through. You will design and build the secure runtime, telemetry, detection, and containment layers that let us observe every agent's behavior, run untrusted agent code under hard isolation, score it against learned baselines, and intervene safely when it deviates - without ever crossing a tenant boundary or stalling a legitimate workload.
This is a hands-on principal role at the intersection of platform engineering, secure runtimes, and applied machine learning (ML). With Agentforce woven into every layer of our platform, our engineers build intelligent systems that automate the repetitive, elevate the strategic, and power better decisions at scale. You will lead technical strategy for sandboxed agent execution, real-time behavioral detection, and policy-driven response orchestration - all under enterprise constraints including multi-tenant isolation, regulatory compliance, customer-trust Service Level Agreements (SLAs), and reversibility on every action that touches a customer's runtime.
This role is based in Bellevue, Washington and is Office Tech-Flexible (hybrid work environment).
What You'll Actually Be Doing
Build and ship high-quality, production-grade software using modern engineering practices, with AI as a core part of your development workflow - pushing the boundaries of AI development tools to deliver secure, optimized, and high-quality code.
Design and orchestrate complex systems where AI agents integrate seamlessly into human workflows, driving efficiency and innovation at scale. Contribute to building and maintaining shared system context - an explicit repository of system designs, constraints, and standards that enables AI to operate accurately and reliably. Critically evaluate code (human- or AI-generated) for correctness, quality, security, and performance.
Sandboxed Agent Execution and Hard Isolation
- Own the secure execution substrate for AI agents - the layer that turns "the agent wants to run code, call a tool, or touch a file" into a strongly bounded operation with a known blast radius.
- Design and operate microVM-based isolation using technology stacks like Firecracker for high-density, fast-boot agent sandboxes; benchmark against Kata Containers for VM-grade isolation under a Kubernetes-native operational model, and choose the right tool per workload (latency-sensitive tool calls vs. long-running code-interpreter sessions vs. third-party tool execution).
- Integrate and extend E2B-style code-interpreter sandboxes for agent code execution: filesystem snapshotting, network-egress allow-listing, per-session lifecycle, and secure artifact return.
- Implement tiered autonomy at the infrastructure layer - defining which actions an agent may take automatically, which require human approval, and which are categorically denied and enforced below the agent, so a compromised or jailbroken agent cannot opt out of the rule.
- Build the capability model: per-agent, per-tool credential scoping, ephemeral identity issuance, egress policy, syscall filtering, and process isolation - so a single agent's failure can never escalate into a tenant-wide or fleet-wide incident.
- Treat the sandbox itself as a threat surface: partner with offensive security to red-team escapes, side channels, and tool-abuse patterns, and design for graceful failure.
Detection, Behavioral Analytics, and Response
- Build the runtime detection layer that scores agent behavior against learned baselines - detecting bulk data access, privilege-escalation reconnaissance, anomalous tool-call sequences, and configuration drift in real time.
- Productionize stateful streaming inference at low latency: feature freshness SLAs, in-memory profile lookups, per-tenant model serving, and safe rollout of new detection logic across a global fleet.
- Own the policy enforcement and response orchestration layer - the single chokepoint through which every containment action flows. Implement reversible, audited response actions: throttle, suspend pending re-authentication (re-auth), revoke a credential, kill a session, and deactivate an agent.
- Deliver the report-only to live-containment rollout discipline: every new detection ships in shadow first, gated on efficacy validation before any customer-impacting action is enabled.
- Make undo trivial. A wrong containment decision against a production agent is a customer-impacting incident; the platform must keep an immutable audit trail and a one-click rollback path for every action it takes.
Telemetry, Schema Contracts, and Observability
- Build the schema-governed, contract-based telemetry pipeline that captures end-to-end agent traces - large language model (LLM) calls, tool invocations, intermediate reasoning steps, sandbox syscalls, network egress, and final outputs - at scale and at low latency.
- Drive the move off brittle application debug logs onto a stable event contract jointly owned by platform, ML, and security stakeholders. Define the schema as a hard interface, not a mere construct; treat it like a public API.
- Instrument the full stack - sandbox runtime, detection plane, response plane - into Prometheus/OpenTelemetry/Grafana with agent-specific SLOs: detection-score drift, containment-action error rate, sandbox cold-start latency, telemetry pipeline lag, and tool-call anomaly bursts.
Evaluation, Replay, and Continuous Quality
- Build evaluation (eval) and replay infrastructure so new detection models, new containment policies, and new sandbox configurations can be tested against real historical traces before going live - and so regressions are caught by automation, not by customers.
- Maintain golden-trace libraries, labeled incident corpora, and red-team prompt sets as first-class platform assets. Gate rollout on these suites the same way we gate code on tests.
- Drive continuous quality improvement by closing the loop from production traces to evaluation to detection, sandbox, and policy refinement, through to rollout.
CI/CD, Infrastructure-as-Code, and Developer Experience
- Build continuous integration and continuous delivery (CI/CD) pipelines (GitHub Actions, ArgoCD) that treat detection eval gates, sandbox security scans, and policy compatibility checks as first-class pipeline steps - no model, sandbox image, or policy ships without passing them.
- Maintain the platform as infrastructure-as-code (Terraform): reproducible, reviewable, and auditable. Hardened container baselines, signed artifacts, Software Bill of Materials (SBOMs), dependency scanning, and key rotation are non-negotiable.
- Build self-service surfaces that let detection engineers, ML scientists, and product teams iterate on agents and detections without platform-team involvement - always inside the guardrails the platform enforces by default.
Reliability, Compliance, and Cross-Team Architecture
- Establish alerting (Grafana, PagerDuty) for both traditional platform health and agent-specific signals; own on-call quality for the workload.
- Own compliance posture (SOC2, FedRAMP, ISO) for the AI-runtime surface area - auditable traces, regulatory-approved telemetry use, and customer opt-in/opt-out enforcement.
- Lead cross-team architecture across applied ML, AI-platform engineering, product security, incident response, and compliance - holding contracts steady across all of them to drive forward progress.
You're Our Person If...
- You have 9+ years of experience as a Platform Engineer, Security Infrastructure Engineer, or Software Engineer building production systems under strict security, compliance, and customer-trust constraints.
- You bring hands-on expertise with microVM and container isolation technologies - Firecracker, Kata Containers, gVisor, or equivalent - including production operation, performance tuning, escape-surface analysis, and integration into orchestration platforms.
- You have direct experience building or operating highly scaled sandboxed code-execution environments for AI agents - filesystem snapshotting, network-egress control, per-session lifecycle, and secure artifact handling.
- You have a strong understanding of tiered autonomy, capability-based security, and blast-radius controls for systems that take privileged action against tenant runtimes.
- You have experience building detection-and-response or policy-enforcement platforms at scale - telemetry pipelines, ML serving, runtime containment, audit trails, and reversibility.
- You have a strong streaming systems background: Kafka, Flink, or equivalent, with real-world experience running stateful streaming inference and feature pipelines in production.
- You have hands-on experience with ML serving in production: model rollout, profile/feature distribution, freshness SLAs, A/B and shadow deployments, and regression gating.
- You bring strong Python and JVM-language engineering skills, with comfort moving across the stack where the workload demands it.
- You have deep expertise in Amazon Web Services (AWS), Google Cloud Platform (GCP), and/or Azure, with comfort across multi-cloud and hybrid substrates.
- You have extensive experience with CI/CD (GitHub Actions, ArgoCD), infrastructure-as-code (Terraform), and containerization/orchestration (Docker, Kubernetes), including hardening patterns (Pod Security Standards (PSS), Open Policy Agent (OPA)/Gatekeeper, network policies, and service-mesh authorization (authz)).
- You have strong security fundamentals: threat modeling, Identity and Access Management (IAM), audit logging, schema governance, and supply-chain security.
- You demonstrate a genuine AI-first approach to engineering - using AI to move faster, build fluency across the stack, and contribute well beyond your core specialty.
- You have experience using AI tools (e.g., Claude Code, GitHub Copilot, Codex, Cursor, etc.) in development workflows.
- You bring advanced prompt engineering skills and the ability to write precise, structured prompts and cultivate the system context that makes AI outputs reliable, secure, and production-ready.
- A related technical degree required.
Even Better If...
- You have direct experience with agent harnesses and AgentOps in production: agentic loops, tool orchestration, structured output handling, multi-turn conversation management, and trajectory evaluation.
- You have familiarity with agent evaluation frameworks (LangSmith, OpenAI Evals, or equivalent) - building eval datasets, regression suites, and quality dashboards.
- You have a background in offensive security or red-teaming AI systems - prompt injection, tool abuse, jailbreaks, sandbox escape research, and ML supply-chain attacks.
- You have experience with vector or graph databases and Retrieval-Augmented Generation (RAG) pipelines, especially over security knowledge bases.
- You have experience with modern data platforms at scale: Iceberg, Kafka, Flink - applied to high-volume security telemetry.
- You have contributions to open-source agent runtime, sandbox, or eval tooling.
Unleash Your Potential
When you join Salesforce, you'll be limitless in all areas of your life. Our benefits and resources support you to find balance and be your best, and our AI agents accelerate your impact so you can do your best. Together, we'll bring the power of Agentforce to organizations of all sizes and deliver amazing experiences that customers love. Apply today to not only shape the future - but to redefine what's possible - for yourself, for AI, and the world.
