Senior Backend Engineer

Instrumentl

$175K — $220K *
US-AnywhereRemote in Canada
Education, Government & Non-Profit
5 - 7 years of experience
Job Overview by Ladders

Qualifications

  • 7+ years in building and shipping production backend systems using Python (such as FastAPI, Celery) with a reliable practices approach.
  • Hands-on experience with LLM features in production, emphasizing tool calling and agent workflows.
  • Strong skills in SQL, schema design, and data pipeline construction for evaluation and retrieval.
  • Proven ability to thrive in a fast-paced startup environment, demonstrating ownership, speed, quality, and simplicity.

Responsibilities

  • Build LLM agents for grant discovery and research assistance, focusing on task planning and multi-step workflows.
  • Transform prototypes into reliable, observable services with defined SLAs and fallback strategies.
  • Establish evaluation and observability to ensure AI effectiveness and cost-efficiency.
  • Write high-quality, tested backend code that supports data pipelines.
  • Collaborate with Product, Design, and GTM teams on project scope and UX measurement.
  • Conduct experiments and iterate based on results to improve features.
  • Enhance engineering standards through maintainable code, thorough documentation, and proactive reviews.

Benefits

  • 100% coverage of health, dental, and vision insurance for employees (50% for dependents).
  • Generous PTO, including parental leave.
  • 401(k) retirement plan.
  • Company laptop and home-office stipend.
  • Bi-annual company retreats.
  • Opportunities for professional growth in a fast-evolving environment.
Full Job Description
About the role

We're hiring a Senior Backend Engineer to own AI features end to end, from rapid prototype to production and the evaluation that keeps them honest. You'll build the APIs, tool-using agents, and RAG pipelines that turn frontier LLMs into grant discovery, application drafting, and research tools our 5,500+ nonprofits rely on every day. It's a high-ownership seat on a small team, where what you ship reaches customers fast and you help shape how we build AI here.

What you'll do

Ship AI to production
  • Build tool-using LLM agents (task planning, function and tool calling, multi-step workflows, guardrails) for grant discovery, application drafting, and research assistance.
  • Turn prototypes into resilient, observable services with clear SLAs, rollback and fallback strategies, and cost and latency budgets.
  • Stand up evaluation and observability so our AI stays grounded, safe, and cost-effective.

Build trustworthy backends
  • Write high-quality, thoroughly tested code across the backend and the data pipelines that power retrieval and evaluation.
  • Contribute to reliability practices: alerts, dashboards, and incident response.

Collaborate and raise the bar
  • Partner with Product, Design, and GTM on scoping, UX, and measurement.
  • Run experiments (A/B, canaries), interpret results, and iterate.
  • Raise engineering standards through clear, maintainable code, tests, docs, and thoughtful review.


What we're looking for

Required
  • 7+ years building and shipping production backend systems in Python (FastAPI, Celery, or equivalent), taking features from prototype to production with real reliability practices like tests, observability, and rollback.
  • Hands-on experience building LLM features in production: tool and function calling, multi-step agent workflows, and the guardrails and evals that keep them grounded, safe, and cost-effective. This is the core of the role.
  • Strong data fundamentals: SQL, schema design, and building pipelines that power retrieval and evaluation.
  • Thrives in a fast, scrappy startup environment with high ownership and a bias for action, speed, quality, and simplicity.

Nice to have
  • TypeScript and Node, plus familiarity with Ruby on Rails (our core platform) or a willingness to learn it.
  • Experience with AWS or GCP, Docker, CI/CD, and observability (logs, metrics, traces).
  • RAG depth: document ingestion, chunking and windowing, embeddings, hybrid search (keyword plus vector), re-ranking, and grounded citations.
  • Experience with re-rankers and cross-encoders, hybrid retrieval tuning, or search and recommendation systems.
  • Evaluation mindset: designing eval suites (RAG/QA, extraction, summarization) using automated and human-in-the-loop methods, with familiarity with frameworks like Ragas, DeepEval, or OpenAI Evals.
  • Orchestration frameworks: LangChain or LangGraph, LlamaIndex, Semantic Kernel, or custom orchestration.


Compensation & Benefits

For US-based candidates, the target salary range for this role is 175,000 - $220,000 USD, plus equity. Final compensation is determined based on experience, skillset, scope of responsibility, interview performance, and geographic location. We're committed to paying competitively and equitably.

For candidates based in Canada, compensation varies by province and will be shared by your recruiter early in the process.

Benefits
  • 100% covered health, dental, and vision insurance for employees (50% for dependents)
  • Generous PTO, including parental leave
  • 401(k)
  • Company laptop and home-office stipend
  • Bi-annual company retreats
  • Instrumentl is evolving rapidly. You'll always have new challenges and opportunities to grow here.


Similar Jobs

More Jobs at Instrumentl

More Education, Government & Non-Profit Jobs

Find similar Senior Backend Engineer jobs: