About the role - Principal Engineer

In this role, you will be responsible for designing and implementing solutions for complex, large-scale systems, including:

• Building distributed 3D content streaming, optimizing resources and implementing load-balancing strategies.
• Developing advanced distributed scheduling solutions.
• Creating resilient validation frameworks that can handle high-stakes environments.

The ideal candidate thrives on solving challenging engineering problems. You will be instrumental in scaling throughput, improving latency for real-time evaluations, and creating comprehensive monitoring systems to ensure the health and performance of our live network.
Key responsibilities
Distributed Systems

• Architect and implement distributed systems for content streaming & distribution, model evaluation, inference pipelines
• Make high-level design decisions and create architectural blueprints, setting the technical direction and strategy for scalability, performance, and innovation
• Build monitoring and observability tools to track system health, throughput, and fairness across the subnet
• Analyze fault-tolerance and high availability issues, performance and scale challenges, and solve them
• Pinpoint problems, instrument relevant components as needed, and ultimately implement solutions
• Develop elastic, load balancing, and failure recovery strategies to ensure performance and resilience
• Design high-availability infrastructure for real-time leaderboards and validator mechanisms.

Protocol Development

• Protocol Architecture: Design and implement a custom, low-latency networking protocol to serve P2P real-time streaming and inference workloads at scale, specifically for cloud-rendered experiences streamed to XR devices.
• Decentralized Systems Engineering: Architect and optimize a resilient distributed system with a focus on Byzantine Fault-Tolerant (BFT) environments, ensuring robust operation and seamless recovery across a decentralized peer-to-peer network infrastructure.
• Algorithmic Research & Development: Investigate and design new data structures and algorithms, applying advanced distributed computing approaches that leverage the latest and state-of-the-art hardware technology.
• Performance Optimization: Develop and integrate a comprehensive framework for latency mitigation and adaptive Quality of Service (QoS), dynamically adjusting stream parameters to ensure a high-performance, immersive experience despite network variability.
• Operationalization of Research: Translate research protocols and advanced P2P concepts into production-grade, scalable systems, collaborating closely with business and tokenomics teams to align technical development with strategic goals.

Mentorship & Collaboration

• Mentor fellow engineers to achieve success by guiding them to make high-level architectural decisions, solve complex distributed systems challenges, and translate innovative research into resilient, production-ready software. (ex. Building scalable architecture or database management)
• Contribute to ongoing system audits and post-mortems to ensure the platform is future-proofed against growing user and transaction volumes
• Foster a collaborative and inclusive team environment by facilitating code reviews, architecture discussions, and problem-solving sessions.
• Project management, keeping the project on track and having all stakeholders informed
• Partner with product and research teams to translate requirements into a clear technical roadmap and break down complex projects into actionable tasks

Education and experience

Required:

• Expert-level backend development experience in Golang and Rust (6+ years experience)
• Proven experience building microservices and cloud-native applications on platforms like AWS, GCP, or Azure
• 4+ years of experience in building large-scale distributed systems features or applications.
• Technical leadership as well as team motivation, direction and pace
• Good understanding of CI/CD pipelines and related tools and technologies
• Concurrency is a challenge that you are comfortable tackling
• Experience in complex Distributed Systems in production, preferably in byzantine settings (blockchain) or similar
• Mastery of software architecture, design patterns, and system design principles.

Nice to Have:

• Experience with Linux system level development, distributed system, or scheduling algorithm is an asset
• Understanding of GPU architecture
• Proven experience with: DHTs, gossip protocols, BFT consensus, or P2P network resilience techniques.

Location

Mawari's Canadian office is at the Waterloo Accelerator Centre - a modern and vibrant facility adjacent to the University of Waterloo campus. It's conveniently located on the Ion elect light rail systems running North-South here in Waterloo Region. The Waterloo Accelerator is a modern work environment with plenty of natural light, open space and flexible meeting areas as well as free coffee/tea service.

Hiring Policy

At Mawari, we're building a team where everyone belongs. We know that diverse backgrounds, perspectives, and experiences make us stronger - and help us create better products for our customers and communities.

We welcome candidates of all races, colours, creeds, ancestries, national origins, disabilities, ages, sexes, sexual orientations, gender identities and expressions, family situations, and more.