Senior Software Engineer, Production Pipelines

This is a production engineering role. You will co-design and productionize the models our team trains and run them reliably and efficiently across very large image volumes. Your work will include systems engineering and innovations in the machine learning pipelines and infrastructure: packaging, scheduling, distributed inference, data movement, monitoring, and an automated loop to send model errors or failures back to annotation and retraining.

You report to the Head of Computation, and work, as needed, with other Mindspan teams (Microscopy and Optics, Expansion Microscopy, Scalable Chemistry, and Automation and Industrialization), using logic and evidence to advance your work so as to boost the performance of all.

The position is intended for an experienced software engineer from industry who has run machine learning in production and wants to apply that experience to science. Research experience is not required. What matters is that you have owned a system other people depended on, measured it, and made it faster and cheaper over time. The job also requires systematic documentation, analysis, and iterative problem solving, to arrive at working pipelines, apply them, and share them.

• Turn trained research models into versioned, tested, deployable inference services and batch jobs.
• Build and run distributed inference over very large, chunked volumes.
• Own the automated training loop.
• Build the evaluation and release process, so that every model version is scored on the same benchmarks before it runs on production data.
• Instrument the pipeline: throughput, GPU use, cost per unit volume, failure rates, and data lineage from raw image to final output.
• Make the system usable by scientists.
• Documentation and training: keep runbooks and design records that others can execute, with the reasoning behind each decision.
  
  

The disorders of the brain affect over 1,000,000,000 people around the world: Alzheimer’s, Parkinson’s, stroke, and other conditions. We operate with a sense of urgency, even as we look to understand the brain at its most fundamental levels.

We are hypercollaborative, and focused on impact over short-term markers of productivity. Creativity and failure are the stepping stones to scale, and working together with “strong opinions, weakly held” is our style.

We solve real engineering problems the best way we can: inventing fast when we have to, and using good existing solutions where they exist. We believe hard problems only yield to many disciplines working together, so collaboration is required, not optional.

We expect people to explore the unknown, which means making mistakes daily, owning them, and learning from them to improve and excel.

We are recruiting a team of scientists and engineers who exemplify all these values.

Required Qualifications.

• Five years building and operating production software, including time spent running machine learning models in production.
• Strong Python and software engineering practice: code review, testing, CI/CD, packaging, and owning code that other people depend on.
• Direct hands-on experience with distributed batch or inference systems on a cluster or in the cloud (for example, SLURM, Kubernetes, Ray, Dask, or Spark).
• Experience with containers, workflow orchestration, and infrastructure as code.
• Experience deploying models for GPU inference and improving inference speed.
  
  

Preferred Qualifications.

• Experience running a data engine or active-learning loop in industry, in autonomous driving, medical imaging, or a comparable setting.
• Experience with chunked arrays and image formats at terabyte scale or larger.
• Experience with experiment tracking, model registries, and dataset versioning.
• Experience operating shared GPU clusters, including scheduling, quotas, and job accounting.
• Experience with structural microscopy data, or other scientific imaging data.
• Contributions to open-source infrastructure or machine learning tooling.
  
  

The ideal candidate is an engineer who has run machine learning systems at scale and wants that work to produce scientific results. You are intellectually curious, relentlessly practical, and happiest when solving problems that have never been solved before. You would rather make a pipeline reliable than clever, and you measure your work by how much faster the scientists around you can move.

You have a strong detail orientation and exceptional organizational, planning, interpersonal, and communication skills. You are good at writing, can work independently and lead and manage, and can prioritize and multitask as needed. Self-motivation, commitment to high quality standards, and a service-oriented mindset are all key. You have good judgment, balancing flexibility and focus, action and planning, speed and thoughtfulness, as appropriate. You are open to giving and receiving feedback, constructively and thoughtfully, to advance scientific missions and personal growth.

This is an in-person job. Flexible hours may be required at times.

The chance to build the production system of a new research institute alongside some of the most adventurous scientists in the field, with real autonomy and a direct line to the people setting the scientific agenda. You will get to contribute to addressing one of the most important unmet medical needs of our time, as part of the initial team. You will learn lots of things that probably cannot be learned any other way, as you solve problems, build, and work with others. As Mindspan grows, we expect this role to grow with it. We are excited to invest in someone who wants to take on increasing responsibility over time.

Comprehensive health, dental, vision, and family benefits, and a matching retirement contribution.

Competitive salary ($180,000 to $235,000), depending on experience.