2023 – 2024
Mozilla
Software Engineer, bugbug
bugbug points machine learning at the parts of Firefox development that do not scale with headcount. Thousands of Bugzilla reports need triaging and routing every week, and an enormous test matrix needs narrowing to whatever a given patch actually justifies running. It is open source and runs in Mozilla's production infrastructure.
Firefox produces thousands of bug reports and an enormous volume of test results every week. Triage is slow and repetitive, and running the whole suite on every push costs real money. Both problems have the same shape: the answer is already sitting in the historical data and nothing was pulling it out. The constraint is that an automated call has to be cheap enough to make on every push, and clear enough that an engineer will act on it rather than ignore it.
Data ingestion
Bug metadata, comment histories, patch diffs and test outcomes come in from Bugzilla's REST API and Mozilla's CI telemetry. They get normalized into a common shape and cached, so training and inference are not hammering upstream services for the same data.
Feature extraction
Bug text is vectorized with TF-IDF and related techniques. Patch diffs are broken down into the files, directories, and functions they touch. Those combine with structured metadata, things like component or priority and the reporter's history, into one feature vector per example.
Training pipeline
Models train offline on labelled history using scikit-learn and XGBoost. The jobs run on Taskcluster, Mozilla's distributed CI, and write versioned artifacts to object storage, so any prediction can be traced back to the model that produced it.
Serving
A Python HTTP service exposes the predictions, cached in Redis with an expiry. Bugzilla tooling and CI consume them asynchronously, so nothing on a developer's critical path ever sits waiting on inference.
Build the evaluation harness first
Reproducible splits, metrics tracked across versions, regression detection in CI. This is what turns a change to a model into a measurement instead of a hunch. It is the least interesting part of the system to build and the part that decides whether anything else can be improved safely.
Stay interpretable on structured inputs
Gradient boosted trees before anything heavier. They train quickly, they can be debugged, and on a mix of structured and text features they hold up well. That puts the effort where it actually pays, in the features, not in an architecture nobody can explain to the engineer whose bug just got misrouted.
Expect drift, not accuracy loss
Schema changes upstream, label distributions that shift under you, caches that go stale. These take models down far more often than a regression in accuracy does, and the tell is that a model scoring on changed inputs still answers confidently. Watching the inputs matters as much as watching the outputs.
Machine learning systems fall over in production for reasons that have nothing to do with the model. Drift, an upstream schema change, a cache that quietly did not invalidate: any one of those will cost you more incidents than a couple of points of accuracy ever will.
Feature engineering that comes out of understanding the domain beats changing the architecture. Watching how the people doing the job actually decide is worth more than any hyperparameter sweep.
The model is a small piece of the system. Monitoring, versioning, rollback, and evaluation take longer to build and matter far more to whether anyone trusts what comes out.
bugbug runs in Mozilla's production infrastructure, where its classifiers back Firefox bug triage and component assignment and its test selection models feed into which tests CI chooses to run.