Food for Thought: Delivering continuous improvement through better experimental analysis
Main content start
Team Members
Danielle DeVera, Philip Clark, Wyatt Pontius, Juliet Daniel
Faculty Mentor
Ross Shachter
The Client: DoorDash
DoorDash is a technology company that uses logistics services to offer food delivery from restaurants on-demand.
The Problem to Solve
The team partnered with the DoorDash data science team to help improve how they experiment with their assignment algorithm. Specifically, DoorDash wanted to improve switchback experiment analysis for their assignment algorithms, and investigate the following questions:
- What modeling strategies can DoorDash use to reduce the misclassification of experimental results?
- How can DoorDash make more accurate predictions about whether a change in its assignment algorithm will cause a change in delivery duration?
- How can DoorDash decide to ship or reject a new assignment algorithm without wasting valuable time or money?
Engineering a Solution
Tools and Methodologies
The team built a Monte Carlo simulation of DoorDash's platform in which they could artificially control the treatment effect and thereby establish a ground truth. They used a variety of modeling techniques to improve the quality of:
- Variance estimation techniques
- Simple causal inference model, which adds in other control variables
- Sequential testing to reduce experiment time
- Operational analysis of the cost of false positives and false negatives
Recommendations and Results
- To best reduce the probability of false positive and false negative results from its experiment data, use a combination of the sandwich cluster covariance estimator and multivariate model.
- To iterate more quickly than standard t-tests on assignment algorithm changes with high confidence in its results, use a combination of a multivariate model and sequential testing.
- Operational analysis shows the total cost of error is reduced by $2.5M, with an overall savings of $3.5M by using the recommended modeling techniques.