Skip to main content Skip to secondary navigation

Food for Thought: Delivering continuous improvement through better experimental analysis

Main content start
Four members of the DoorDash senior project team pose for a photo

Team Members
Danielle DeVera, Philip Clark, Wyatt Pontius, Juliet Daniel

Faculty Mentor
Ross Shachter

Doordash logo

The Client: DoorDash

DoorDash is a technology company that uses logistics services to offer food delivery from restaurants on-demand.

The Problem to Solve

The team partnered with the DoorDash data science team to help improve how they experiment with their assignment algorithm. Specifically, DoorDash wanted to improve switchback experiment analysis for their assignment algorithms, and investigate the following questions:

  • What modeling strategies can DoorDash use to reduce the misclassification of experimental results?
  • How can DoorDash make more accurate predictions about whether a change in its assignment algorithm will cause a change in delivery duration?
  • How can DoorDash decide to ship or reject a new assignment algorithm without wasting valuable time or money?

Engineering a Solution

Tools and Methodologies

The team built a Monte Carlo simulation of DoorDash's platform in which they could artificially control the treatment effect and thereby establish a ground truth. They used a variety of modeling techniques to improve the quality of: 

  • Variance estimation techniques
  • Simple causal inference model, which adds in other control variables
  • Sequential testing to reduce experiment time
  • Operational analysis of the cost of false positives and false negatives

Recommendations and Results

  • To best reduce the probability of false positive and false negative results from its experiment data, use a combination of the sandwich cluster covariance estimator and multivariate model.
  • To iterate more quickly than standard t-tests on assignment algorithm changes with high confidence in its results, use a combination of a multivariate model and sequential testing.
  • Operational analysis shows the total cost of error is reduced by $2.5M, with an overall savings of $3.5M by using the recommended modeling techniques.
Embed Code

All 2019 senior projects