Skip to main content Skip to secondary navigation

Meet our graduates | Hannah Li (PhD '22)

Main content start

July 19, 2022. Interviewed by Jim Fabry. Audio and text edited for length and clarity.

As part of MS&E's 2022 Graduates podcast series, we chat with Hannah Li, a graduate of the PhD program in MS&E.

Hannah shares how her desire to create impact for large numbers of people led her to study operations research in MS&E. She describes her research on eliminating bias in algorithms driven by artificial intelligence, how she blends the study of technical systems with the study of the people involved with those systems, and her plans to continue in academia as a professor at Columbia Business School. 

oEmbed URL

My name is Hannah Li, and I'm a PhD candidate in operations research in MS&E.

Can you tell me a little bit about your background? Where did you grow up and where did you study before Stanford?

I was born in China, then spent most of my time in Kentucky, New York, and then California. It was like growing up all over the place. I did my undergrad at Pomona College, which is a small liberal arts college with 400 people in each grade. That's where I developed my interest in math, and that's really what motivated me to do further studies in grad school.

I think Pomona definitely influenced a lot of my research interests; there's a lot of emphasis there on interdisciplinary research. A large part of what motivates me now is thinking about how to use operations research techniques with data science, statistics methods, and economics techniques to study things in socially impactful areas.

Are there any specific experiences that really defined your interest in engineering?

I almost stumbled into engineering. I was doing math at Pomona, and I really liked the technical aspect. I really enjoyed proofs and theorems and all these things that we were doing there, the very theoretical aspects of it.

But then I realized what I really wanted to do was use this math to help people, to actually make an impact somewhere. And that translated very naturally into engineering, because it’s using a lot of mathematical tools to help design better systems.

Can you tell me about your research—what sort of things are you working on, and how did you become interested in them?

My research focuses on a couple things centered around data science methods. I like to say data science methods for complex systems where traditional data science techniques aren’t really good enough.

For example, one of my research projects is about how to run experiments, or A/B tests, in marketplace settings. Typically, when we run experiments where we’re thinking about introducing a new intervention, we randomize people into treatment and control groups. Then we compare the average behavior of the two groups and we say the difference in the behavior is due to this new intervention.

This is very common in healthcare trials and A/B testing for platforms, but it actually becomes a lot more complicated in settings like marketplaces or social networks where you have people interacting with each other. In those settings, when you give a change to one individual, it affects how other individuals relate to them and it affects the behavior of other individuals.

This complicates the statistics, and if you just use traditional statistical methods you actually end up getting biased or wrong answers. So one of my research projects is thinking about, if we know this effect is occurring, how do we use our understanding of interactions in the marketplace—meaning economic interactions, the engagement between individuals. How can we incorporate that knowledge into the data science process to get rid of this bias and have better estimates?

What’s an example of an impact that you’re looking to have?

These days, basically any online platform uses A/B testing to help them make decisions. Think anything like Google, Facebook, Microsoft, or Airbnb. All these platforms are running tens of thousands of experiments per year.

Anytime these platforms want to make a change—for example, how they show information to users, the user interface, how a webpage looks, or pricing—they’re going to run an experiment on the platform to see how the new change affects the users before they decide whether or not to introduce it broadly.

In this setting, if you're running an experiment and you have biased estimates, it can potentially affect whether you're making the right decisions about which new features to introduce. But by having a better experimentation protocol with less biased and more reliable experimental estimates, you’re creating the ability to make better decisions that are affecting the millions of people that use these platforms.

How did the pandemic affect your time at Stanford?

The pandemic forced me to become a lot more intentional with everything that I was doing. During that time, the default was to be alone in my room unless I reached out to do something, either in person or virtually. The standard water cooler chats with people that you run into in the hallway were not there. A lot of the routines we had, the people we just ran into along the way—that wasn't there anymore.

If I wanted to work with someone I would have to reach out to them. If I wanted to socialize with someone, I’d have to make that effort. It was really a period of self-reflection, thinking about who I was reaching out to, and who I wished I was reaching out to more during those times.

How do you imagine your research interests might change over the next 5-10 years?

One thing that I think is always going to be constant in my research interests is the motivation to have an impact somewhere, to have my research help people and actually be applicable enough that people will put it into use. I think no matter what I do, that's one of the core guiding principles behind my research.

Right now, I'm looking into experiment design with A/B testing in marketplace platforms, and I'm also doing some work in the education space as well. And in the future, the precise topic I am working on may change, or maybe I’ll find other areas where I think I can be more impactful, but I think it will always be under the lens of wanting to create something that reaches people.

What are your career plans after Stanford and how did you decide on them?

I will be a professor at Columbia Business School, and I’ll start there in about a year after doing a post-doc at MIT. During the post-doc, I’ll be studying the interaction between human decision making and AI systems like recommendation systems.

Right now, so much of what platforms are optimizing is the information they show you, using recommendation systems or different machine learning systems. But there's also a lot more to study beyond that, like how people engage with these interactions to create their own decisions. Ultimately, that's what we care about.

For example, when people interact with these systems, maybe in strategic ways, does that influence the information that’s collected on these platforms and thus what's fed into future algorithms? We’re still deciding on specific research questions, but we'll be studying that general topic.

It seems like your work focuses on user experience. Would you say that's a good characterization of your interests?

I do think that's a common thread. Ultimately, my goal is to create research that impacts people, so a lot of that work involves studying the people, too. A lot of my projects are a blend of more traditional engineering or statistics projects, but with the added element of studying the people more.

What excites you most about your future?

It’s the ability to work on problems that I think are interesting and have the freedom to decide what I’m working on. Going out and finding questions that people haven't looked at before is super exciting for me. That's the reason I wanted to be a professor and go into academia, the ability to keep defining my own questions.

What advice do you have for future MS&E students? How can they make the best use of their time in the department and at Stanford?

Reach out to people. There are so many resources here at Stanford—within MS&E and also in different departments. Stanford has such a culture of interdisciplinary research and very little barriers between departments and groups, so if you're interested in something—even if it's in a slightly different topic—reach out to people.

Everyone at Stanford has a lot going on, so they're not necessarily going to reach out to you if you don't do it first. But I find that once you do reach out, people are willing to help you, talk with you, and engage. So if you're interested in something, go make it happen, because there are a lot of resources here.

What will you miss the most about Stanford and the Bay Area?

There's such an entrepreneurial aspect about everything here, both in the startup culture—a lot of my friends are graduating and forming their own startups—and also in the research itself.

I think there's the same energy in the research culture of, “I want to do something new, and if I want to do it I’m just going to go after it and make it happen; I will put all my energy into doing it.” That's something that's super inspiring and something that I hope to take with me into the future.

Student stories & voices