Research
Working Papers
- Designing Human-AI Collaboration: A Sufficient-Statistic ApproachNikhil Agarwal , Alex Moehring, and Alexander Wolitzky2025
We develop a sufficient-statistic approach to designing collaborative human-AI decision-making policies in classification problems, where AI predictions can be used to either automate decisions or selectively assist humans. The approach allows for endogenous and biased beliefs, and effort crowd-out, without imposing a structural model of human decision-making. We deploy and validate our approach in an online fact-checking experiment. We find that humans under-respond to AI predictions and reduce effort when presented with confident AI predictions. AI under-response stems more from human overconfidence in own-signal precision than from under-confidence in AI. The optimal policy automates cases where the AI is confident and delegates uncertain cases to humans while fully disclosing the AI prediction. Although automation is valuable, the additional benefit from assisting humans with AI predictions is negligible.
- Personalization, engagement, and content quality on social media: An evaluation of Reddit’s news feedAlex MoehringRevise and Resubmit at Management Science, 2024
Digital platforms increasingly curate their content through personalized algorithmic feeds. Platforms have an incentive to promote content that increases the predicted engagement of each user to lift advertising revenues. This paper studies how ranking content to maximize engagement affects the credibility of news content with which users engage. In addition, I evaluate how the ranking algorithm itself can be designed to promote engagement with high-credibility content. Using data from the Reddit politics community, I exploit a novel discontinuity in the ranking algorithm to identify the causal effect of a post’s rank on the number of comments it receives. I use this discontinuity to identify a model of user comment decisions and estimate the credibility of news content that users engage with under a personalized engagement-maximizing algorithm. The personalized engagement-maximizing algorithm exacerbates differences in the credibility of news content with which users engage. I then evaluate a credibility-aware algorithm that explicitly promotes credible news publishers and find the platform can substantially increase the share of engagement with high-credibility publishers for a small reduction in total engagement. These findings suggest algorithmic interventions can be a useful tool for managers to balance engagement quantity and content quality.
- Combining human expertise with artificial intelligence: Experimental evidence from radiologyNikhil Agarwal , Alex Moehring, Pranav Rajpurkar , and Tobias SalzRevise and Resubmit at Econometrica, 2024
While Artificial Intelligence (AI) algorithms have achieved performance levels comparable to human experts on various predictive tasks, human experts can still access valuable contextual information not yet incorporated into AI predictions. Humans assisted by AI predictions could outperform both human-alone or AI-alone. We conduct an experiment with professional radiologists that varies the availability of AI assistance and contextual information to study the effectiveness of human-AI collaboration and to investigate how to optimize it. Our findings reveal that (i) providing AI predictions does not uniformly increase diagnostic quality, and (ii) providing contextual information does increase quality. Radiologists do not fully capitalize on the potential gains from AI assistance because of large deviations from the benchmark Bayesian model with correct belief updating. The observed errors in belief updating can be explained by radiologists’ partially underweighting the AI’s information relative to their own and not accounting for the correlation between their own information and AI predictions. In light of these biases, we design a collaborative system between radiologists and AI. Our results demonstrate that, unless the documented mistakes can be corrected, the optimal solution involves assigning cases either to humans or to AI, but rarely to a human assisted by AI.
- Social influence and news consumptionAlex Moehring, and Carlos Molina2023
Populations in several countries have become decidedly more polarized over the last decades. Many believe that social media, which facilitates interactions within echo chambers, is partly to blame. These interactions can trigger two distinct effects on the demand for biased news. First, individuals could be influenced by the news consumption of their peers, for example, because they value keeping an ideologically congruent news diet with their peers. Second, individuals might purposefully skew their news consumption preferences, anticipating that their choices will be observed by their peers. We design a field experiment on Twitter to separately identify the importance of both mechanisms. Our experiment induces variation in both an individual’s perceptions of the political leanings of their peers’ news consumption and the visibility of their own news preferences to their social media followers. We track participants’ sharing behavior and news consumption, proxied by the news outlets that the participants follow. Our main result is that participants alter their news diet when they believe these choices will be observed by their peers, as in the second channel above. We find that individuals primarily value presenting themselves as following a balanced set of news which attenuates the demand for biased news. In contrast, we do not find evidence in favor of the first channel: our experimental variation does influence beliefs about the news diets of their peers but participants do not respond by changing their own news diets. In sum, we find little evidence supporting the hypothesis that, at least through the lenses of these two mechanisms, online interactions with like-minded people are a major contributor to the demand for polarized news content. Instead, our paper uncovers one mechanism for which we should expect the opposite: as social media platforms amplify the visibility of user interactions, the importance of social image concerns increases and, as a result, users adjust their news consumption to be more balanced.
- Social influence and news consumptionSusan Athey , Christian Catalini , Alex Moehring, and Catherine TuckerConditionally Accepted at Management Information Systems Quarterly, 2025
‘Notice and Choice’ has been a mainstay of policies designed to safeguard consumer privacy. This paper investigates distortions in consumer behavior when faced with notice and choice which may limit the ability of consumers to safeguard their privacy. We use data derived from a field experiment at MIT that distributed a new product, Bitcoin, to all undergraduates. There are two primary findings. First, small navigation costs have a tangible effect on how privacy-protective consumers’ choices are, often in sharp contrast with individual stated preferences about privacy. Second, the introduction of irrelevant, but reassuring, information about privacy protection makes consumers less likely to avoid surveillance, regardless of their stated preferences toward privacy.
- Information frictions and heterogeneity in valuations of personal dataAvinash Collis , Alex Moehring, Ananya Sen , and Alessandro AcquistiRevise and Resubmit at Information Systems Research, 2024
We investigate how consumer valuations of personal data are affected by real world information interventions. Proposals to compensate users for the information they disclose to online services have been advanced in both research and policy circles. These proposals may be hampered by information frictions that limit consumers’ ability to assess the value of their own data. We use an incentive-compatible mechanism to capture consumers’ willingness to share their social media data for monetary compensation and estimate distributions of valuations of social media data, before and after an information treatment. We find evidence of significant dispersion and heterogeneity in valuations before the information intervention, with women, Black, and low income individuals reporting systematically lower valuations than other groups. In both samples, the provision of information leads to a reduction in dispersion in data valuations. The reduction takes the form of increasing valuations by low-valuation individuals—in which women, low income, and Black users are over-represented. These participants with low valuations and high probability of revisions belong to groups that are traditionally associated with low levels of digital literacy. The findings suggest that strategies aimed at increasing information availability in markets for personal data may affect consumer welfare gains from data markets.
- News feeds and user engagement: Evidence from the reddit news tabAlex Moehring2022
We study how the introduction of a new non-personalized news feed impacts user engagement quantity, quality, and diversity on Reddit. In June 2018, Reddit introduced the News tab on iOS devices that surfaces popular content from a curated list of news-related communities. We leverage this natural experiment to identify the causal effects of the News tab on iOS user engagement in a difference-in-differences design. We find that the News tab increases the share of iOS devices that engage with news-related content and the new engagement is not meaningfully different in quality from existing engagement. Additionally, we find that the diversity of engagement within news categories and within articles from publishers across the political spectrum increases as a result of the News tab. These results suggest that non-personalized feeds can be an important tool to mitigate algorithmic filter bubbles.
Published Articles
- Heterogeneity and predictors of the effects of AI assistance on radiologistsFeiyang Yu , Alex Moehring, Oishi Banerjee , Tobias Salz , and 2 more authorsNature Medicine, 2024
The integration of artificial intelligence (AI) in medical image interpretation requires effective collaboration between clinicians and AI algorithms. Although previous studies demonstrated the potential of AI assistance in improving overall clinician performance, the individual impact on clinicians remains unclear. This large-scale study examined the heterogeneous effects of AI assistance on 140 radiologists across 15 chest X-ray diagnostic tasks and identified predictors of these effects. Surprisingly, conventional experience-based factors, such as years of experience, subspecialty and familiarity with AI tools, fail to reliably predict the impact of AI assistance. Additionally, lower-performing radiologists do not consistently benefit more from AI assistance, challenging prevailing assumptions. Instead, we found that the occurrence of AI errors strongly influences treatment outcomes, with inaccurate AI predictions adversely affecting radiologist performance on the aggregate of all pathologies and on half of the individual pathologies investigated. Our findings highlight the importance of personalized approaches to clinician–AI collaboration and the importance of accurate AI models. By understanding the factors that shape the effectiveness of AI assistance, this study provides valuable insights for targeted implementation of AI, enabling maximum benefits for individual clinicians in clinical practice.
- Comparative Advantage of Humans versus AI in the Long TailNikhil Agarwal , Ray Huang , Alex Moehring, Pranav Rajpurkar , and 2 more authorsAEA Papers and Proceedings, 2024
Machine learning algorithms now exceed human performance on several predictive tasks, generating concerns about widespread job displacement. However, supervised learning approaches rely on large amounts of high-quality labeled data and are designed for specific predictive tasks. Thus, humans may be required for a large number of tasks, each of which is not commonly encountered—the long tail—because humans can make predictions for a broader range of outcomes and with exposure to much less data. We show that a self-supervised algorithm for chest X-rays, which does not require specifically annotated disease labels, closes this gap even in the long tail of diseases.
- Providing normative information increases intentions to accept a COVID-19 vaccineAlex Moehring, Avinash Collis , Kiran Garimella , M Amin Rahimian , and 2 more authorsNature Communications, 2023
Despite the availability of multiple safe vaccines, vaccine hesitancy may present a challenge to successful control of the COVID-19 pandemic. As with many human behaviors, people’s vaccine acceptance may be affected by their beliefs about whether others will accept a vaccine (i.e., descriptive norms). However, information about these descriptive norms may have different effects depending on the actual descriptive norm, people’s baseline beliefs, and the relative importance of conformity, social learning, and free-riding. Here, using a pre-registered, randomized experiment (N = 484,239) embedded in an international survey (23 countries), we show that accurate information about descriptive norms can increase intentions to accept a vaccine for COVID-19. We find mixed evidence that information on descriptive norms impacts mask wearing intentions and no statistically significant evidence that it impacts intentions to physically distance. The effects on vaccination intentions are largely consistent across the 23 included countries, but are concentrated among people who were otherwise uncertain about accepting a vaccine. Providing normative information in vaccine communications partially corrects individuals’ underestimation of how many other people will accept a vaccine. These results suggest that presenting people with information about the widespread and growing acceptance of COVID-19 vaccines helps to increase vaccination intentions.
- Global survey on COVID-19 beliefs, behaviours and normsAvinash Collis , Kiran Garimella , Alex Moehring, M Amin Rahimian , and 6 more authorsNature Human Behaviour, 2022
Policy and communication responses to COVID-19 can benefit from better understanding of people’s baseline and resulting beliefs, behaviours and norms. From July 2020 to March 2021, we fielded a global survey on these topics in 67 countries yielding over 2 million responses. This paper provides an overview of the motivation behind the survey design, details the sampling and weighting designed to make the results representative of populations of interest and presents some insights learned from the survey. Several studies have already used the survey data to analyse risk perception, attitudes towards mask wearing and other preventive behaviours, as well as trust in information sources across communities worldwide. This resource can open new areas of enquiry in public health, communication and economic policy by leveraging large-scale, rich survey datasets on beliefs, behaviours and norms during a global pandemic.
- Interdependence and the cost of uncoordinated responses to COVID-19David Holtz , Michael Zhao , Seth G Benzell , Cathy Y Cao , and 7 more authorsProceedings of the National Academy of Sciences, 2020
Social distancing is the core policy response to coronavirus disease 2019 (COVID-19). But, as federal, state and local governments begin opening businesses and relaxing shelter-in-place orders worldwide, we lack quantitative evidence on how policies in one region affect mobility and social distancing in other regions and the consequences of uncoordinated regional policies adopted in the presence of such spillovers. To investigate this concern, we combined daily, county-level data on shelter-in-place policies with movement data from over 27 million mobile devices, social network connections among over 220 million Facebook users, daily temperature and precipitation data from 62,000 weather stations, and county-level census data on population demographics to estimate the geographic and social network spillovers created by regional policies across the United States. Our analysis shows that the contact patterns of people in a given region are significantly influenced by the policies and behaviors of people in other, sometimes distant, regions. When just one-third of a state’s social and geographic peer states adopt shelter-in-place policies, it creates a reduction in mobility equal to the state’s own policy decisions. These spillovers are mediated by peer travel and distancing behaviors in those states. A simple analytical model calibrated with our empirical estimates demonstrated that the “loss from anarchy” in uncoordinated state policies is increasing in the number of noncooperating states and the size of social and geographic spillovers. These results suggest a substantial cost of uncoordinated government responses to COVID-19 when people, ideas, and media move across borders.