The Impact of LLM Adoption on Online User Behavior

Discover the impact of Large Language Models on user search habits and online content consumption

These days, I don’t use Google as often as I used to. If ChatGPT doesn’t give me a compelling answer, then I search for keywords on Google. Many people are doing the same—relying heavily on AI tools like ChatGPT and turning less frequently to traditional search engines. But how exactly do LLMs change online user behavior? And what does this shift mean for advertising revenue, which has long been the backbone of the internet? Here's the answer.

Paper reviewed: Padilla, Nicolas and Lam, H. Tai and Lambrecht, Anja and Hollenbeck, Brett, The Impact of LLM Adoption on Online User Behavior (August 15, 2025). Available at SSRN: https://ssrn.com/abstract=5393256 or http://dx.doi.org/10.2139/ssrn.5393256

Summary

A recent study reveals that the adoption of Large Language Models (LLMs) significantly alters online user behavior, leading to a substantial decrease in search activity and website traffic, particularly affecting smaller sites and education-related content. This shift has implications for online content providers and advertisers.

Key Findings

The adoption of Large Language Models (LLMs) significantly alters users' online behavior, leading to a decrease in online search activity, with a drop of more than 20% in search volume after adoption.
The reduction in search activity is not immediate but rather a gradual process, suggesting that users learn to substitute traditional search with LLMs over time.
LLM adoption affects website traffic, with smaller websites experiencing a significant drop in visits, while frequently visited websites remain largely unaffected.
The decrease in website traffic to smaller sites translates into a reduction in display ad exposures, particularly for users with high retail activity, potentially impacting the revenue of online content providers.
Education-related websites and certain user-generated content platforms, like Stack Overflow, see a decline in visits following LLM adoption, indicating that LLMs can substitute for some types of online content consumption.

Implications

Business and Policy Implications

Online content providers, especially those relying on advertising or subscriptions, may need to adapt their business models to mitigate the impact of LLM adoption on their traffic and revenue.
The shift towards LLMs could lead to a reevaluation of how content is monetized, potentially through new models such as charging LLMs for access to content.
GenAI firms and policymakers must consider the implications of LLMs on content creation and the digital economy, ensuring that creators are incentivized to produce high-quality content.
The findings have implications for copyright and fair use policy, as LLMs may substitute for original content, potentially affecting the market for copyrighted works.

Introduction

The rise of AI tools, particularly Large Language Models (LLMs), is transforming how users interact with online content. LLMs have the potential to either substitute or complement traditional digital resources, significantly impacting the online ecosystem. This research examines the effect of LLM adoption on users' online behavior, focusing on search activity, website traffic, and advertising exposure.

Background and Context

The development and adoption of LLMs have been rapid, with significant advancements in their ability to process and generate human-like language. This has led to concerns among content creators and providers about the potential disruption to their business models. Previous research has explored the impact of emerging technologies on media and information consumption, highlighting the complex dynamics between new tools and existing behaviors.

The paper is structured as follows: after introducing the research question and context, it reviews related literature on the impact of emerging technologies and LLMs. The empirical setting and data are then described, followed by the estimation and identification strategy. The results are presented, focusing on the effects of LLM adoption on online search, browsing activity, and advertising exposure, as well as specific domain types like education-related websites and user-generated content platforms.

The analysis is based on a comprehensive panel dataset tracking users' online browsing behavior over two years. The data allows for the identification of individual LLM adoption and its impact on various online activities. The estimation approach employs a staggered difference-in-difference methodology to account for heterogeneous treatment effects across cohorts and time.

The findings of this research have significant implications for online content providers, GenAI firms, and policymakers. As LLMs continue to evolve and become more integrated into the online ecosystem, understanding their impact on user behavior is crucial for adapting business models and ensuring the continued creation of high-quality online content.

The remainder of this part will delve into the background and context of LLM adoption, setting the stage for a detailed examination of its effects on online user behavior.

The adoption of LLMs is not a uniform process; it varies across different user groups and is influenced by factors such as the type of content being sought and the specific use cases for LLMs. For instance, educational tasks can be completed more efficiently using LLMs, potentially substituting for visits to education-related websites. Similarly, LLMs can provide answers to complex queries, potentially reducing the need to visit knowledge-sharing platforms.

As LLMs continue to evolve, their impact on the online ecosystem is likely to grow. The ability of LLMs to provide direct answers to user queries, rather than directing users to other websites, has significant implications for content creators and advertisers. The reduction in website traffic and advertising exposure following LLM adoption suggests that these models can disrupt traditional online business models.

The impact of LLM adoption on online user behavior is multifaceted, affecting various aspects of the digital landscape. By examining the changes in search activity, website traffic, and advertising exposure, this research provides insights into the broader implications of LLM adoption for the online ecosystem.

To fully understand the effects of LLM adoption, it is essential to consider the heterogeneity in user behavior and the varying impacts across different types of online content. The following sections will explore these aspects in more detail, providing a comprehensive analysis of the impact of LLMs on online user behavior.

The data used in this analysis comes from the Comscore Web-Behavior Panel dataset, which tracks detailed URL-level data for desktop browsing. The sample is restricted to users who adopted LLMs during the observation period, allowing for a comparison between pre-adoption and post-adoption behavior.

The estimation strategy relies on a staggered difference-in-difference approach, which accounts for the varying timing of LLM adoption across users. This methodology allows for the estimation of cohort and time-specific average treatment effects on the treated (ATT), providing a nuanced understanding of how LLM adoption affects online behavior.

By examining the effects of LLM adoption on various aspects of online user behavior, this research contributes to a deeper understanding of the implications of emerging technologies for the digital economy. The findings have significant implications for businesses, policymakers, and content creators, highlighting the need for adaptive strategies in response to the evolving online landscape.

The analysis presented in this paper is based on data from 2022 and 2023, a period during which LLMs were still in the early stages of adoption. As LLM technology continues to advance and adoption becomes more widespread, the impacts identified in this research are likely to evolve. Future studies will be necessary to fully capture the long-term effects of LLM adoption on online user behavior and the broader digital ecosystem.

In conclusion, the adoption of LLMs is reshaping the online landscape, with significant implications for content creators, advertisers, and the broader digital economy. Understanding these changes is crucial for developing strategies that mitigate potential negative impacts and capitalize on new opportunities. As the online ecosystem continues to evolve, ongoing research will be essential for navigating the challenges and opportunities presented by emerging technologies like LLMs.

Main Results

The study examines the impact of Large Language Model (LLM) adoption on online user behavior, focusing on search activity, website traffic, and advertising exposure. The main findings are presented below.

Search Activity

The adoption of LLMs leads to a significant decrease in online search activity. The drop in searches is gradual, suggesting a learning period during which users adapt to using LLMs. Eventually, the level of online search drops by more than 20% compared to the pre-adoption period.

The decrease in search activity is observed for complex queries, such as those using question words (e.g., how, what, where).
Searches with navigational terms only (e.g., 'reddit') do not drop significantly, likely because LLMs did not provide outlinks during the study period.
Both short and long searches decrease, indicating that LLMs substitute for traditional search engines across various query lengths.

Website Traffic

LLM adoption affects website traffic, particularly for smaller websites.

The total number of URL calls to all websites does not show a significant decline.
However, visits to smaller websites (not in the top 500 or 1000) drop significantly, suggesting that LLMs serve as a substitute for these sites.
When stratifying websites by traffic quartiles, the bottom quartile (smallest websites) experiences a significant drop in traffic.

Advertising Exposure

The reduction in website visits due to LLM adoption has implications for advertising exposure.

Overall, ad exposures drop significantly, particularly for Google display ads.
The drop in ad exposures is more pronounced for users with high retail activity, who are likely valuable targets for advertisers.
There is no significant effect on search ads, possibly because LLMs are less likely to substitute for navigational searches where ads are more likely to be displayed.

Methodology Insights

The study employs a staggered difference-in-difference estimation approach, comparing users who have adopted LLMs to those who have not yet adopted. This methodology allows for the estimation of causal effects while controlling for user-specific and time-specific factors.

The use of not-yet-treated users as controls helps to address potential biases in the estimation of treatment effects.
The study controls for users' baseline internet activity using their activity levels in email, retail, and news, which are unaffected by LLM adoption.

Analysis and Interpretation

The findings suggest that LLMs are substituting for traditional online activities, particularly for complex searches and visits to smaller websites. This substitution has significant implications for online content providers and advertisers.

The drop in advertising exposure, especially for display ads, threatens the revenue models of content providers who rely on advertising.
The effect on education-related websites is particularly pronounced, with significant drops in traffic to these sites, especially those monetizing through subscriptions or B2B sales.
User-generated content platforms are affected heterogeneously, with Stack Overflow experiencing a decline in visits, while Wikipedia and Reddit remain unaffected.
Social media usage does not show a significant change following LLM adoption, suggesting that LLMs do not effectively substitute for human social interaction.

The study's results have important implications for online content providers, GenAI firms, and policymakers. As LLM technology continues to evolve, understanding its impact on online behavior will be crucial for developing strategies to mitigate potential negative impacts and capitalize on new opportunities.

Practical Business Insights and Applications

Alternative Revenue Models: Online content providers may need to explore alternative revenue models, such as charging LLMs for access to content or adopting paywalls and freemium models.
Content Creation Incentives: The substitution effect of LLMs may disincentivize content creation if providers are not adequately remunerated for their content.
Fair Use and Copyright Policy: The study's findings have implications for fair use under U.S. copyright law, particularly regarding the fourth factor, which assesses the impact of use on the market for the original work.

Strategic Implications for Companies and Managers

Adapting to Changing User Behavior: Companies must adapt their strategies to the changing online landscape, where users are increasingly relying on LLMs for information.
Diversifying Revenue Streams: Businesses should consider diversifying their revenue streams to mitigate the potential negative impacts of LLM adoption on advertising revenue.
Investing in AI Technologies: Companies may need to invest in AI technologies, including LLMs, to remain competitive and provide value to their users.

Real-World Implementation Considerations

Monitoring LLM Adoption: Businesses should continuously monitor LLM adoption and its impact on their specific industry or market.
Assessing Content Value: Content providers need to assess the value of their content in the context of LLMs and consider how to maintain or increase this value.
Developing AI-Compatible Business Models: Companies should explore business models that are compatible with AI technologies, such as LLMs.

Competitive Advantages and Market Opportunities

Early Adoption: Early adopters of LLMs may gain a competitive advantage by understanding how to effectively integrate these technologies into their operations.
New Content Opportunities: LLMs may create new opportunities for content creation, such as generating high-quality content that complements human-created content.
AI-Driven Innovation: The adoption of LLMs can drive innovation, enabling businesses to develop new products, services, or experiences that leverage the capabilities of these models.

Actionable Recommendations for Business Leaders

Stay Informed: Business leaders should stay informed about the latest developments in LLM technology and its adoption.
Assess Business Impact: Leaders should assess the potential impact of LLMs on their business, including effects on revenue models and user behavior.
Develop Strategic Responses: Businesses should develop strategic responses to the challenges and opportunities presented by LLMs, including exploring new revenue models and investing in AI technologies.

Practical Implications

The study on the impact of LLM adoption on online user behavior has significant practical implications for businesses, managers, and policymakers. The findings suggest that LLMs are substituting for traditional online activities, particularly for informational queries and smaller websites.

Real-World Applications

The adoption of LLMs is changing how users interact with online content, with a significant decrease in online search activity and website traffic, especially for smaller websites.
The reduction in website traffic and ad exposures has implications for online content creators and advertisers, who may need to adapt their revenue models.

Strategic Implications

Businesses and managers need to understand the impact of LLMs on their online presence and revenue streams.
The shift towards LLMs may require content creators to explore alternative revenue models, such as charging LLMs for access to content or developing new products and services that leverage LLM capabilities.

Who Should Care

Online content creators and publishers, as they may need to adapt to changing user behavior and revenue models.
Advertisers and marketers, as the reduction in ad exposures may impact their advertising strategies.
Policymakers, as the shift towards LLMs raises questions about the future of online content creation and the need for regulatory frameworks to ensure fair compensation for content creators.

Actionable Recommendations

To navigate the changing landscape driven by LLM adoption, businesses and managers can take the following actions:

Monitor LLM Adoption and Usage: Continuously track the adoption and usage of LLMs among your target audience to understand the impact on your business.
Diversify Revenue Streams: Explore alternative revenue models, such as subscription-based services or sponsored content, to reduce dependence on advertising revenue.
Invest in AI Technologies: Consider investing in AI technologies, including LLMs, to enhance your products and services and stay competitive.
Optimize Content for LLMs: Optimize your content to be LLM-friendly, ensuring that it is easily discoverable and accessible through these models.
Develop Strategic Partnerships: Collaborate with LLM providers or other businesses to develop new products and services that leverage LLM capabilities.

Implementation Considerations

When implementing these recommendations, consider the following:

Assess Your Business Model: Evaluate your current business model and identify areas that may be impacted by LLM adoption.
Invest in Data Analysis: Invest in data analysis capabilities to track user behavior and understand the impact of LLMs on your business.
Develop a Strategic Roadmap: Develop a strategic roadmap that outlines your plans for adapting to the changing landscape driven by LLM adoption.

Conclusion

The adoption of LLMs is transforming the online landscape, with significant implications for businesses, managers, and policymakers. By understanding the impact of LLMs on user behavior and adapting their strategies accordingly, businesses can stay competitive and thrive in this new environment.

Main Takeaways

LLMs are substituting for traditional online activities, particularly for informational queries and smaller websites.
The reduction in website traffic and ad exposures has implications for online content creators and advertisers.
Businesses need to adapt their revenue models and strategies to the changing landscape driven by LLM adoption.

Final Thoughts

The rise of LLMs presents both challenges and opportunities for businesses and policymakers. By staying informed, assessing the business impact, and developing strategic responses, businesses can navigate this changing landscape and thrive in the future. As the adoption of LLMs continues to grow, it is essential to prioritize innovation, adaptability, and strategic planning to remain competitive.