Is web scraping legal? What US case law says (and a note on Mexico)

The short answer

Web scraping is a technique, not a crime. Whether a specific scraping project is lawful depends on what you collect, how you access it and what you do with the data. In the US, the main questions are the Computer Fraud and Abuse Act (CFAA), contract law through a website's terms of use, copyright, and privacy rules for personal data.

The rough rule that comes out of recent case law: scraping pages that anyone can see without logging in is unlikely to violate the CFAA, but it can still create civil liability, especially if you agreed to terms that prohibit it, used fake accounts, or collected personal data.

I am a developer, not a lawyer. This guide explains the landscape so you can ask better questions; for a real project with legal risk, talk to an attorney in the relevant jurisdiction.

The CFAA and the Van Buren decision

The CFAA, codified at 18 U.S.C. § 1030, punishes accessing a computer "without authorization" or "exceeding authorized access." For years, companies argued that scraping after a cease-and-desist letter, or against a site's terms, was unauthorized access under this law.

In Van Buren v. United States (2021), the Supreme Court narrowed the "exceeds authorized access" clause. It held that the clause covers people who access parts of a computer system, such as files or databases, that are off-limits to them, not people who misuse information they are otherwise entitled to access. The approach is often described as a "gates-up-or-down" inquiry: the question is whether you passed a gate you were not allowed through, not why you looked.

hiQ Labs v. LinkedIn: what it decided and what it did not

hiQ Labs scraped public LinkedIn profiles to build analytics products. LinkedIn sent a cease-and-desist letter claiming further scraping would violate the CFAA, and hiQ sued to keep its access.

After Van Buren, the Ninth Circuit reconsidered the case and, in April 2022, affirmed a preliminary injunction in hiQ's favor. Its reasoning: public websites have no gate to lift or lower, so accessing publicly available profiles likely is not access "without authorization" under the CFAA.

That is a strong signal, but it was a preliminary-injunction ruling, not a final judgment on the merits, and it binds courts only within the Ninth Circuit. And the story did not end there. Later in 2022, the district court found hiQ had breached LinkedIn's user agreement, including through fake accounts used to access the platform. In December 2022, the parties agreed to a consent judgment with a $500,000 judgment against hiQ and a permanent injunction requiring hiQ to stop scraping and delete the data.

The practical lesson: the CFAA is a weaker weapon against scraping public data than it used to be, but contract and tort claims are very much alive.

Where scraping gets risky

Most scraping disputes involve one or more of these situations:

• Logging in to scrape. Accessing data behind a login, especially with fake or shared accounts, moves you closer to the CFAA's gates and clearly into contract territory.
• Bypassing technical barriers. Circumventing CAPTCHAs, IP blocks or paywalls weakens any argument that you were authorized.
• Accepting terms that prohibit scraping. If you agreed to a site's terms (for example, by creating an account), breaching them can create liability.
• Copying expressive content. Facts are generally not protected by copyright, but articles, photos and creative descriptions are. Republishing them is a different matter from analyzing them.
• Collecting personal data. Names, emails, phone numbers or profiles trigger privacy laws such as the CCPA in California, the GDPR in Europe or Mexico's data protection law.
• Overloading servers. Aggressive crawling that degrades a site can support trespass-type claims, and it is simply bad practice.

What about robots.txt?

robots.txt is a file where a site tells crawlers which paths it prefers they avoid. RFC 9309, which standardized the Robots Exclusion Protocol, explicitly states that its rules are not a form of access authorization. In other words, robots.txt is a request, not a lock.

Ignoring it is not automatically illegal, but respecting it is the professional default. It shows good faith, reduces the chance of being blocked and is often the first thing a court or a site owner will ask about.

A note on Mexico

Because I am based in Mexico, clients often ask about Mexican law too. Mexico has no statute that prohibits web scraping in general. The relevant limits come from the federal law on the protection of personal data held by private parties (LFPDPPP), whose current version was published in March 2025, copyright law and the federal criminal code's provisions on unauthorized access to protected systems.

The practical rules are the same as in the US: collect public, non-personal data; do not break access controls; respect terms and robots.txt; and do not republish protected content as your own.

Best practices for a responsible scraping project

These are the rules I apply before writing a single line of a scraper:

• Check for an official API or data export first; if one exists, use it.
• Read the site's terms of use and robots.txt, and document the decision.
• Collect only public data you actually need, and avoid personal data unless you have a lawful basis and a plan to protect it.
• Never log in with fake accounts or bypass CAPTCHAs, paywalls or blocks.
• Rate-limit requests and identify your crawler; scrape during off-peak hours.
• Store facts and analysis, not copies of articles or photos, unless you have permission.
• Get legal advice when the data is sensitive, the volume is large or the target has objected.

Alternatives to scraping

Scraping is often the second-best option. Before building a scraper, check whether the data is available through an official API, a public dataset, a paid data provider or a partnership with the site owner. APIs are more stable, they come with clear terms, and they do not break every time the site redesigns its pages.

When no alternative exists and the data is public, a carefully built scraper is a reasonable tool. When an alternative exists, the extra cost is usually worth the legal clarity and the lower maintenance.

How I approach scraping work

I build data extraction pipelines in Python, including NLP Pipeline, a personal project that extracts, normalizes and translates web content in batches. For client work I start with a feasibility review of each source: whether the data is public, what the terms and robots.txt allow, and whether an official API would be better. Sources that prohibit extraction are discarded, not worked around.