Data

Web scraping

Web scraping is reading a web page with a program and pulling structured data out of it. It is how nearly all business contact data is collected, including by vendors who avoid the word.

A scraper fetches a page exactly as a browser does, then extracts the parts it was told to look for — a name in a heading, an address in a footer, a mailto link on a contact page. No privileged access is involved and nothing is broken into; the content is what any visitor would see.

The legal position is more nuanced than either side of the argument online admits, and it turns on which question you are asking: copyright in the content, computer-misuse law, the site's terms of service, and — separately and importantly — data protection law when the data is personal.

The practical constraints are rate limits and robots directives. Fetching politely, identifying yourself honestly, and staying inside what a site declares it will tolerate is both the cheap way to avoid being blocked and the version of this that is defensible.

Goes deeper: Is Scraping Google Maps Legal?