Web Scraping as a Service: How to Sell Public Web Data Legally (and Price It Above $500)

Businesses pay for data that’s already public. A distributor wants every hardware store in three provinces with a website and phone number. An online shop wants competitor prices every morning. A consultant wants 2,000 product listings in a spreadsheet so they can write a market report. All of it is on public web pages, and none of them want to copy it by hand.

That’s the job: you collect public web data, clean it, and deliver it as a spreadsheet, a feed or a report. It’s a grey area, not a black market. It’s legal in most cases, but there are a few clear lines, and the people who cross them get sued or fined.

This guide covers:

• what the hiQ Labs v. LinkedIn case actually decided (and what it didn’t)
• the difference between grey and illegal
• a tested Python scraper (requests + BeautifulSoup) and what one run really costs
• what freelancers charge, and how to package the work so it can sell for $500 or more instead of $50

This is not legal advice. Laws differ by country. If a client’s project involves personal data or a site that has told you to stop, talk to a lawyer before you start.

The legal case everyone quotes: hiQ Labs v. LinkedIn

hiQ Labs was a startup that scraped public LinkedIn profiles and sold “flight risk” analytics to employers. In 2017 LinkedIn sent a cease-and-desist letter and said further scraping would break the Computer Fraud and Abuse Act (CFAA), the US anti-hacking law. hiQ sued first and asked for an injunction.

What happened next:

1. 2017: The district court ordered LinkedIn to stop blocking hiQ.
2. 2019: The Ninth Circuit agreed. Pages anyone can see without logging in likely aren’t accessed “without authorization” under the CFAA.
3. 2021: The Supreme Court sent the case back in light of Van Buren v. United States, which read the CFAA narrowly: what matters is whether a gate is up or down, not why you go through it.
4. 2022: The Ninth Circuit reached the same conclusion again. Scraping public pages is probably not hacking under the CFAA.
5. Late 2022: hiQ still lost. In November the district court found hiQ broke LinkedIn’s User Agreement, partly because it used fake accounts to see logged-in data. The case ended in December 2022 with a consent judgment: a permanent injunction, deletion of the scraped data and code, and a $500,000 judgment against hiQ. By then the company had basically shut down.

The lesson isn’t “scraping is legal.” It’s narrower: for public, logged-out pages, the hacking law is probably not your biggest risk in the US. Contract law (terms you agreed to by making an account), copyright, privacy law and the cost of a lawsuit still are. Meta’s case against Bright Data (summary judgment, January 2024) shows the same split: the court sided with Bright Data because it scraped logged-out public pages, so Meta’s user terms didn’t bind it there.

Grey vs. illegal: the lines that matter

Personal data is the big one. Under the EU’s GDPR, public personal data is still personal data. Clearview AI was fined €20 million each by regulators in Italy, Greece and France, and €30.5 million by the Dutch regulator in 2024, for scraping public photos. Poland’s regulator fined Bisnode about €220,000 for using public business-register data on people without telling them. Indonesia’s personal data protection law (UU PDP, Law 27/2022) has applied in full since October 2024. Even in a “business” lead list, a sole trader’s name and mobile number is personal data.

robots.txt is a risk filter, not a law. In most countries it isn’t legally binding on its own. But if a site says Disallow: /, scraping it anyway is the fastest way to get blocked, get a legal letter and give the other side a strong argument that you knew you weren’t welcome. Treat it as a rule for your business: if robots.txt says no, you don’t take the job, or you ask the site owner first.

What you need

You don’t need proxies to start. If a project only works with rotating residential IPs, that usually means the site is actively trying to block you, which is a warning sign, not a technical problem to solve.

Step 1: Set up the project

mkdir scraper && cd scraper
python3 -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install requests beautifulsoup4

Step 2: Check the target before you write any code

Do this for every client job, and save the answers in a project note:

1. Open https://site.com/robots.txt. Is your path disallowed? Is there a Crawl-delay?
2. Read the terms of service. Do they ban automated access? Do you have an account there? If you do, don’t scrape while logged in.
3. Is any of it personal data? If yes, either remove those fields or stop and get advice.
4. Is there an API or a download? Many directories and government registers offer CSV exports or APIs. Using one is cheaper, faster and safer.

For the example below we use books.toscrape.com, a sandbox site made for scraping practice. It’s set up like a real catalogue or directory: listing pages, 20 items per page, a “next” link and 50 pages in total.

Step 3: The scraper

Save this as scrape_directory.py:

import csv, sys, time
from datetime import datetime, timezone
from urllib.parse import urljoin, urlparse
from urllib.robotparser import RobotFileParser

import requests
from bs4 import BeautifulSoup

START_URL = "https://books.toscrape.com/catalogue/page-1.html"
OUT_FILE = "listings.csv"
DELAY = 2.0                      # seconds between requests
MAX_PAGES = int(sys.argv[1]) if len(sys.argv) > 1 else 100
USER_AGENT = "SolvincDataBot/1.0 (+mailto:you@yourdomain.com)"
VPS_PER_HOUR = 6 / 730           # $6/month VPS, ~730 hours in a month

RATINGS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}

session = requests.Session()
session.headers["User-Agent"] = USER_AGENT

def robots_check(url):
    root = "{0.scheme}://{0.netloc}".format(urlparse(url))
    rp = RobotFileParser(root + "/robots.txt")
    rp.read()
    return rp.can_fetch(USER_AGENT, url), rp.crawl_delay(USER_AGENT)

def fetch(url, tries=3):
    for attempt in range(tries):
        resp = session.get(url, timeout=20)
        if resp.status_code == 200:
            return resp
        if resp.status_code in (429, 503):       # the site is asking you to slow down
            time.sleep(30 * (attempt + 1))
            continue
        resp.raise_for_status()
    raise RuntimeError(f"Gave up on {url}")

def parse_page(html, page_url):
    soup = BeautifulSoup(html, "html.parser")
    rows = []
    for card in soup.select("article.product_pod"):
        link = card.select_one("h3 a")
        rows.append({
            "name": link["title"],
            "url": urljoin(page_url, link["href"]),
            "price_gbp": card.select_one(".price_color").get_text(strip=True).lstrip("£"),
            "rating": RATINGS.get(card.select_one("p.star-rating")["class"][1]),
            "in_stock": "In stock" in card.select_one(".availability").get_text(),
            "scraped_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        })
    nxt = soup.select_one("li.next a")
    return rows, (urljoin(page_url, nxt["href"]) if nxt else None)

def main():
    allowed, crawl_delay = robots_check(START_URL)
    if not allowed:
        sys.exit("robots.txt disallows this path. Stop and ask the site owner.")
    delay = max(DELAY, crawl_delay or 0)

    started, url, pages, all_rows, bytes_in = time.time(), START_URL, 0, [], 0
    while url and pages < MAX_PAGES:
        resp = fetch(url)
        bytes_in += len(resp.content)
        rows, url = parse_page(resp.content, resp.url)
        all_rows.extend(rows)
        pages += 1
        print(f"page {pages}: {len(rows)} rows (total {len(all_rows)})")
        if url:
            time.sleep(delay)

    with open(OUT_FILE, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=all_rows[0].keys())
        writer.writeheader()
        writer.writerows(all_rows)

    hours = (time.time() - started) / 3600
    print(f"\n{len(all_rows)} rows, {pages} requests, {bytes_in / 1e6:.1f} MB, "
          f"{hours * 60:.1f} min, compute cost ${hours * VPS_PER_HOUR:.4f}")

if __name__ == "__main__":
    main()

What it does, and why it matters when you sell the work:

• Checks robots.txt before the first request and uses the site’s Crawl-delay if it’s longer than yours.
• Identifies itself with a User-Agent that includes a contact address. If the site owner has a problem, they can email you instead of calling a lawyer.
• Backs off on 429 Too Many Requests and 503 instead of retrying right away.
• Passes raw bytes (resp.content) to BeautifulSoup so it detects the encoding itself. If you pass resp.text here, the £ sign turns into £.
• Adds scraped_at to every row. Clients need to know how fresh the data is.

Step 4: Run it and check the real cost

Test on 3 pages first:

python scrape_directory.py 3

Then run the full job:

python scrape_directory.py

Our run, October 16, 2026:

page 50: 20 rows (total 1000)

1000 rows, 50 requests, 2.5 MB, 2.2 min, compute cost $0.0003

1,000 rows for $0.0003 of server time. On your laptop it’s $0. The cost of a scraping job is almost never compute. It’s your time: finding selectors, handling edge cases, cleaning the data and fixing the scraper when the site changes.

To adapt the scraper to a real directory, change START_URL, the CSS selectors in parse_page() and the li.next a selector. Find them with right-click → Inspect on the page. If the data isn’t in the HTML from requests (view the page source: if it’s missing there, JavaScript loads it), check the browser’s Network tab for a JSON endpoint first. Use Playwright only if there isn’t one.

Step 5: Clean before you deliver

Raw scraped data isn’t the product. Clean data is. Before delivery:

• remove duplicates (same URL or same name + address)
• standardize formats: prices as numbers, phone numbers in one format, one date format
• count blank fields per column and tell the client (“phone missing for 14% of rows”)
• remove any field you decided not to collect, such as personal names or personal emails

A short “data notes” file with the source, the date, the row count and known gaps makes a $50 CSV look like a $500 deliverable.

What the market pays

Search “web scraping” on Upwork and you’ll see a wide range. These are typical patterns, not official statistics, so check current listings yourself:

• Small one-off jobs (“scrape this site to Excel”): often $30–$200, with many bids from low-cost freelancers.
• Hourly scraping and data-extraction freelancers: commonly listed at $15–$40/hour for beginners and $50–$100+/hour for experienced developers with job history.
• Recurring data feeds and custom pipelines: the $500–$5,000 projects are here: multi-site monitoring, scheduled deliveries, cleaning and integration with the client’s sheet or database.

Platform costs: Since May 2025, Upwork’s freelancer service fee has been variable, 0–15% per contract (shown before you submit a proposal), and proposals cost Connects, which sell for $0.15 each. Most jobs take several Connects per proposal. Fiverr takes 20% of every order.

The pattern is clear: “scrape a site” is a commodity. “Get this business a reliable data feed it can act on” isn’t.

How to sell this as a service

Don’t sell scraping. Sell the result. Three packages that work:

1. B2B lead lists from public business directories Example: “All 640 furniture manufacturers in Central Java with a website, business phone and product categories.” Only company-level data, no personal names or private emails. Suggested pricing: $300–$1,500 per list, depending on size and how much checking is needed. Each refresh is extra.

2. Price and stock monitoring Example: “Daily prices and stock status for your 200 SKUs across 4 competitor stores, delivered to a Google Sheet, with an alert when someone undercuts you by 10%.” (We built a basic version in our competitor price tracker guide.) Suggested pricing: $300–$800 setup plus $150–$500/month. This is the best package because it pays every month.

3. Market research datasets and reports Example: “Every listing in a product category on 3 marketplaces, with price ranges, top sellers and gaps, in a 10-page summary.” Clients are consultants, investors and new market entrants. Suggested pricing: $800–$5,000 per report. The analysis is what they pay for, not the rows.

Example economics (illustrative, not market data): one monitoring client at $300/month takes about 2 hours a month to maintain once it runs. Five clients is $1,500/month for about 10 hours, plus the occasional evening fixing a broken selector. Getting to five clients can take months of outreach and proposals, and many freelancers never get there.

Where to find buyers: Upwork and Fiverr for your first reviews, then direct outreach. Send an online shop a free sample: their 20 best-selling products and the competitors’ prices today. A sample with their own data beats any pitch.

Put the legal checks in your offer. Add a line to every proposal: “Public, logged-out pages only. No personal data. I follow robots.txt and rate limits.” Serious clients like it, and it keeps away the clients you don’t want.

Reality check

Scraping behind a login is where people get hurt. hiQ won on the hacking law and still lost the case, partly over fake accounts. Logging in means you agreed to terms. Buying or creating accounts to scrape LinkedIn, Instagram or Facebook is how a “grey” project becomes a lawsuit. Turn these jobs down.

Selling personal data is the illegal part. “Scrape 10,000 Instagram profiles with emails” or “get me phone numbers of every doctor in Jakarta” is where GDPR, UU PDP and similar laws apply, and “it was public” doesn’t protect you. The Clearview fines happened because the data was public.

Clients ask for the risky job, not the safe one. Many scraping posts on Upwork are for social media, login-walled data or contact details. Plenty of freelancers take them because they pay. That’s the clearest sign of where the risk is.

Scrapers break. Sites change their HTML, add bot protection or move to JavaScript rendering. Monthly pricing has to cover maintenance, and your contract should say what happens when a source goes offline or blocks you.

Anti-bot systems are a signal. Cloudflare challenges, CAPTCHAs and aggressive blocking mean the site doesn’t want you there. Getting around them with CAPTCHA-solving services moves you from “public data” to “getting past technical barriers,” which is a much worse legal position.

Copyright still applies. Prices, addresses and specs are facts. Product descriptions, photos, reviews and articles usually aren’t. Collecting them for analysis is different from republishing them on the client’s site.

Don’t compete on price. Someone will always do “scrape to Excel” for $20. Your edge is reliability, clean output and knowing which jobs not to take.

Final checklist

• robots.txt and terms checked, notes saved for each project
• Public, logged-out pages only, no accounts
• No personal data, or written legal advice if there is
• User-Agent with a contact address, 2+ seconds between requests
• Tested on 3 pages before the full run
• Data cleaned, deduplicated and delivered with a “data notes” file
• Monthly price covers maintenance when the site changes
• You stop right away if the site owner asks

Start with one package, one niche and one free sample. A price-monitoring client in month one is worth more than ten $30 “scrape this site” gigs.