SOLUTION · WEB SCRAPING AND DATA EXTRACTION
Competitor price monitoring with web scraping
Competitor price monitoring means automatically and regularly collecting public prices from other websites, matching them to your products and alerting you when they change. I build Python scrapers that do this with public data, store the history and deliver a report or dashboard you can act on.
What’s happening today
Pricing without knowing how the market moves is guessing. In many companies, someone is assigned to check competitors' websites — online stores, marketplaces, distributor catalogs — and copy prices into a spreadsheet. It's a repetitive task done with enthusiasm the first week and less every week after, until it only happens when a salesperson complains that "they're cheaper over there".
With a large catalog, manual checks leave gaps: products nobody reviewed, prices that are days old, copy errors or comparisons between different pack sizes (a quart against 30 ounces, a 12-pack against a 24-pack). A pricing decision made with that data can cost margin or sales without anyone noticing.
Without history, you can't see patterns either: does the competitor cut prices every other week? Do their promotions last three days or three weeks? Do they raise prices before Black Friday only to "discount" them? That information is worth more than today's price, and it only exists if it's recorded systematically.
Scraping automates the collection, but it has to be done responsibly: public data only, no logging in or bypassing access controls, respecting terms of use and robots.txt, and a frequency that doesn't overload the sites being checked. Good monitoring is quiet, stable and easy to maintain.
How I’d solve it
- 01
Define sources, products and matches
We choose which sites to monitor, review their terms and robots.txt, and build a matching table between their products and yours (brand, model, size). That table is the key: without it you're comparing apples and oranges. For large catalogs, matches can be proposed automatically and reviewed by hand.
- 02
Robust, respectful collection
I build each scraper with rate limits, retries and detection of structural changes on every site. If a page changes its design and the scraper starts returning empty or odd values, the system alerts you instead of silently storing garbage.
- 03
Normalization and history
Prices are cleaned and normalized (currency, tax included or not, unit price, promotions) and stored with date and time. You can analyze trends, not just the latest value, and compare true unit prices across pack sizes.
- 04
Report, dashboard and alerts
You get a periodic Excel report or a dashboard with the relevant changes, the gap against your own price and alerts when a competitor moves more than the threshold you set. The goal is to see what matters in two minutes, not a table with thousands of rows.
- 05
Planned maintenance
Websites change, and an unmaintained scraper breaks sooner or later. From day one we define how failures are detected and fixed, so monitoring doesn't become an abandoned tool.
Example weekly flow for an auto parts store
- Every Monday at 6 a.m., the scraper checks three competitors' public product pages.
- It extracts name, part number, pack size, price and availability.
- It matches each product to your catalog using the matching table.
- It normalizes prices and stores the week's history.
- It calculates the gap against your price and flags products outside your threshold.
- At 8 a.m. you receive a report with the relevant changes and a link to the full detail.
Related work
Go deeper
Frequently asked questions
Is scraping competitor prices legal?
Collecting publicly displayed prices is a common practice in retail, but it isn't a blank check. You need to review each site's terms of use, respect robots.txt, avoid bypassing logins or blocks, and stay away from personal data. Rules also vary by country. I assess each source before building anything and drop those that expressly prohibit it. If your case has specific legal implications, check with a lawyer.
Why not use an off-the-shelf price monitoring tool?
If your competitors sell on large sites that a commercial platform already covers, that may be faster and cheaper. A custom build makes sense when your sources are smaller sites or distributor catalogs those platforms don't cover, when product matching is complex, or when you want the data inside your own system without paying per tracked product.
What happens if a competitor's site changes its design?
That site's scraper may stop extracting data correctly. That's why I include validations that catch missing data, out-of-range prices or pages that no longer have the expected structure, and alert immediately. Fixing it is usually quick if the system is well built, which is why maintenance should be planned.
How often should prices be monitored?
It depends on how fast prices move in your market. For most businesses, once a day or once a week is enough. Higher frequency means more load on the sites you check, more risk of being blocked and higher infrastructure costs, without necessarily leading to better decisions.
What determines the cost of price monitoring?
The number of sources, how technically complex they are (heavy JavaScript, pagination, bot protection), the number of products and how hard they are to match to your catalog, collection frequency, and whether you need a dashboard with alerts or a report is enough. Monitoring a few sources with a weekly report is a well-bounded project.
This solution is part of my web scraping and data extraction service.
Want to talk it through?
If your situation sounds similar, tell me how you work today. I’ll tell you which part is worth automating or building first.