Real transparency on what's working and what's not. Here's exactly how our price scraper is performing right now across 12 US retailers, with every metric pulled live from the run log.
What "live" actually means here. Live scraping currently delivers reliable price updates from GNC, Bodybuilding.com, Target, Vitacost, Muscle & Strength, Tiger Fitness, Costco, MyProtein, and Transparent Labs. Amazon, Walmart, and iHerb prices come from public listings and update less frequently while we finish their affiliate API integrations.
Frequency. Prices are checked at least daily for every working retailer. Some product × retailer pairs refresh several times a day; others only once. Every price displayed on the site carries the timestamp of its last successful scrape, so you can always see how fresh it is.
A quick rundown of the pipeline behind every price you see on ProteinPrice.com.
The scraper runs on a scheduled cycle and attempts every product × retailer combination it has a URL for: currently up to 249 products × 12 retailers = 1,740 fetches per full cycle. Working retailers refresh at least daily; some refresh several times a day.
Each scraped price passes through sanity-range filters, swing-reject (vs prior price), consensus override (vs other retailers), and median-outlier checks before it's accepted.
If a fetch fails or returns garbage, the last known good price stays in place. We never display stale data as fresh: every price carries the timestamp of its last successful scrape.
Run-level success rates, per-retailer health, and the last 10 cycles are all published here. If something's broken, you'll see it before we even fix it.
Not every retailer plays nice with automated price tracking. Here's the honest situation.
Anti-bot defenses are common. Large retailers like Amazon, Walmart, and iHerb invest heavily in blocking automated traffic. When their defenses kick in, our scraper returns either a captcha page or a generic block response: we record that as a failed fetch and move on.
We don't fake it. When a retailer is consistently blocked, we mark it red on this page. We never substitute a guess or last-month's price and pretend it's live. The product comparison either shows the last successful scrape (with its real timestamp) or omits that retailer.
Affiliate APIs are the long-term fix. We're working on official affiliate API integrations for Amazon, Walmart, and iHerb: these give us clean, sanctioned price feeds that bypass scraping entirely. As each integration goes live, the corresponding retailer will flip from blocked to OK on this dashboard.