I've been thinking about the whole poisoned SERPs situation, and I wonder if we're approaching the problem from the wrong direction? I've been experiencing issues with SERP scraping in the past month In my opinion scrapers are pretty good at telling you when something goes wrong. CAPTCHA, blocked IP, timeout, empty response, whatever, so what would actually happens when nothing obviously goes wrong? You get a 200 response, the HTML parses correctly, and there are 10 organic results you get in your csv file or something Technically, your scraper worked. But can you trust the results is my question I don't think there's an easy fix for this, but one approach that makes sense to me is to stop treating every successfully parsed page as valid data! For example: Keep a history of results for each keyword. Not just yesterday's rankings, but the domains that consistently appeared over the past few weeks Try to flag unusual changs rather than immediately accepting them. If seven familiar domains suddenly disappear and get replaced by unrelated results, that's probably worth checking Try to retry suspicious queries through a different proxy session. If the second request produces completely different results, maybe neither should be trusted immediately I started separating suspicious data from confirmed failures. Instead of reporting a massive ranking drop, I marked the result as uncertain until there's enough evidence But then again, how do you actually know what's a poisoned SERP and what's just Google changing its rankings? And wouldn't comparing every search against historical results create a huge number f false positives? I've been exploring these options with an open-source scraper called SERP Maven that can detect certain wrong-query results and retry them through a fresh proxy session It's a fairly narrow check, not some magical solution to poisoned SERPs I actually think the bigger challenge is figuring out how to validate results without doubling your scraping costs. Has anyone tried building some kind of confidence scoring into their SERP collection? Would be interesting to know whether comparing historical rankings and occasionally repeating suspicious requests is actually useful at scale, or whether it just creates another layer of false positives   submitted by   /u/mckrile [link]   [comments]