Hi all, I keep seeing “just scrape google at night, less traffic, less blocks” in multiple scraping threads. Nobody ever shows numbers for it so i went through my logs and decided to share some insight if this could be useful for you, since I think this is a common misconseption For the hour comparison i only used “cold” searches, the very first search on a fresh IP, with nothing opened in the browser before it. That’s 3979 searches Success = google showed a normal results page. If it showed a captcha or the “unusual traffic” page, i counted it as a fail, even when the status code was 200 (google does that a lot) First i just grouped everything by time of day. Times are UTC Hours (UTC) Success 00-03 15% 04-07 26% 08-11 21% 12-15 23% 16-19 33% 20-23 29% So night was the WORST block and evening looked 2x better than night. Pretty much the opposite of the advice. I almost posted that lol But then i noticed something. Some days google let almost half my searches through, other days almost nothing. And my runs didn’t cover every hour on every day. So if more of my evening tests just happened to land on good days, evening looks great even if the hour has nothing to do with it So i redid it. Instead of mixing all days together, i compared each search with the average of its own day. +2.6 means that block did 2.6 percentage points better than the rest of the same day Hours (UTC) vs the same day’s average 00-03 +0.4 04-07 -1.9 08-11 +1.0 12-15 -2.6 16-19 +2.6 20-23 -0.6 Every block is within about 3 points of its own day. 16-19 UTC is still the best, but only barely. Night is just average, not better and not worse So for google the hour barely matters, which day you run on matters way more. I wouldn’t plan scraping around the clock My setup was the same the whole time: real chrome window (patchright) on a windows server, nodemaven residential proxies, new IP for every test. The query was typed into the search box like a normal user would, not opened as a search link. Aug 12 to oct 22, around 5000 searches Couple other things i noticed: - If the first 5 searches of a run all failed, the rest of that run averaged 17% success. If at least one of the 5 worked, the rest averaged about 30%. So now i run 5 searches first to see what kind of day it is - Browsers that opened 6 pages before the first search got a results page 84% of the time, at any hour, vs 15-33% for cold ones   submitted by   /u/mckrile [link]   [comments]