the product is https://webfetch.run. url goes in, clean content comes out. what took the time wasnt the scraping. it was figuring out why a scrape failed. my old code would return an empty body and i'd spend an hour guessing. so the whole thing now returns a verdict. ok, blocked, whatever. each one is a different problem with a different fix. the nicest part is the cache. repeat fetch of the same url within 24h is instant and free. i fetch the same pages over and over while testing and it was a dumb tax on myself. not sure what to do next. the thing works, which after 6 months is honestly a weird feeling.   submitted by   /u/mkotsollaris [link]   [comments]