How are you giving your agents web access these days? I want opinions and suggestions as I've been checking through the web access tools people use for agents lately. What I've noticed, specifically for search, Tavily, Exa, Parallel and Brave all run on their own indexes. That's fine for grounding, but it isn't the same as what Google shows right now. Most of the results are ghost/dead and you usually have to recheck a live SERP If you need current prices or news. So that isn't ideal. For page reads, Firecrawl is good when you want schema-based JSON from random pages, the solution is quicker compared to others, at least from my experience, also their docs say reads can come from a cache up to two days old by default. TinyFish and Exa cache similarly. Then for browsers, Browserbase and Browser Use are the ones I see mentioned most. Maybe someone had experience with them? I also found new players, well relatively new, product wise, Oxylabs Web API, yet another web search API, but somehow this one tells me that it was some sort of web scraper api upgrade. It has a /search endpoint for live Google results and a /scrape endpoint for pulling pages back as parsed JSON or Markdown. Though with Google poisoning SERPS I doubt this is also viable right now. There's also an open source MCP server too. Where I find this option weaker, well, there's one subscription now (Starter is $15/mo), but each product still has its own meter, so the bill at the end of the month isn't always easy to predict if you are utilizing several products at once. The core products have no recurring free tier, just welcome credits. I would say an all-in-one setup makes more sense when you're targeting protected sites and don't want to have several different vendors together. Are you guys still on traditional scraping path or have you moved to web search apis? Was it the right choice? Financial, efficiency, optimization wise?   submitted by   /u/MilkywayAnimal [link]   [comments]