I built CrawlSpider by repurposing pieces of two things I had already built.... One was InfoCaptor, where I already had data visualization, dashboard and data processing pipeline. The other was an internal link builder I had built under CrawlSpider, which already knew how to crawl sites, extract pages and work with URLs/content. I started combining those pieces into something different: an AI visibility tracker. The basic pipeline is now: brand + prompts → multiple LLMs → collect responses → detect brand mentions/citations → store results → track visibility over time. Even with availability of coding agents I prefer to stay with consistent architecture and stack. To be honest, if I had just let vibecoded crawlspider from scratch I would have been totally lost and spent several days getting to know the system. But now it is live and all pieces are working together. PS I also published a blog post recently how to build this from scratch and some of the pit falls.   submitted by   /u/Pleasant-Weakness959 [link]   [comments]