I run a small market-research platform (Veridian Signal), and I'm sharing our real numbers because the filtering problem is the whole point of what we build. Last 7 days: 1,138 news items came in 903 were left after removing duplicates (about 20% were the same story repeated) 771 (85% of the unique stories) cleared our quality bar and got published The biggest cut is duplicates. The same event gets rewritten by a dozen outlets, and each rewrite feels like new information. Happy to answer questions about how the dedupe and quality checks work. Educational only, not investment advice.   submitted by   /u/Sup3riorSn1per [link]   [comments]