If you scrape from JVM stacks (Spring, Kotlin, Spark), I've ported trafilatura to Java. Point it at a URL or raw HTML and you get the main content as Markdown, JSON or text, plus title, author and date, without nav, cookie banners or related-links boxes. It also does feed and sitemap discovery and has a small focused crawler, so you can go from a homepage to a set of clean articles. CLI: java -jar trafilatura-java.jar -u https://example.org/article --markdown Repo: https://github.com/danielonet/trafilatura-java. I'm looking for sites where it fails, so please open issues with URLs.   submitted by   /u/doverakov [link]   [comments]