I want to try using AI to find possibly overlooked interventions for irritable bowel syndrome (IBS) and create a cocktail of interventions that AI thinks would give the highest expected benefit, attacking the illness from different angles, different failure points, etc. There are >100k scientific papers on IBS topic - no agent can ingest all of them into a single context window. I need to process these documents with knowledge synthesis into a more compressed representation than what a collection of papers can give, so that downstream usage of this knowledge would be cheaper in terms of the amount of tokens needed to find relevant knowledge and connections. The LLM wiki idea appears to fit this use case, however, I understand that during ingestion of a document, the AI will be making editorial decisions - do we create a new page, do we edit existing page, which page, do we rename page, should we note this relationship, etc. I feel like this necessitates an editorial policy, but it could be a lot of work to get it done right. Or at least editorial instructions, kinda like "be the best wiki maintainer" without having explicit policy, betting on the hope that the raw intelligence from base model will carry your implicit intent forward reasonably well. Has anyone tried ingesting hundreds, thousands of papers and has experience with such an editorial policy? Maybe could even share your editorial policy? My pipeline would be as follows: Export list of papers from pubmed; download open access PDFs; download PDFs from pirate library; convert PDF to markdown using unlimited OCR; do ingestion in batches of 25 items: create new LLM wiki worktree for each item, have 1 agent for 1 item, let the agent do work inside the worktree, then at the end of batch have reconciliation agent look all 25 worktree diffs and merge into main; then start new batch. This could allow better parallelism than ingesting one by one.   submitted by   /u/gintrux [link]   [comments]