I built Obyflow, an open-source observability tool, because I kept running into the same problem: debugging an LLM application rarely looks like debugging a normal web app.A stack trace tells you where the code broke. It doesn’t tell you that the model call succeeded but the retriever returned nothing. It doesn’t tell you that a vector search technically worked but came back with terrible similarity scores. It doesn’t tell you that your agent is stuck waiting on a tool call three steps deep, or that a model’s behavior quietly shifted after a deploy.You can piece all of that together from logs, traces, and a vector DB console. I did it manually for a long time. Eventually I decided the actual goal shouldn’t be “collect more telemetry.” It should be answering the question I actually cared about:What went wrong?That’s what I built Obyflow to do.What it isObyflow is an open-source, CLI-first observability tool for AI applications — LLM calls, embeddings, vector databases, LangChain/LangGraph workflows, and tool calls. It’s local-first, using SQLite with no backend to stand up, and it uses an LLM of your choice to turn raw evidence into a plain-English explanation of an incident.I want to walk through a few pieces of the implementation that matter more to me than the feature list does, because they reflect decisions I made deliberately rather than things that simply fell out of the architecture.The project structureIt’s a pnpm + Turborepo monorepo, TypeScript-first, with a separate Python package that mirrors the same event contract:packages/ core/ event model, storage, config, evidence & anomaly logic cli/ the `obyflow` CLI node-sdk/ @obyflow/node instrumentation SDK adapters/ adapter-framework/ LangChain callback handler adapter-vectordb/ Pinecone / Qdrant / Weaviate / Chroma / pgvector / Milvus llm/ llm-core/, llm-anthropic/, llm-openai/, llm-gemini/, llm-ollama/python/ obyflow-python/ Python SDKEverything downstream depends on getting the event model right, so that’s where I started. I defined it once in TypeScript with Zod, and the Python implementation mirrors that schema rather than reinventing it. Keeping the two in sync is something I treat as a hard requirement, not a nice-to-have; it’s also called out explicitly in CONTRIBUTING.md.There are ten event types: trace, log, metric, error, embedding, vector_op, chain, tool_call, llm_call, and custom.I deliberately didn’t make the AI-specific ones generic key-value blobs.An llm_call event carries provider, model, prompt/completion token counts, latency, and stop reason as typed fields. A vector_op event carries the DB provider, similarity scores, result count, and query latency.That specificity is the whole point.A generic log line saying “retrieval completed” isn’t something you can build much automated diagnosis on top of. “0 documents returned from a query that normally returns 8” is.Where the actual diagnosis happensThe part of the codebase I’m most attached to is packages/core/src/evidence. This is where I encoded the failure patterns I kept seeing in practice into things the system can check for automatically.Chain-step diagnosis (chain-diagnosis.ts) walks correlated chain, tool_call, and llm_call events looking for:a step that failed outrighta tool call that exceeded its timeout, or whose result text reads like a timeouta retriever step that returned zero documentsa step whose duration has regressed significantly against its own historical baseline, measured with a z-scoreI made sure these aren’t just booleans. Every signal carries a severity, a human-readable reason, and a detail object with the actual numbers behind the conclusion.For example, a duration regression reports the observed duration, the baseline mean, and the z-score together.I wanted the diagnosis layer, and the person reading it, to have something concrete to check rather than just a red flag.Retrieval diagnosis does the equivalent for vector-DB operations: empty results, unusually low similarity scores, slow queries, and embedding latency spikes.This is the piece where I think Obyflow stops feeling like a generic tracer and starts feeling purpose-built. It’s not just recording that a database call happened; it’s recognizing that a specific pattern of database behavior is suspicious in the context of a retrieval pipeline.I didn’t want the confidence score to just be the LLM’s opinionThis was a deliberate choice. I didn’t want “how confident should you be in this diagnosis?” to be something the model decides for itself. Models aren’t particularly good judges of their own certainty.So packages/core/src/confidence/confidence.ts computes confidence from the evidence directly.It considers evidence volume, strength of statistical deviation, the number of anomalous metrics, correlated services, deployment correlation, and whether I actually established real parent/child span relationships or just inferred correlation from timing.Stronger z-scores score higher. More independent evidence sources score higher. The result maps to HIGH, MEDIUM, or LOW.The part I care about most is that I keep the reasons that contributed to the score, not just the final tier.If Obyflow tells you it’s HIGH confidence, you can see why. It’s not another opaque AI-generated number that you have to take on faith.Making the LLM show its workThis is probably my favorite piece of the whole codebase.Once I hand evidence to an LLM and ask it for a diagnosis, there’s an obvious risk: it can cite evidence that doesn’t actually exist.So I built validateEvidenceGrounding in:packages/llm/llm-core/src/grounding.tsThe model returns evidence_refs for whatever it claims to have used, and I check every one of those IDs against the real evidence object.Anything that doesn’t match gets flagged as ungrounded, and the result includes a groundedness ratio plus an explicit warning when references look hallucinated.It’s a set-membership check. I didn’t try to make it more sophisticated than that, and I don’t think it needs to be.I’m not trying to verify whether an entire paragraph of reasoning is “true.” I’m verifying whether the specific evidence IDs the model claims to be citing actually exist.That’s a much more tractable problem, and it catches a meaningful class of unsupported citations.I put this in llm-core rather than duplicating it across providers, so every adapter — Anthropic, OpenAI, Gemini, and Ollama — goes through the same check.It remembers past incidentsObyflow doesn’t treat every investigation as a blank slate.packages/core/src/incident/memory.ts fingerprints each investigated incident from things like affected services, anomaly types, change types, and error signatures, then compares future incidents against past ones using Jaccard similarity over those token sets.I kept this deliberately simple. No embeddings, no vector search over incident history, just set overlap on structured tokens.When a new incident looks similar enough to a past one, Obyflow surfaces that past incident, its resolution status, and whatever fix was recorded.There’s also a small aggregation step that shows which recommendation got applied most often across similar resolved incidents.The feedback loop is one command:obyflow incident resolve --status resolvedThat resolution becomes part of what future investigations can draw on.I went with something modest here on purpose. There’s not much value in building a fancy retrieval system on top of incident data before the underlying data is actually useful, and this approach gets more useful as a team builds up incident history.Node and Python don’t do instrumentation the same way, and that’s intentionalThe two SDKs share the same event model, but I didn’t try to force identical instrumentation strategies onto two very different runtimes.The Node SDK patches Node’s http module when you call start(), so inbound tracing for Express, Koa, Fastify, or a raw http.createServer app works with zero extra code.I didn’t try to replicate that in Python.Instead, I shipped explicit ASGI/WSGI middleware: ObyflowASGIMiddleware for FastAPI/Starlette and ObyflowWSGIMiddleware for Flask/Django, which you register once.Monkeypatching Python’s HTTP stack the way I do in Node felt like the wrong tradeoff, so I didn’t force it.Both SDKs propagate x-obyflow-trace-id and x-obyflow-parent-span-id across outbound requests, which is what keeps a request that crosses service boundaries part of the same trace without manual header wiring.A basic Node setup looks like this:import { start } from "@obyflow/node";const obyflow = start({ service: "checkout-api" });const pineconeIndex = obyflow.instrument.pinecone(index);const langchainHandler = obyflow.instrument.langchain();LangChain and six vector databases — Pinecone, Qdrant, Weaviate, Chroma, pgvector, and Milvus — are supported out of the box, along with embedding calls from OpenAI, Anthropic, and Cohere.A few smaller decisions worth mentioningGit metadata gets attached automatically.Every event carries hostname, PID, runtime version, and the current commit SHA. The SHA is pulled from CI environment variables such as GITHUB_SHA and VERCEL_GIT_COMMIT_SHA, falling back to a local git rev-parse HEAD.I did this specifically so “what changed?” correlation can point at a real commit instead of a vague time window.Redaction isn’t just field-name matching.Beyond configured field names such as password, token, credit card, SSN, and API key, I added value-pattern detection — Luhn-validated card numbers, SSN-shaped strings, and bearer tokens.The reason is simple: people inevitably log things they didn’t mean to log, often under field names nobody configured for.The anomaly detection genuinely differs between Node and Python, and I say so rather than pretending otherwise.The TypeScript core, used by investigate, ask, and incident, does mean/stddev and median/MAD baselining with rolling, deployment-aware buckets and a configurable z-score threshold.The Python analysis module is a simpler, separate toolkit with fixed severity thresholds, plus an IsolationForest-based detect_ml_anomalies function that has no TypeScript equivalent.If you’re expecting the two to behave identically, they won’t. I’d rather document that than paper over it.Retries target specific transient failures, not everything.The retry logic uses exponential backoff for 429s, 503s, and transient network failures such as ECONNRESET, ETIMEDOUT, and ECONNREFUSED.That’s preferable to a blanket retry-on-any-error policy that can make an already-bad incident worse.Trying itThe basic workflow is short:npm install -g obyflownpx obyflow initnpx obyflow startnpx obyflow config llm --provider anthropicnpx obyflow investigate npx obyflow ask "why did checkout fail today?"There’s a --no-llm mode too.That matters because it lets you see what the deterministic parts of the system — evidence collection, anomaly detection, and confidence scoring — are finding without spending a token or waiting on a model.Sometimes I just want the facts.Where this is goingThe part of Obyflow I’d defend hardest isn’t the CLI. It’s the decision to model AI-specific events explicitly and build real diagnosis logic on top of them instead of leaning entirely on an LLM to make sense of generic logs.A conventional tracer can tell you that a database query took 1.8 seconds.The goal with Obyflow is to tell you that a vector retrieval returned zero documents, that similarity scores were unusually low, or that a chain step regressed against its own baseline — because those are the failures that actually happen in RAG and agent pipelines, and they’re specific enough to detect deterministically.I also didn’t want to hand the entire diagnosis over to an LLM and call it done.Evidence collection, anomaly detection, confidence scoring, and evidence-grounding validation are deterministic pieces that exist around the model-generated explanation so the result can be inspected and challenged.The project is still early, and there is plenty left to build. More integrations, broader framework coverage, and better anomaly baselines are all areas I’m continuing to work on.But the basic idea is something I keep coming back to:Observability shouldn’t stop at showing you what happened. It should help you understand why.If you’re building LangChain agents, RAG systems, or other LLM-heavy applications and your current debugging workflow consists of jumping between logs, traces, model dashboards, and vector database consoles, Obyflow is built for exactly that problem.Demo: obyflowapp.onrender.comRepo: github.com/Obyflow/obyflownpm: npmjs.com/package/obyflowPyPI: pypi.org/project/obyflow-pythonIssues, discussions, and pull requests are welcome. If you try it, I’d be particularly interested in hearing where the diagnosis is wrong, where the instrumentation gets in your way, or what failure mode you wish it could catch.