so we moved our support bot onto our own self-hosted model earlier this year so customer data stays in house. It answers from our help docs with RAG and for months it was fine and just ast week a customer sent a screenshot where the bot told him accidental damage was covered under warranty and we’d repair it for free. It isn’t and we don’t. The doc it pulled was about the extended plan, which he doesn’t have, and the bot blended the two we had to honor it cause arguing with a screenshot of your own bot is a losing game, air Canada literally lost a tribunal case over its chatbot giving misleading info about bereavement fares in 2024.haunting part is nothing in our logs looked obviously wrong. The answer was fluent and on topic, it was just built on the wrong chunk and from reading around, catching this before the customer does seems to split into two jobs one is to logging the retrieved chunks next to every answer so you can trace where things went wrong, and tools like Langfuse or Arize Phoenix help with that. The other is a check on the live answer before it gets sent, blocking anything about coverage or refunds that the retrieved evidence doesn’t actually back up then second part we’re looking at Orq AI, which runs evaluators as guardrails that block a response when a check fails, and has an on-prem install so nothing leaves our setup. The alternative is Nemo Guardrails and writing more of the checks ourselves Still not sure whether a guardrail on the way out catches more than someone reading transcripts every morning, especially when the bot mixes up two policies that are both technically in the docs   submitted by   /u/CommonMinimum587 [link]   [comments]