9 Questions That Expose Fake AI on a Product Roadmap

Wait 5 sec.

I have sat in the room where a vendor demo got a standing reaction, then watched the same product fail a single edge case in week two of the pilot. The demo was a chatbot with three canned branches. The deck called it an autonomous agent.This happens because "AI-powered" has no agreed definition, so anyone can claim it. I run Salesforce and Vlocity CPQ for a living, currently as Lead Product Owner on the T-Mobile account through Mphasis, and before that I spent 15 years building ERP and CRM systems at Amazon, PwC, and Accenture. I have shipped real machine learning. At Amazon, I co-built a clause-extraction system that hit 95% accuracy across a $40B annual contract portfolio. I have also shipped plain rules engines that I was tempted to call AI and did not.The cost of the confusion is not vanity. A 2025 MIT NANDA study found that 95% of enterprise generative AI pilots delivered no measurable P&L impact, against $30 to $40 billion in spend. A large share of that money went to things that were sold as adaptive and were actually static.Here is the detox I run on a roadmap before any "AI" claim ships. Nine questions. If a feature cannot pass, the label comes off.1. What decision does it make without a human?An agent takes an action and accepts the consequence. A chatbot returns text and waits for a person to act.Ask the team to name one decision the system commits to on its own. Approving a discount. Routing a contract. Changing a quote. If the only answer is "it suggests, then a human clicks," you have a recommendation feature. That can be useful. It is not an agent.2. What happens on an input it has never seen?Deterministic systems fail loudly on unmapped inputs. They hit the else branch and stop.Ask for a live test on an input nobody prepared. A real agent degrades in a graceful, probabilistic way. An if-statement in a costume throws an error or returns the default. I have caught more fake AI with one unscripted input than with any architecture review.3. Can it explain why, or only what?A model can give you a confidence score, a ranked set of features, or a retrieved source. Rules can only restate the rule.If you ask "why did it choose that?" and the answer is a flowchart, the intelligence lives in the flowchart, not in a model.4. Does it change behavior as data changes?The MIT finding that stuck with me: generic tools stall in the enterprise because they do not learn from the workflow. Adaptation is the dividing line.Feed the system a month of new outcomes. If its behavior is identical, nothing is learning. You have a configuration, and configurations are fine, but they are not agents.5. Where does the model actually sit in the call path?This is the one that exposes the most AI-washing. Trace the request end to end. Many "AI" features call a model for one cosmetic step, like rewording a confirmation message, while every real decision runs through hard-coded logic.Draw the call path on a whiteboard. Mark the exact line where the model output changes an outcome. If that line does not exist, the model is decoration.6. What is the token cost per action, and who is tracking it?Real model use has a real bill. If a vendor claims heavy AI but cannot tell you the cost per inference, one of two things is true. Either they are not calling a model much, or they are not watching their own economics. Both are problems.I ask for cost per transaction in dollars and tokens. A team running genuine agents knows this number because it shows up in their cloud invoice every month.7. Can it run a multi-step task and recover mid-way?Agents chain steps and handle a failure in step three without restarting. Scripted flows run a fixed sequence and break when a step returns something unexpected.Give it a task that needs four steps where step two returns a surprise. Watch whether it adapts or collapses.8. What did it cost to build, in data, not just model access?This is the tell I trust most, because I have lived it. Real machine learning needs clean, labeled, structured data. My Amazon clause model did not work because the model was clever. It worked because we did the unglamorous data work first: labeling, validation, catalog hygiene.If a team claims a sophisticated agent but spent nothing on data preparation, the claim is hollow. Ask what their training and grounding data looked like and who cleaned it. Silence is your answer.9. Would it survive an architect reading the code?The final filter. Every AI claim I make has to hold up when a solution architect opens the repo. Most AI-washing dies here, because the marketing language and the source code tell different stories.If the people writing the deck will not let an engineer narrate the code to the steering committee, the label is doing work the product cannot.How to run the detoxTake your current roadmap. For every feature tagged AI, agent, or intelligent, walk the nine questions and write the honest answer next to each.Then re-tag in three buckets:Agent: makes an unsupervised decision, adapts to data, has a real model in the decision path, and carries a measurable inference cost.Model-assisted: uses a model for a real step, but a human makes the call. Honest, valuable, and not an agent.Rules: deterministic logic. Often the right tool. Call it what it is.I have moved features from bucket one to bucket three in my own roadmaps. It is not a downgrade. A clean rules engine that closes a quote in seconds beats a fragile agent that fails an audit. The point of the detox is that the label matches the build, so the next person who funds the work knows what they are paying for.The teams that cross the MIT GenAI divide are not the ones with the boldest labels. They are the ones whose labels are true, because true labels are the only ones an architect, a CFO, and a customer can all act on.If your roadmap cannot survive these nine questions, the work this quarter is not more AI. It is honest tagging, then real data preparation under the few features worth the name.