If you are experimenting with AI — and if you aren’t, you’ll soon be left irrevocably behind — you need the basics in place before you can reap the benefits. In fact, despite tens of billions invested, poor data strategy and organizational silos cause 95% of AI experiments to fail.This is why the streaming giant Spotify is now offering development teams the Confidence to be more agile and experiment more easily, within Spotify Portal. The company’s pre-built internal developer portal, now generally available with SOC 2 compliance, comes with Spotify’s Confidence experimentation platform built right in, to allow organizations to run experiments and make data-based decisions throughout the business.Confidence allows users to “fail fast and learn fast” with A/B testing, monitoring and alerts, and automated rollbacks, doing it in a way that allows for collaboration and experiment reviews to break down the silos holding AI performance back.Tyson Singer, Spotify’s senior vice president of technology and platforms, told The New Stack about Portal’s move to GA, building Confidence, how the AiKA AI knowledge assistant is faring, how to Soundcheck your organizational health, and more. Read on for how to embrace more efficient, scalable and measurable experimentation — without breaking your business.Spotify: Still an Extreme DogfooderThe rise of the internal developer portal first came in response to a need for a translator — from engineering expense to business value and back again. A portal creates a single pane of glass to search, manage and understand access and the software developer life cycle.Backstage — the open source framework for building internal developer portals that Spotify donated to the Cloud Native Computing Foundation (CNCF) — is already being used daily by just about every developer working at the audio streaming service. Over the past year, Spotify engineers have inner sourced their own internal chatbot and Model Context Protocol (MCP) servers that are now also integrated into the Portal.As Spotify scales its premium Portal offering, it continues to integrate features that have long been dogfooded by its own developers.“Rather than having to go out of their context to do their work, we built AiKA, which is our AI chatbot, our knowledge assistant, into Portal so that they would be able to get that information in the space,” Singer said. “It really is designed to have all of your corporate information built into it, so you don’t have to try to go over to a third-party chatbot, inject all that stuff, to try to manufacture some sort of [retrieval-augmented generation] based thing [and] try to plug in all the MCP servers to get the right answers.”All of this comes built in, he said, and is then integrated in the rest of the development environment the dev team has chosen.This unified organizational context, Singer argued, is powerful for his organization because Portal:Scaffolds development environment standards. which are reflected in any new code.Integrates the AiKA chatbot into Spotify developers’ main communication tool, Slack.The Role of AiKAAs each Spotify squad supports another, “it’s been really having an amazing impact on reducing time to getting answers,” Singer said. “We’ve seen those times go down by almost 50% to resolve internal support requests, which is huge, but it’s also huge from the perspective of not disrupting the developer flow,” as this deep integration synthesizes all the internal information.Of course, as with all monitoring processes, this could see an increase in alert fatigue. But, he promised, it’s usually quite the opposite, particularly with different but integrated development teams mostly in Sweden and the U.S., six to nine hours apart. The AiKA support Slackbot can allow different teams to get answers even when people aren’t working.AiKA learns from both documentation and code, and tries to present the highest quality answer, Singer said, but it’s not always correct — which is when everyone needs to go back and improve its knowledge base.“The AI is not any smarter than the information we give it,” he said. “So you want the information to be good and you want to create positive ways to do that [knowing that] developers don’t love to do documentation.”When some responses are inevitably incorrect, requesting teams can find out the source of that answer and then fix it themselves in the technical documentation, “so the AI is smarter the next time around. And it creates this positive flywheel for quality information,” encouraging self-sustaining growth via intrinsic motivation.With 86% of Spotify’s active GitHub users now using AiKA daily, there’s been a 47% reduction in production support time.Experimentation at an Enterprise Engineering ScaleThe Spotify engineering organization conducts about 10,000 experiments a year, with more than 300 teams running and measuring them inside the Confidence experimentation platform.The generally available version of Confidence includes monitoring and metrics to prevent regressions, plus progressive delivery actions including A/B testing, feature flags and controlled rollouts.Singer gave the example of an experiment at Spotify that looked to increase podcast consumption. The metrics need to not only monitor if that number is going up, but also if an increase is at the expense of other metrics that are important to the business.At Spotify, developers roll back about 42% of experiments to prevent another business metric from regressing, which he believes is in line with other organizations that have reached that level of experimentation maturity.Portal Confidence comes with some business-centric engineering metrics built in, but customers can set up their own default metrics. Spotify also provides a service to assist with this.“We’ve designed some things for customers so that they can kind of get started. But every company is different, right?” Singer said. “Spotify has just a different set of end-user goals than other companies, so we can help people set those things up and run, but I don’t think there’s any single answer for everybody, because everybody’s business is different.”Spotify has already had Portal success stories with long-time design partners including PagerDuty, TrueCaller, Metimeter and Levincoin, each having different business goals, but each looking to include more roles in experimentation.“Experiments are hard when you do it in a sophisticated way,” Singer said. “This is why we want all these capabilities of being able to detect regressions automatically.”Which is why, he added, “it’s often been the remit of only data scientists who can do this, but that doesn’t necessarily scale.”With Confidence, engineers and product managers can safely run experiments too.Simple, Opinionated — and Flexible?The Spotify team’s goals for GA, Singer said, are to release a version of Portal that is simple to onboard your catalog and users, while remaining both opinionated and extensible.“Sometimes simple and flexible, they don’t necessarily match,” he said. “And then we also want it to be opinionated, because we do want people to be able to take advantage of the way that Spotify does it to really provide a lot of the expertise that we have inside of Portal. Those are all things that are in tension with each other, typically — flexible, opinionated, simple.”Just like with Backstage open source users, Portal plugins are the solution to set this balance, with a mix of:Spotify plugins, including AiKA, Confidence feature flags and role-based access control.Recommended plugins, including partners PagerDuty, New Relic, Azure DevOps and several GitHub ones.Other integrations and extensions, including those that organizations build themselves.“We’ve made some changes to our Portal Plugin Studio, a way for users to more quickly built-in integrations with their Portal instance,” Singer said. “This is about that flexibility, so that they get more instantaneous feedback while they’re building this, and they have the ability to run against their existing data within their Portal instance in a more sort of safe and isolated way.”Spotify has also added in a Catalog Wizard, which, he said, “is about making it simple so that you can really get things into your catalog and manipulate them through the Portal.”This is in part because Spotify is a heavy YAML user, but it has received feedback from customers and design partners who want to use, for example, JSON, instead. This way, through the Catalog Wizard, users can access plugins via the Portal UI or YAML.Measure What Matters“When I go around and talk to my peers about the insights that Spotify has about its technology ecosystem, usually their jaws drop open,” Singer said. “They’re like: Wow, you’ve got a lot of data.“We like to instrument the heck out of things. So then we like to analyze it, to understand things, and then make hypotheses on it and see if things are working or not.”Backstage Soundcheck was created internally and later released as a scorecard for frontline development teams to monitor quality and security standards at the component level. Now it includes Tech Insights, which creates an organizational-level view across teams and components for engineering leaders.“They want to see whether folks are meeting golden tech standards, quality standards, security standards, costs, whatever it is they want to put into that environment,” Singer said. “Because Spotify itself is a bottoms-up oriented engineering culture, we hadn’t really baked that into rolling up that for more senior engineering leaders.”Now it’s native to Backstage for all organizations at all levels to, as he put it, herd the cats across the organization towards whatever goals they want.Sign up now to be one of the first to receive The New Stack’s free new eBook: AI for the Enterprise: The Playbook for Developing and Scaling Your AI Strategy.The post Confidence: Spotify Gives Developers a Platform for Experiments appeared first on The New Stack.