Prometheus was long the de facto standard for metrics in observability. Then came VictoriaMetrics, originally a fork of Prometheus, and Grafana’s Mimir, a newer metrics backend designed for scalability. As OpenTelemetry matured, it introduced a unified standard for metrics, logs and traces — appearing to compete with Prometheus in the metrics space.But this competition was only surface-level. Prometheus now integrates well with OpenTelemetry and remains a core part of the ecosystem. That said, Prometheus 2.0 introduced challenges when used as a metrics backend through OpenTelemetry, complicating adoption early on.There have been compatibility problems that trace back to before the release of Prometheus 3.0 this year — or even long before that. On a very basic level, there has been a difference in design philosophy.OpenTelemetry can be described more as a push-based metrics system, issuing metrics data when changes occur in what is being produced, in the runtimes and networks being monitored. In contrast, Prometheus is constantly pulling metrics data into a time series chart — the kind Grafana has made famous with its attractive panels.There are other incompatibility issues, but before going into those, let’s look at some fundamental differences between the two. OpenTelemetry’s strength — and the recognition it deserves — stems from how it is designed to level the playing field by standardizing the various ways metrics are obtained. Behind this design are the providers, who are expected to remain compatible with the OpenTelemetry standard. Once metrics data goes through the OpenTelemetry pipeline, providers can then apply their analytics and observability monitoring capabilities.Prometheus 2.0 and @opentelemetry didn’t get along, but 3.O is better after last year when @grafana ´s Goutham Veeramachaneni said at GrafanaCon 2025: «If you go back just a year, using OTel wasn’t great but.. can now push OTel metrics in Prometheus. » @thenewstack pic.twitter.com/W0trrbFTSZ— BC Gain (@bcamerongain) May 7, 2025Again, before Prometheus 3.0 was released, there were basic incompatibility issues where Prometheus either did not work with OpenTelemetry or had significant difficulties. Indeed, the list of issues was long.“Otel is gaining adoption and we really like Otel,” Goutham Veeramachaneni, a product manager at Grafana Labs and a Prometheus team member, said during GrafanaCon 2025 earlier this year. “If you go back just a year, using Otel with Prometheus was not great…Well, it was painful.”During the talk “Why Prometheus 3.0 Will Change How You Do Observability: OTel Support, Remote Write 2.0, and More” co-presented with Carrie Edwards, a staff software engineer at Grafana, Veeramachaneni noted that default, resource attributes like namespace and cluster with Prometheus are sent to a different metric, which then must be joined, making queries hard. To fix this, a configuration option called Promote Resource Attributes was added to copy these attributes into labels so the labels could be used normally.Still, with Prometheus 3.0 and OpenTelemetry, problems remain, such as breaking changes. One change affected how queries are processed when the time window and scrape interval align to the millisecond, which can happen with Grafana caching, Veeramachaneni said. “Things that used to work before did not work after the change. Another issue broke compatibility with certain Ruby clients, like Sinatra,” he said. “The client sent commas instead of semicolons in a protocol response; this previously worked but failed after the update.”Support for UTF-8 with Prometheus was introduced to resolve another integration issue: Prometheus previously did not support dots in metric names, a character used by OpenTelemetry. Some of the problems stemmed in part from UTF-8 support and other fundamental limitations. Specifically, Prometheus — prior to version 3.0 — was not compatible with OpenTelemetry’s handling of certain characters. OpenTelemetry 3.0, for example, did not support basic characters like dots, UTF-8 symbols, emojis, and other special characters. Periods, for whatever reason, remained an issue until the release of Prometheus 3.0.“A remaining compatibility challenge is the handling of delta temporality. Prometheus natively supports only cumulative metrics,” Veeramachaneni said. Other systems like StatsD, Datadog, and Otel can push deltas. Current support in Prometheus is a “hack” via a feature flag, and the team is still figuring out how to support deltas natively.Compatibility problems related to data formats within Prometheus, specifically with histograms and the data forwarding protocol, remain an issue with OpenTelemetry, Edwards said. Historically, monitoring latency in Prometheus required classic histograms. Classic histograms presented several challenges. Defining appropriate bucket boundaries ahead of time could be difficult. Changing these boundaries necessitated re-instrumentation and redeployment. A significant compatibility issue was that classic histograms could only be aggregated if the bucket boundaries matched exactly, Edwards said.To solve these issues, native histograms were introduced. A new feature in development, native histograms with custom buckets, further addresses compatibility with legacy systems where instrumentation cannot be changed, Edwards said. This feature allows for the conversion of classic histograms into native histograms with custom buckets at scrape time, meaning no re-instrumentation is needed, Edwards said.Edwards also touched on the Remote Write protocol. Remote Write 2.0 improves efficiency and supports newer Prometheus features. “The specification for version 2.0 includes fields in the data for new features like native histograms, addressing shortcomings of the previous version,” Edwards said.All You Need Is LoveOngoing issues remain, as described above, with Prometheus 3.0 and OpenTelemetry. But work is ongoing, and with the support of the open source community, I can confirm firsthand that there is a lot of momentum to overcome these issues, which will be addressed in upcoming Prometheus releases.With the release of Prometheus 3.0, there are already a number of improvements that make collecting metrics through OpenTelemetry significantly better and less painful than in the past.An interesting statistic highlights a common misconception. For those who do not follow observability, Prometheus, or even OpenTelemetry closely, it may be tempting to think it’s an either/or proposition. It is not, as a Grafana survey showed. According to the survey results, 53 percent and 50 percent of those surveyed said their organize plans to increase their use of Prometheus and OpenTelemetry, respectively, The majority of enterprises surveyed use both OpenTelemetry and Prometheus, underscoring how the two are far from mutually exclusive.“There’s still a lot of specific problems that the Prometheus and OpenTelemetry communities need to resolve in order to make that compatibility better, but they’re working on it and they’re making a lot of progress,” Myrle Krantz, director of engineering, Grafana Labs, said, discussing the survey results. “This increase in usage that we’re seeing here, an increase in interest that we’re seeing here between both Prometheus and OpenTelemetry, makes a lot of sense. And all of the signals also go towards people continuing to increase that.”OpenTelemetry is still under development, of course, but it has, in many ways, transformed observability and been a godsend for compatibility by offering standards and a number of other benefits now extending across metrics, logs, and traces. More progress is on the horizon, and Prometheus will likely remain the most popular choice — at least in the near term — not only for Kubernetes but for metrics collection in general. Prometheus will continue to serve its critical purpose and OpenTelemetry will increase its scope of usage and standardization and the developers and contributors will find continue to find ways so they both work better together.The post Prometheus and OpenTelemetry Just Couldn’t Get Along appeared first on The New Stack.