OpenSearch 3.2 Delivers Hybrid Search, Enhanced Observability Tools

Wait 5 sec.

AMSTERDAM — OpenSearch’s latest 3.2 release offers a good look not only at what’s available for its analytics and now increasingly observable capabilities, but how the project has grown and expanded following its donation to the Linux Foundation.Ongoing improvements to OpenSearch — a flexible open source data ingestion and analytics engine — cover testing, observability and faster and improved feedback for contributors. The new version also continues to develop agentic AI support through the Model Context Protocol (MCP).A lot of these new features, thanks to a new range of backers well beyond its original creator, Amazon Web Services, stem from large organizations such as ByteDance, IBM DataStax, SAP and Uber. Google Cloud and Microsoft Azure also offer OpenSearch managed services.Hundreds of other larger and comparatively small user organizations are finding utility in OpenSearch’s expanding capabilities, making it one of the faster-growing open source projects under the Linux Foundation.This is in no way an endorsement of OpenSearch versus Elasticsearch and Kibana, from which OpenSearch was forked. Those products do seem to be bifurcating somewhat, following Elastic‘s licensing changes, which led to the OpenSearch fork.What’s New in OpenSearch 3.2?This bifurcation became more evident while speaking at Open Source Summit Europe here with Pallavi Priyadarshini, engineering head at Amazon OpenSearch Service Dataplane for AWS. The main features of OpenSearch since the introduction of version 3.0, Priyadarshini said, include:Release automation, which addresses the high volume of code and dependencies from numerous contributors and repositories. A new AI chatbot was introduced to automate release workflows. Additionally, decoupling was introduced to help those with certain needs both contribute to and facilitate the implementation of new features under the umbrella of OpenSearch.“The goal of this bot is to democratize the release process, making it accessible to anyone in the community without needing a central release team or deep knowledge of commands, as it is driven by natural language,” Priyadarshini told The New Stack.“The project also aims to decouple codebases so that individual components can be released independently, which is useful for smaller features that come in as plugins.”More reactive feedback loops have also been implemented. If users and contributors see something really important that you’re working on, they can initiate the pull request knowing its status within eight weeks, maximum.“Open source users expect the code contributed to be included in releases quickly, without long waits,” Priyadarshini said. This expectation is a key reason for the project maintainers’ hyper focus on maintaining an eight-week release cycle and investing in release automation.The necessity to automate was underscored by how interest in the project has drawn a sharp increase in contributors — and code. There are now about 3,300 contributors, with over 2  million lines of code and more than 110 individual repositories, according to OpenSearch statistics released during the Open Source Summit.The release cycle serves as the vehicle for all of this, which is why significant focus is placed on release automation. The sheer volume of people, code, repositories and dependencies requires investment in automation.Search capabilities for AI are another area of focus for OpenSearch’s maintainers, Priyadarshini said. This includes a major push toward a vector database for “trillion-scale” search, GPU acceleration with NVIDIA and “agentic search memory.”Most users need a mix of vector and keyword search, so OpenSearch has invested in “hybrid search” to improve the relevance of results, Priyadarshini said.Search Relevance Workbench, a feature that went into general availability in OpenSearch version 3.2, uses user behavior insights and telemetry to continuously improve search results. This system also supports “semantic search experiences” that learn from a user’s past interactions. The project is also building foundational blocks for agentic search experiences, which can be driven by a human or, in the future, an agent, Priyadarshini said.The project benefits from what Priyadarshini called a “big push toward observability,” with investments in Piped Processing Language (PPL) and combining logs, metrics and traces into a single experience. An SAP case study, Priyadarshini said, demonstrates how OpenSearch and OpenTelemetry are being used for this.Performance-wise, OpenSearch has been optimized to improve latency and resource consumption, despite the addition of new features. OpenSearch 3.2 is reported to be 11x faster than the initial 1.0 release, Priyadarshini said. The project also provides a benchmarking tool to allow for repeatable and reproducible performance testing.Contributions From Multiple CompaniesWhile Amazon was OpenSearch’s initial sponsor, the project is seeing contributions from multiple companies. They include:ByteDance: Contributed to the derived source feature for OpenSearch k-NN and made performance improvements to the segment replication protocol.DTEX: Contributed AI-powered capabilities for insider risk management and data security.IBM DataStax: Contributed the JVector engine to OpenSearch vector search.Intel: Contributed SIMD to OpenSearch k-NN, enabling performance gains on supported hardware.SAP: Contributed toward FIPS compliance, which helps OpenSearch support workloads that require conformance with FIPS specifications.Uber: Contributed a pull-based ingestion feature that removes the need for complex client tuning and configuration. New features like gRPC and pull-based ingestion are particularly of use for Uber, according to the project’s maintainers.Priyadarshini said, “This shared innovation is happening because many organizations realize their homegrown, bespoke platforms cannot keep up with the rapid pace of AI innovation.”The post OpenSearch 3.2 Delivers Hybrid Search, Enhanced Observability Tools appeared first on The New Stack.