Elastic Introduces New Vector Storage Format DiskBBQ for More Efficient Vector Search
New alternative to HNSW brings faster, more cost-effective search
Elastic (NYSE:ESTC), the Search AI Company, announced DiskBBQ, a new disk-friendly vector search algorithm in Elasticsearch that delivers more efficient vector search at scale than traditional industry-standard search techniques used in many vector databases. DiskBBQ eliminates the need to keep entire vector indexes in memory, delivers predictable performance, and costs less.
Hierarchical Navigable Small Worlds (HNSW) is the most commonly used search technique in vector databases because of its speed and accuracy in similarity search. However, it requires all vectors to reside in memory, which can be costly at large scale. DiskBBQ, available now in Elasticsearch 9.2, uses BBQ (Better Binary Quantization) to address this by compressing vectors efficiently and clustering them into compact partitions for selective disk reads. This reduces RAM usage, avoids spikes in data retrieval time, and improves system performance for data ingestion and organization.
"As AI applications scale, traditional vector storage formats force them to choose between slow indexing or significant infrastructure costs required to overcome memory limitations," said Ajay Nair, general manager, Platform at Elastic. "DiskBBQ is a smarter, more scalable approach to high-performance vector search on very large datasets that accelerates both indexing and retrieval."
In benchmark testing, DiskBBQ demonstrated a balance of speed, stability and efficiency that is ideal for large-scale vector search on lower-cost memory infrastructure and object storage. As a disk-friendly ANN algorithm, it requires far less memory than HNSW, which keeps the entire graph in RAM by offloading data to disk and reading only relevant vector clusters at query time. This design removes memory as a limiting factor, enabling Elasticsearch to scale to massive datasets limited only by CPU and disk.
DiskBBQ sustained query latencies of roughly 15 milliseconds while operating in as little as 100 MB of total memory, where traditional HNSW indexing could not run. As available memory increased, DiskBBQ's performance scaled smoothly without the sharp latency cliffs typical of in-memory graph approaches.
To learn more about DiskBBQ, read the Elastic blog.
Availability
DiskBBQ is available in technical preview in Elasticsearch Serverless.
About Elastic
Elastic (NYSE:ESTC), the Search AI Company, integrates its deep expertise in search technology with artificial intelligence to help everyone transform all of their data into answers, actions, and outcomes. Elastic's Search AI Platform — the foundation for its search, observability, and security solutions — is used by thousands of companies, including more than 50% of the Fortune 500. Learn more at elastic.co.
Elastic and associated marks are trademarks or registered trademarks of Elasticsearch BV and its subsidiaries. All other company and product names may be trademarks of their respective owners.
View source version on businesswire.com: https://www.businesswire.com/news/home/20251030805965/en/
Media Contact
Elastic PR
[email protected]