Glossary · 4 minute read
What Is BM25? The Keyword Ranking Behind Search Engines
BM25 is a keyword ranking function that scores how well a document matches a query by combining how often query terms appear in the document, how rare each term is across the collection, and how long the document is relative to average. It is fast, interpretable, and strong on exact terms, so it remains core to hybrid retrieval.
Before embeddings, search meant matching words, and the best way to rank word matches was worked out decades ago as BM25. It has not gone away. Vector search finds meaning but blurs exact terms, so modern retrieval systems keep BM25 as the lexical half of hybrid search, where it anchors precision on identifiers, names, and jargon at negligible cost. This explainer covers how BM25 scores documents, how to tune it, and where it fits, drawing on FISTA Solutions' AI enablement practice. The combined design is in what is hybrid search and the build guide in how to build a hybrid search system.
What is BM25?
BM25 is a probabilistic ranking function used by lexical search engines to score documents against a query. For each query term, it computes a contribution based on the term's frequency in the document, with diminishing returns so that repetition saturates, scaled by the term's inverse document frequency so that rare terms matter more than common ones, and adjusted for document length so that long documents do not win merely by containing more words. The sum across query terms is the document's score. Language processing foundations are in what is natural language processing.
What are the components of the score?
| Component | What it captures | Effect on score |
|---|---|---|
| Term frequency | How often a query term appears in the document | Higher, with diminishing returns |
| Inverse document frequency | How rare the term is across the collection | Rare terms contribute more |
| Document length normalization | Length relative to the collection average | Long documents are penalized |
| Saturation parameter | How quickly repeated terms stop adding | Tunable |
| Length parameter | How strongly length is penalized | Tunable |
Why does term rarity matter?
A term that appears in most documents, such as a common product word, tells the ranker little; a term that appears in few, such as a specific error code, is highly discriminating. Inverse document frequency encodes this, which is why BM25 excels exactly where vector search struggles: rare, specific tokens.
How do tokenization and stemming shape results?
BM25 matches tokens, so how text is split, lowercased, stemmed, and cleaned determines what counts as a match. Stemming lets "configure" match "configuration"; synonym expansion lets "invoice" match "bill"; handling of hyphens, codes, and numbers decides whether identifiers survive intact. These choices are part of retrieval design, and errors here look like ranking errors. Chunking interacts with them; see what is chunking in rag.
How is BM25 tuned?
Two parameters govern saturation and length normalization; defaults suit many corpora, and tuning against a golden set of queries and relevant passages helps when documents are unusually long or short or when queries are dominated by rare terms. Field weighting, boosting titles over body text, and phrase handling are further levers. Evaluation practice is in what is a golden dataset.
What does BM25 miss?
Paraphrase and synonyms unless expanded, meaning across languages, intent that is not expressed in matching words, and context that changes a term's meaning. It cannot know that a passage answers a question phrased differently. Vector search supplies this, which is why the two are combined. Semantic retrieval is described in what is semantic search and vector infrastructure in what is a vector database.
How does BM25 fit into hybrid retrieval?
BM25 produces a ranked list of lexical candidates, vector search produces a semantic list, fusion merges them, and a reranker orders the merged pool. BM25's cheapness and precision make it the safety net for queries containing exact terms. Fusion and reranking are in what is hybrid search and what is a reranker.
Why do interpretability and cost matter in enterprises?
BM25 scores can be explained term by term, which helps debugging and supports compliance conversations about why a document ranked; it runs on commodity infrastructure at very low latency and cost. For high-volume search, these properties are as valuable as accuracy. Search program context is in enterprise search ai.
What does BM25 look like in practice?
A field service knowledge base is searched mostly by part numbers and fault codes. Vector-only retrieval returned plausible but wrong manual sections; adding BM25 with tokenization that preserves codes restored exact matches, and hybrid fusion with a reranker handled the natural-language questions technicians also ask. Recall on the golden set rose and technician time per search fell. Query preprocessing further improved results; see what is query rewriting.
How FISTA Solutions uses BM25
FISTA Solutions includes a tuned lexical index in every retrieval system, configures tokenization to preserve the identifiers a client's users search for, fuses with vector retrieval, and evaluates configurations on client golden sets rather than assuming either method suffices. The AI enablement practice delivers retrieval platforms, AI agents draw on them, and forward deployed engineers embed with client knowledge and search teams. The record behind the approach is 150+ projects with 99.9% uptime.
To get exact-term precision back into AI search, message FISTA on WhatsApp, or read what is a cross-encoder for the reranking stage that follows.
Share-ready article cover
Download the generated social format.
Clear answers
Questions raised by this field note.
Straightforward guidance for evaluating scope, fit, and the next step.
01What is BM25 in simple terms?
A formula that ranks documents for a keyword query. A document scores higher when it contains the query words often, when those words are rare across the collection, and when the document is not so long that the matches are diluted. It powers most traditional search boxes.
02Why does BM25 still matter with vector search available?
Because embeddings blur exact identifiers, rare terms, and precise phrases. BM25 finds the passage containing the exact error code or product name reliably and cheaply, and hybrid systems combine it with vector search to cover both exact terms and meaning.
03What are the BM25 parameters?
Two tunable values: one controls how quickly repeated occurrences of a term stop adding score, and the other controls how strongly document length is penalized. Defaults work for many corpora; tuning on a golden set helps for unusual document lengths.
04What does BM25 miss?
Synonyms and paraphrase unless expanded, meaning across languages, word order and phrases beyond exact matching, and context that changes what a term means. Tokenization and stemming choices also affect what counts as a match.
05How is BM25 used in RAG systems?
As the lexical retriever in hybrid search, producing candidates that are fused with vector results and reranked, and sometimes alone for corpora dominated by exact terms. It also supports filtering and highlighting in search interfaces.
Continue exploring
Related capabilities
Start with the hard problem
Need the outcome owned, not merely analyzed?
Tell us where delivery is constrained. We’ll map the fastest credible path from intent to verified production.