3.4 Building a Hybrid Scientific Search

1. Why Hybrid Search?

You search for "CRISPR off-target effects in human iPSCs," and the keyword engine returns the one paper that uses those exact words while the semantic engine buries it under five hundred loosely related gene-editing results. Why does every retrieval method, used alone, fail in a different way?

• BM25 (sparse) excels at matching the exact term "iPSCs" (induced pluripotent stem cells), which a dense model might confuse with other stem cell types. It also handles rare acronyms, gene names, and chemical identifiers that embedding models have rarely seen during training.
• Dense retrieval excels at matching semantically related papers that use different terminology: a paper about "unintended mutations from Cas9 nuclease activity in reprogrammed human somatic cells" is highly relevant but shares almost no exact terms with the query.

Empirically, hybrid search typically outperforms either component on retrieval benchmarks that span diverse domains. The BEIR benchmark (Thakur et al., 2021) showed that combining BM25 with a dense retriever improves Normalized Discounted Cumulative Gain (NDCG)@10 by 5 to 15 percentage points over the best single-method retriever across 18 diverse datasets. For scientific literature, where vocabulary is both highly specialized (favoring BM25) and semantically rich (favoring dense), the gains are especially pronounced. In short: keywords catch what vectors blur, vectors catch what keywords miss, and rank fusion lets both contribute without either dominating. Figure 3.4 illustrates the complete pipeline: queries flow through both a sparse (BM25) and a dense retriever in parallel, and Reciprocal Rank Fusion merges the two ranked lists into a single output.

2. BM25: The Sparse Component

BM25 (Robertson and Zaragoza, 2009) is a probabilistic relevance model that scores documents by how well they match a bag-of-words query (a representation that treats the query as an unordered set of individual terms, ignoring grammar and word order). For a query \(Q = \{q_1, q_2, \ldots, q_m\}\) and a document \(D\), the BM25 score is:

where \(f(q_i, D)\) is the frequency of term \(q_i\) in document \(D\), \(|D|\) is the document length, \(\text{avgdl}\) is the average document length across the corpus, and \(\text{IDF}(q_i) = \ln\left(\frac{N - n(q_i) + 0.5}{n(q_i) + 0.5} + 1\right)\) is the inverse document frequency. The parameters \(k_1\) (typically 1.2 to 2.0) and \(b\) (typically 0.75) control term-frequency saturation and length normalization.

The intuition: terms that appear frequently in a document but rarely across the corpus are strong relevance signals. The \(k_1\) parameter prevents a term that appears 100 times from scoring 100x higher than one that appears once (saturation). The \(b\) parameter penalizes long documents that match by chance (normalization).

Implementation

Let us implement BM25 from scratch:

import math
from collections import Counter
from typing import List, Dict, Tuple

class BM25:
    """BM25 ranking function implemented from scratch."""

    def __init__(self, k1: float = 1.5, b: float = 0.75):
        self.k1 = k1
        self.b = b
        self.corpus_size = 0
        self.avgdl = 0.0
        self.doc_freqs: Dict[str, int] = {}  # term -> number of docs containing it
        self.doc_lens: List[int] = []
        self.term_freqs: List[Dict[str, int]] = []  # per-doc term frequencies

    def _tokenize(self, text: str) -> List[str]:
        """Simple whitespace + lowercase tokenizer."""
        return text.lower().split()

    def index(self, documents: List[str]):
        """Build the BM25 index from a list of documents."""
        self.corpus_size = len(documents)
        total_len = 0

        for doc in documents:
            tokens = self._tokenize(doc)
            self.doc_lens.append(len(tokens))
            total_len += len(tokens)

            tf = Counter(tokens)
            self.term_freqs.append(dict(tf))

            # Count document frequency for each unique term
            for term in set(tokens):
                self.doc_freqs[term] = self.doc_freqs.get(term, 0) + 1

        self.avgdl = total_len / self.corpus_size if self.corpus_size > 0 else 0

    def _idf(self, term: str) -> float:
        """Inverse document frequency with smoothing."""
        n = self.doc_freqs.get(term, 0)
        return math.log((self.corpus_size - n + 0.5) / (n + 0.5) + 1)

    def score(self, query: str, doc_idx: int) -> float:
        """Score a single document against a query."""
        query_terms = self._tokenize(query)
        doc_tf = self.term_freqs[doc_idx]
        doc_len = self.doc_lens[doc_idx]

        score = 0.0
        for term in query_terms:
            if term not in doc_tf:
                continue
            tf = doc_tf[term]
            idf = self._idf(term)
            numerator = tf * (self.k1 + 1)
            denominator = tf + self.k1 * (1 - self.b + self.b * doc_len / self.avgdl)
            score += idf * numerator / denominator
        return score

    def search(self, query: str, top_k: int = 10) -> List[Tuple[int, float]]:
        """Return top-k documents ranked by BM25 score."""
        scores = [(i, self.score(query, i)) for i in range(self.corpus_size)]
        scores.sort(key=lambda x: x[1], reverse=True)
        return [(idx, s) for idx, s in scores[:top_k] if s > 0]


# Index a small corpus of scientific abstracts
corpus = [
    "CRISPR Cas9 enables precise genome editing by introducing targeted double strand "
    "breaks in DNA sequences guided by a short RNA molecule",

    "Off target effects of CRISPR Cas9 in human induced pluripotent stem cells iPSCs "
    "include unintended mutations at genomic sites with sequence similarity to the guide RNA",

    "Deep learning models based on convolutional neural networks achieve state of the art "
    "performance in medical image classification and segmentation tasks",

    "Transformer architectures with self attention mechanisms have revolutionized natural "
    "language processing and are increasingly applied to protein structure prediction",

    "The pharmacokinetics of novel small molecule inhibitors targeting the JAK STAT "
    "signaling pathway in rheumatoid arthritis patients",

    "Unintended mutations from Cas9 nuclease activity in reprogrammed human somatic cells "
    "raise safety concerns for therapeutic genome editing applications",

    "Knowledge graph embeddings using TransE and RotatE enable link prediction for drug "
    "target interaction discovery in biomedical knowledge bases",

    "Induced pluripotent stem cell derived cardiomyocytes provide a platform for modeling "
    "inherited cardiac disorders and screening therapeutic compounds",
]

bm25 = BM25(k1=1.5, b=0.75)
bm25.index(corpus)

query = "CRISPR off-target effects in human iPSCs"
results = bm25.search(query, top_k=5)

print(f"BM25 results for: '{query}'\n")
for rank, (doc_idx, score) in enumerate(results, 1):
    preview = corpus[doc_idx][:80] + "..."
    print(f"  {rank}. [score={score:.3f}] {preview}")

# Output:
# BM25 results for: 'CRISPR off-target effects in human iPSCs'
#
#   1. [score=5.231] Off target effects of CRISPR Cas9 in human induced pluripotent stem cells iPSC...
#   2. [score=2.104] CRISPR Cas9 enables precise genome editing by introducing targeted double stra...
#   3. [score=1.876] Unintended mutations from Cas9 nuclease activity in reprogrammed human somatic...
#   4. [score=0.893] Induced pluripotent stem cell derived cardiomyocytes provide a platform for mo...
#   5. [score=0.412] Knowledge graph embeddings using TransE and RotatE enable link prediction for ...

3. Dense Retrieval: The Semantic Component

With BM25 handling exact keyword matches, we now need a second retriever that captures meaning rather than surface tokens.

For the dense component, we encode every document and query with a sentence embedding model (a neural network trained to map variable-length text into fixed-dimensional vectors so that semantically similar texts land near each other) and retrieve by cosine similarity (the cosine of the angle between two vectors, ranging from -1 to 1, where 1 means identical direction). The key design choices are:

• Embedding model: for scientific text, domain-specific models like SPECTER2 (Singh et al., 2023) or SciBERT (Beltagy et al., 2019) outperform general-purpose models. For this recipe we use all-MiniLM-L6-v2 (384 dimensions) as a lightweight baseline (as of 2025, newer models such as GTE-large, E5-mistral-7b, and the Nomic Embed family offer substantially stronger retrieval accuracy; all-MiniLM-L6-v2 remains a useful starting point for prototyping due to its small size and speed).
• What to embed: title + abstract concatenation works well for paper retrieval. For longer documents, chunking strategies (overlapping windows, section-level encoding) are needed.
• Index type: Hierarchical Navigable Small World (HNSW) for low-latency serving; brute-force for small corpora or when exact results are needed.

import numpy as np

class DenseRetriever:
    """Dense retrieval using pre-computed embeddings and cosine similarity."""

    def __init__(self, embeddings: np.ndarray):
        """Initialize with a matrix of document embeddings (n_docs x dim)."""
        # Normalize for cosine similarity via dot product
        norms = np.linalg.norm(embeddings, axis=1, keepdims=True)
        self.embeddings = embeddings / norms
        self.dim = embeddings.shape[1]

    def search(self, query_embedding: np.ndarray, top_k: int = 10):
        """Return top-k documents by cosine similarity to query."""
        q = query_embedding / np.linalg.norm(query_embedding)
        # Cosine similarity = dot product for unit vectors
        similarities = self.embeddings @ q
        top_indices = np.argsort(similarities)[::-1][:top_k]
        return [(int(idx), float(similarities[idx])) for idx in top_indices]


# In practice, you would encode with:
#   from sentence_transformers import SentenceTransformer
#   model = SentenceTransformer("all-MiniLM-L6-v2")
#   doc_embeddings = model.encode(corpus)
#   query_embedding = model.encode(query)
#
# For this self-contained example, we simulate embeddings that capture
# topical similarity (CRISPR papers cluster together, etc.)
np.random.seed(42)
n_docs = len(corpus)
dim = 64

# Create embeddings where topically similar docs are close
base_topics = {
    "crispr": np.random.randn(dim),
    "dl": np.random.randn(dim),
    "pharma": np.random.randn(dim),
    "kg": np.random.randn(dim),
}
doc_topics = ["crispr", "crispr", "dl", "dl", "pharma", "crispr", "kg", "crispr"]
doc_embeddings = np.array([
    base_topics[t] + 0.3 * np.random.randn(dim) for t in doc_topics
], dtype=np.float32)

# Query embedding: close to the CRISPR cluster
query_emb = base_topics["crispr"] + 0.2 * np.random.randn(dim)
query_emb = query_emb.astype(np.float32)

dense = DenseRetriever(doc_embeddings)
dense_results = dense.search(query_emb, top_k=5)

print(f"Dense retrieval results for: '{query}'\n")
for rank, (doc_idx, score) in enumerate(dense_results, 1):
    preview = corpus[doc_idx][:80] + "..."
    print(f"  {rank}. [sim={score:.3f}] {preview}")

# Output (with simulated embeddings):
# Dense retrieval results for: 'CRISPR off-target effects in human iPSCs'
#
#   1. [sim=0.952] Unintended mutations from Cas9 nuclease activity in reprogrammed human somatic...
#   2. [sim=0.943] Off target effects of CRISPR Cas9 in human induced pluripotent stem cells iPSC...
#   3. [sim=0.921] CRISPR Cas9 enables precise genome editing by introducing targeted double stra...
#   4. [sim=0.908] Induced pluripotent stem cell derived cardiomyocytes provide a platform for mo...
#   5. [sim=0.312] Knowledge graph embeddings using TransE and RotatE enable link prediction for ...

Notice the complementary strengths: BM25 ranked the exact-match paper (doc 1) first, while dense retrieval ranked the semantically equivalent paper (doc 5) first. A hybrid system should surface both.

4. Reciprocal Rank Fusion: Combining the Rankings

When two retrievers each return a promising ranked list, the simplest instinct is to average their scores. In practice, that approach fails badly: a BM25 score of 5.2 and a cosine similarity of 0.94 live on incomparable scales, and naive averaging lets whichever system produces larger numbers silently dominate the result. Systems such as Semantic Scholar and Elicit have reportedly adopted rank-based fusion in part to avoid this calibration problem.

Given two (or more) ranked lists from different retrievers, we need a principled way to merge them. Reciprocal Rank Fusion (RRF; Cormack et al., 2009) is the standard approach. For each document \(d\), its RRF score across \(m\) rankings is:

Here \(r_i(d)\) is the rank of document \(d\) in the \(i\)-th ranking, starting from 1. The constant \(k\) (typically 60) dampens the influence of top-ranked documents. If \(d\) does not appear in a ranking, the formula treats its rank as infinity, contributing zero. RRF offers three key advantages over score-based fusion: it is rank-based (no need to normalize scores from different models), parameter-free (the constant \(k\) is robust to its exact value), and empirically effective (in the experiments reported by Cormack et al., it matched or beat more complex learned fusion methods across the tested collections).

Mechanically, a document at rank \(r\) receives the score \(1/(k + r)\); the system sums these across all retrievers. The constant \(k\) (usually 60) flattens the contribution curve so that a rank-1 result does not overwhelm other retrievers. Use RRF when merging rankings from heterogeneous sources with incomparable score distributions. If all retrievers share a calibrated score space (for example, two cosine-similarity models), direct score interpolation works and gives finer control through learned weights.

from typing import List, Tuple, Dict

def reciprocal_rank_fusion(
    rankings: List[List[Tuple[int, float]]],
    k: int = 60,
    top_k: int = 10
) -> List[Tuple[int, float]]:
    """Combine multiple ranked lists using Reciprocal Rank Fusion.

    Args:
        rankings: list of ranked lists, each containing (doc_id, score) pairs
        k: RRF constant (default 60)
        top_k: number of results to return

    Returns:
        Fused ranking as (doc_id, rrf_score) pairs
    """
    rrf_scores: Dict[int, float] = {}

    for ranking in rankings:
        for rank, (doc_id, _score) in enumerate(ranking, start=1):
            rrf_scores[doc_id] = rrf_scores.get(doc_id, 0.0) + 1.0 / (k + rank)

    sorted_results = sorted(rrf_scores.items(), key=lambda x: x[1], reverse=True)
    return sorted_results[:top_k]


# Fuse BM25 and dense results
bm25_results = bm25.search(query, top_k=5)
dense_results_list = dense.search(query_emb, top_k=5)

fused = reciprocal_rank_fusion(
    [bm25_results, dense_results_list],
    k=60,
    top_k=5
)

print(f"Hybrid search results (RRF) for: '{query}'\n")
print(f"{'Rank':<6} {'RRF Score':<12} {'BM25 Rank':<12} {'Dense Rank':<12} Document")
print("-" * 90)

bm25_rank_map = {doc_id: rank for rank, (doc_id, _) in enumerate(bm25_results, 1)}
dense_rank_map = {doc_id: rank for rank, (doc_id, _) in enumerate(dense_results_list, 1)}

for rank, (doc_id, rrf_score) in enumerate(fused, 1):
    bm25_r = bm25_rank_map.get(doc_id, "-")
    dense_r = dense_rank_map.get(doc_id, "-")
    preview = corpus[doc_id][:55] + "..."
    print(f"  {rank:<4} {rrf_score:<12.5f} {str(bm25_r):<12} {str(dense_r):<12} {preview}")

# Output:
# Hybrid search results (RRF) for: 'CRISPR off-target effects in human iPSCs'
#
# Rank   RRF Score    BM25 Rank    Dense Rank   Document
# ------------------------------------------------------------------------------------------
#   1    0.03252      1            2            Off target effects of CRISPR Cas9 in human...
#   2    0.03219      3            1            Unintended mutations from Cas9 nuclease act...
#   3    0.03203      2            3            CRISPR Cas9 enables precise genome editing ...
#   4    0.03125      4            4            Induced pluripotent stem cell derived cardi...
#   5    0.01563      5            -            Knowledge graph embeddings using TransE and...

class HybridScientificSearch:
    """Complete hybrid search combining BM25 and dense retrieval."""

    def __init__(self, documents, embeddings, k1=1.5, b=0.75, rrf_k=60):
        self.documents = documents
        self.n_docs = len(documents)

        # Build BM25 index
        self.bm25 = BM25(k1=k1, b=b)
        self.bm25.index(documents)

        # Build dense index
        self.dense = DenseRetriever(embeddings)

        self.rrf_k = rrf_k

    def search(self, query_text, query_embedding, top_k=10,
               bm25_weight=1.0, dense_weight=1.0):
        """Hybrid search with configurable component weights."""
        # Retrieve from both indices
        bm25_results = self.bm25.search(query_text, top_k=top_k * 2)
        dense_results = self.dense.search(query_embedding, top_k=top_k * 2)

        # Weighted RRF
        scores = {}
        for rank, (doc_id, _) in enumerate(bm25_results, 1):
            scores[doc_id] = scores.get(doc_id, 0.0) + \
                bm25_weight / (self.rrf_k + rank)
        for rank, (doc_id, _) in enumerate(dense_results, 1):
            scores[doc_id] = scores.get(doc_id, 0.0) + \
                dense_weight / (self.rrf_k + rank)

        ranked = sorted(scores.items(), key=lambda x: x[1], reverse=True)
        return [(doc_id, score, self.documents[doc_id]) for doc_id, score in ranked[:top_k]]

# Usage
search = HybridScientificSearch(corpus, doc_embeddings)
results = search.search(query, query_emb, top_k=3)
for rank, (doc_id, score, text) in enumerate(results, 1):
    print(f"{rank}. [{score:.4f}] {text[:70]}...")

Right Tool: LangChain and LlamaIndex for Hybrid Search

Production hybrid search pipelines benefit from the abstractions in LangChain and LlamaIndex. The entire pipeline above (BM25 + dense + RRF) becomes:

from langchain.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever
from langchain_community.vectorstores import Qdrant
from langchain_community.embeddings import HuggingFaceEmbeddings

# BM25 retriever
bm25_retriever = BM25Retriever.from_texts(corpus)
bm25_retriever.k = 10

# Dense retriever backed by Qdrant
embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
vectorstore = Qdrant.from_texts(corpus, embeddings, location=":memory:")
dense_retriever = vectorstore.as_retriever(search_kwargs={"k": 10})

# Hybrid with RRF
hybrid = EnsembleRetriever(
    retrievers=[bm25_retriever, dense_retriever],
    weights=[0.5, 0.5]  # equal weighting
)
results = hybrid.invoke("CRISPR off-target effects in human iPSCs")

Listing 3.15: LangChain EnsembleRetriever replacing the from-scratch BM25, dense retriever, and RRF implementation with a single high-level abstraction backed by Qdrant and HuggingFace embeddings.

LangChain handles tokenization, embedding, vector store management, and fusion. Line count reduction: roughly 10x. It also provides a clean path to adding reranking (Cohere Rerank, cross-encoder models (rerankers that jointly encode the query and each candidate document in a single pass, producing a more accurate relevance score at the cost of higher latency) and connecting the retriever to a retrieval-augmented generation (RAG) pipeline in Chapter 37.

5. Evaluation: Measuring Search Quality

The hybrid pipeline produces a single fused ranking, but is it better than what either retriever delivers alone? Answering that question requires standard information retrieval metrics.

A search system is only as good as its evaluation. The standard metrics for information retrieval are:

• Precision@K: fraction of the top-\(K\) results that are relevant. \(\text{P@10} = 0.7\) means 7 of the top 10 results are relevant.
• Recall@K: fraction of all relevant documents that appear in the top \(K\).
• NDCG@K (Normalized Discounted Cumulative Gain): measures ranking quality with graded relevance (relevance labels that distinguish degrees of match, such as 0 for irrelevant, 1 for somewhat relevant, and 2 for highly relevant, rather than a simple yes/no). A relevant document at rank 1 contributes more than one at rank 10. Formally:

where \(\text{rel}_i\) is the relevance grade of the document at rank \(i\) and IDCG is the DCG of the ideal ranking.

• MRR (Mean Reciprocal Rank): average of \(1/\text{rank}\) of the first relevant result across queries.

Let us implement NDCG and evaluate our hybrid system:

import numpy as np
from typing import List

def dcg_at_k(relevances: List[int], k: int) -> float:
    """Discounted Cumulative Gain at rank k."""
    relevances = relevances[:k]
    return sum((2**rel - 1) / np.log2(i + 2) for i, rel in enumerate(relevances))

def ndcg_at_k(relevances: List[int], k: int) -> float:
    """Normalized DCG at rank k."""
    dcg = dcg_at_k(relevances, k)
    ideal = dcg_at_k(sorted(relevances, reverse=True), k)
    return dcg / ideal if ideal > 0 else 0.0

def evaluate_search(search_results, relevance_judgments, k=5):
    """Evaluate a search system given results and relevance labels.

    Args:
        search_results: list of (doc_id, score) tuples
        relevance_judgments: dict mapping doc_id -> relevance grade (0, 1, 2)
        k: evaluation depth
    """
    # Build relevance vector aligned with result ranking
    rels = [relevance_judgments.get(doc_id, 0)
            for doc_id, _ in search_results[:k]]

    ndcg = ndcg_at_k(rels, k)
    precision = sum(1 for r in rels if r > 0) / k
    # Reciprocal rank of first relevant result
    mrr = 0.0
    for i, r in enumerate(rels):
        if r > 0:
            mrr = 1.0 / (i + 1)
            break

    return {"ndcg": ndcg, "precision": precision, "mrr": mrr}


# Ground-truth relevance for our query
# 2 = highly relevant, 1 = somewhat relevant, 0 = not relevant
relevance = {
    0: 1,  # CRISPR Cas9 genome editing: somewhat relevant
    1: 2,  # Off-target effects in iPSCs: highly relevant
    2: 0,  # Deep learning medical imaging: not relevant
    3: 0,  # Transformer protein prediction: not relevant
    4: 0,  # JAK-STAT pharmacokinetics: not relevant
    5: 2,  # Unintended Cas9 mutations: highly relevant
    6: 0,  # Knowledge graph embeddings: not relevant
    7: 1,  # iPSC cardiomyocytes: somewhat relevant (iPSC topic)
}

# Evaluate each method
print(f"{'Method':<15} {'NDCG@5':<10} {'P@5':<10} {'MRR':<10}")
print("-" * 45)

# BM25 only
bm25_eval = evaluate_search(bm25_results, relevance, k=5)
print(f"{'BM25':<15} {bm25_eval['ndcg']:<10.3f} {bm25_eval['precision']:<10.3f} "
      f"{bm25_eval['mrr']:<10.3f}")

# Dense only
dense_eval = evaluate_search(dense_results_list, relevance, k=5)
print(f"{'Dense':<15} {dense_eval['ndcg']:<10.3f} {dense_eval['precision']:<10.3f} "
      f"{dense_eval['mrr']:<10.3f}")

# Hybrid (RRF)
fused_as_pairs = [(doc_id, score) for doc_id, score in fused]
hybrid_eval = evaluate_search(fused_as_pairs, relevance, k=5)
print(f"{'Hybrid (RRF)':<15} {hybrid_eval['ndcg']:<10.3f} {hybrid_eval['precision']:<10.3f} "
      f"{hybrid_eval['mrr']:<10.3f}")

# Output:
# Method          NDCG@5     P@5        MRR
# ---------------------------------------------
# BM25            0.782      0.600      1.000
# Dense           0.831      0.600      1.000
# Hybrid (RRF)    0.893      0.800      1.000

6. From Search to Discovery

The hybrid search engine we have built is a foundational component of the Discovery Workbench, but search is just the beginning. The output of a search system, a ranked list of documents, becomes the input to downstream discovery tasks:

• Literature review automation (Chapter 36): given a research question, retrieve relevant papers, extract claims, and synthesize a structured review.
• Retrieval-augmented generation (Chapter 37): feed retrieved passages to a language model as context, grounding its output in evidence.
• Knowledge graph population (Chapter 38): extract entities and relations from retrieved papers to grow the knowledge graph from Section 3.2.
• Hypothesis generation (Chapter 39): combine graph structure, embedding similarity, and retrieved evidence to propose novel hypotheses.

Each of these tasks takes the representations from this chapter (ontologies, graphs, embeddings) and combines them with the reasoning techniques from Chapter 4 and the language models from Part III. The hybrid search index is the first working component of the Discovery Workbench; by Chapter 53, it will be one module in a fully autonomous scientific discovery system.