7.3 Building a Hypothesis-Driven Dev Workflow

1. The Recipe: From Vague Idea to Executable Hypotheses

Teams that skip structured decomposition often discover, months into a build, that they solved the wrong problem or optimized the wrong metric. A single afternoon of hypothesis formulation can, in many cases, prevent weeks or months of misdirected engineering. Imagine it is Monday morning and your team has exactly one sentence on the whiteboard: "Build a tool that helps researchers discover relevant papers." By Friday, that sentence will either still be a sentence, or it will be a Git repository with falsifiable claims (where "falsifiable" means each claim is stated precisely enough that a concrete experiment could prove it wrong), prioritized experiments, and skeleton tests that know precisely what "success" looks like. The difference is a five-stage pipeline, illustrated in Figure 7.3 below. Each stage produces an artifact that the next stage consumes, and a team can execute the entire sequence in a single afternoon for a new project.

1. Idea decomposition: break the vague idea into distinct uncertainty categories (user needs, technical feasibility, business value, integration constraints).
2. Hypothesis formulation: for each uncertainty, write a falsifiable claim with a test procedure and success criterion.
3. Prioritization: rank hypotheses by information value (a composite score combining how uncertain a hypothesis is with how many downstream decisions depend on its answer).
4. Test scaffolding: write skeleton pytest tests that encode the success criteria. These tests fail initially (red) and pass only when the hypothesis is confirmed through implementation (green).
5. Repository structure: organize everything in a Git repo with branches for each hypothesis experiment, a hypothesis registry file, and continuous integration (CI) that runs the test suite on every push.

The following stages use a running example: a scientific paper recommendation engine for the Discovery Workbench.

2. Stage 1: Idea Decomposition

The vague idea is: "Build a tool that helps researchers discover relevant papers they would not find through keyword search alone." This single sentence conceals at least a dozen uncertainties. We systematically decompose it using the four categories from Section 7.1.

"""Stage 1: Decompose a vague product idea into structured uncertainties.

This module provides a systematic approach to breaking down product ideas
into the four uncertainty categories that drive hypothesis generation.
"""

from dataclasses import dataclass, field
from typing import List
from enum import Enum


class UncertaintyCategory(Enum):
    USER = "user_needs"           # What do users actually want?
    TECHNICAL = "technical"       # What technical approach will work?
    VALUE = "value"               # Will this deliver measurable value?
    INTEGRATION = "integration"   # How does this fit existing workflows?


@dataclass
class Uncertainty:
    """A single uncertainty extracted from a product idea."""
    question: str
    category: UncertaintyCategory
    assumptions: List[str] = field(default_factory=list)
    impact_if_wrong: str = ""     # What happens if our assumption is incorrect?


@dataclass
class IdeaDecomposition:
    """Structured decomposition of a vague product idea."""
    original_idea: str
    uncertainties: List[Uncertainty] = field(default_factory=list)

    def add(self, question: str, category: UncertaintyCategory,
            assumptions: List[str], impact: str) -> None:
        self.uncertainties.append(Uncertainty(
            question=question,
            category=category,
            assumptions=assumptions,
            impact_if_wrong=impact
        ))

    def summary(self) -> str:
        lines = [f"Idea: {self.original_idea}", ""]
        for cat in UncertaintyCategory:
            items = [u for u in self.uncertainties if u.category == cat]
            if items:
                lines.append(f"  {cat.value} ({len(items)} uncertainties):")
                for u in items:
                    lines.append(f"    - {u.question}")
        lines.append(f"\n  Total uncertainties: {len(self.uncertainties)}")
        return "\n".join(lines)


# Decompose the paper recommendation idea
decomp = IdeaDecomposition(
    "Build a tool that helps researchers discover relevant papers "
    "they would not find through keyword search alone"
)

decomp.add(
    "Do researchers want serendipitous discovery or targeted recommendations?",
    UncertaintyCategory.USER,
    ["Researchers value novelty", "Current search tools are insufficient"],
    "Building for serendipity when users want precision wastes the entire product"
)

decomp.add(
    "Do researchers trust algorithmic recommendations for scientific reading?",
    UncertaintyCategory.USER,
    ["Trust in AI recommendations transfers from consumer to scientific domains"],
    "Low trust means low adoption regardless of recommendation quality"
)

decomp.add(
    "Which similarity metric best captures 'relevance' for scientific papers?",
    UncertaintyCategory.TECHNICAL,
    ["Embedding similarity correlates with human relevance judgments"],
    "Wrong similarity metric produces irrelevant recommendations"
)

decomp.add(
    "Can we achieve sub-200ms latency with dense retrieval over 200M papers?",
    UncertaintyCategory.TECHNICAL,
    ["Approximate nearest neighbor search scales to this corpus size"],
    "High latency kills interactive use; must fall back to batch recommendations"
)

decomp.add(
    "Will recommendations lead to papers being read, or just bookmarked?",
    UncertaintyCategory.VALUE,
    ["Click-through rate is a valid proxy for reading"],
    "Optimizing for clicks produces clickbait, not useful recommendations"
)

decomp.add(
    "Can we integrate with Zotero and Mendeley without breaking their APIs?",
    UncertaintyCategory.INTEGRATION,
    ["Reference manager APIs are stable and well-documented"],
    "No integration means users must manually track recommendations"
)

print(decomp.summary())

3. Stage 2: Hypothesis Formulation

Each uncertainty becomes one or more hypotheses using the DevHypothesis class from Section 7.1. The critical discipline is making each hypothesis falsifiable: it must be possible to run an experiment whose outcome could prove the hypothesis wrong. "Users will like our recommendations" is not falsifiable. "At least 20% of users will click on at least one recommendation per session" is falsifiable.

"""Stage 2: Convert uncertainties into falsifiable DevHypotheses.

Each hypothesis must have:
- A specific, falsifiable claim
- A concrete test procedure achievable within one sprint
- A measurable success criterion with a numerical threshold
"""

import json
from pathlib import Path
from datetime import datetime
from dataclasses import dataclass, field, asdict
from typing import List


@dataclass
class DevHypothesis:
    """A falsifiable development hypothesis (reproduced from Section 7.1)."""
    id: str
    claim: str
    category: str
    test_procedure: str
    success_criterion: str
    confidence: float = 0.5
    evidence: list = field(default_factory=list)
    status: str = "untested"
    created_at: str = field(default_factory=lambda: datetime.now().isoformat())
    branch: str = ""              # Git branch for this experiment
    depends_on: List[str] = field(default_factory=list)

    def update_confidence(self, observation: str, confirms: bool,
                          strength: float = 0.1) -> float:
        if confirms:
            self.confidence += strength * (1.0 - self.confidence)
        else:
            self.confidence -= strength * self.confidence

        self.evidence.append({
            "observation": observation,
            "confirms": confirms,
            "strength": strength,
            "timestamp": datetime.now().isoformat(),
            "confidence_after": round(self.confidence, 4)
        })

        if self.confidence > 0.9:
            self.status = "confirmed"
        elif self.confidence < 0.1:
            self.status = "refuted"
        else:
            self.status = "testing"
        return self.confidence


@dataclass
class HypothesisRegistry:
    """A persistent registry of all hypotheses for a project.

    Stored as a JSON file in the repository root, this serves as the
    project's 'lab notebook' tracking what the team believes and why.
    """
    project_name: str
    hypotheses: List[DevHypothesis] = field(default_factory=list)

    def add(self, hypothesis: DevHypothesis) -> None:
        self.hypotheses.append(hypothesis)

    def save(self, path: Path) -> None:
        """Persist the registry to a JSON file (committed to Git)."""
        data = {
            "project": self.project_name,
            "updated_at": datetime.now().isoformat(),
            "hypotheses": [asdict(h) for h in self.hypotheses]
        }
        path.write_text(json.dumps(data, indent=2))

    @classmethod
    def load(cls, path: Path) -> "HypothesisRegistry":
        """Load registry from a JSON file."""
        data = json.loads(path.read_text())
        registry = cls(project_name=data["project"])
        for h_data in data["hypotheses"]:
            registry.add(DevHypothesis(**h_data))
        return registry

    def active(self) -> List[DevHypothesis]:
        return [h for h in self.hypotheses if h.status in ("untested", "testing")]

    def dashboard(self) -> str:
        """Print a summary dashboard of hypothesis status."""
        lines = [f"Hypothesis Registry: {self.project_name}", "=" * 60]
        status_counts = {}
        for h in self.hypotheses:
            status_counts[h.status] = status_counts.get(h.status, 0) + 1

        for status, count in sorted(status_counts.items()):
            lines.append(f"  {status:<12}: {count}")

        lines.append(f"  {'total':<12}: {len(self.hypotheses)}")
        lines.append("")

        for h in self.hypotheses:
            marker = {"untested": "[ ]", "testing": "[~]",
                      "confirmed": "[+]", "refuted": "[-]"}.get(h.status, "[?]")
            lines.append(f"  {marker} {h.id}: {h.claim[:55]}...")
            lines.append(f"      confidence: {h.confidence:.2f}  |  "
                         f"evidence: {len(h.evidence)} observations")
        return "\n".join(lines)


# Build the registry for our paper recommendation project
registry = HypothesisRegistry("paper-recommender")

registry.add(DevHypothesis(
    id="H-001",
    claim="At least 20% of researchers click on a recommendation per session",
    category="value",
    test_procedure="Deploy MVP to 100 beta users for 2 weeks; track CTR",
    success_criterion="Session CTR >= 0.20",
    confidence=0.5,
    branch="experiment/h001-ctr-baseline"
))

registry.add(DevHypothesis(
    id="H-002",
    claim="SPECTER2 embeddings produce more relevant results than BM25",
    category="implementation",
    test_procedure="Offline evaluation on TREC-COVID benchmark",
    success_criterion="SPECTER2 nDCG@10 exceeds BM25 nDCG@10 by >0.05",
    confidence=0.65,
    branch="experiment/h002-specter-vs-bm25",
    depends_on=[]
))

registry.add(DevHypothesis(
    id="H-003",
    claim="Approximate nearest neighbor search achieves p95 < 200ms on 50M papers",
    category="implementation",
    test_procedure="Benchmark FAISS HNSW index with 50M SPECTER2 vectors",
    success_criterion="p95 latency < 200ms with recall@10 > 0.95",
    confidence=0.7,
    branch="experiment/h003-faiss-latency",
    depends_on=["H-002"]  # Only relevant if we use embeddings
))

registry.add(DevHypothesis(
    id="H-004",
    claim="Researchers prefer citation-augmented recs over pure content similarity",
    category="requirement",
    test_procedure="Within-subjects A/B test: 50 users rate both approaches",
    success_criterion="Mean preference score for citation-augmented > 3.5/5.0",
    confidence=0.55,
    branch="experiment/h004-citation-augmented",
    depends_on=["H-002"]
))

registry.add(DevHypothesis(
    id="H-005",
    claim="Zotero plugin integration increases weekly active usage by >30%",
    category="integration",
    test_procedure="Release Zotero plugin to 50% of beta users; compare WAU",
    success_criterion="WAU ratio (with plugin / without) > 1.30",
    confidence=0.45,
    branch="experiment/h005-zotero-integration",
    depends_on=["H-001"]  # Only matters if baseline engagement exists
))

print(registry.dashboard())

Several hypotheses above reference domain-specific tools: SPECTER2 is a transformer-based embedding model trained specifically on scientific documents; BM25 is a classic term-frequency retrieval baseline widely used in search engines; and nDCG@10 (normalized discounted cumulative gain at rank 10) is a standard metric that measures how well a ranked list places relevant results near the top. You do not need deep familiarity with these tools to follow the workflow; what matters is that each hypothesis names a concrete method and a measurable threshold.

4. Stage 3: Prioritization by Information Value

With five hypotheses in the registry, the team must decide which to test first. The prioritization logic from Section 7.2 combines uncertainty with dependency fan-out. Uncertainty is quantified here using binary entropy, an information-theory measure that peaks at 1.0 when confidence is 50/50 and drops toward 0 as confidence approaches certainty in either direction. The formula then factors in test cost: a cheap test with moderate information value often beats an expensive test with marginally higher information value.

import numpy as np
from typing import List, Tuple


def information_value(hypothesis: DevHypothesis,
                      all_hypotheses: List[DevHypothesis]) -> float:
    """Compute the information value of testing a hypothesis.

    Combines three factors:
    1. Uncertainty: binary entropy, maximized when confidence = 0.5
    2. Fan-out: number of other hypotheses that depend on this one
    3. Category weight: requirement and value hypotheses get a bonus
       because they can redirect the entire project
    """
    # Factor 1: Uncertainty (0 to 1)
    c = np.clip(hypothesis.confidence, 1e-10, 1 - 1e-10)
    uncertainty = -(c * np.log2(c) + (1 - c) * np.log2(1 - c))

    # Factor 2: Dependency fan-out
    dependents = sum(
        1 for h in all_hypotheses
        if hypothesis.id in h.depends_on
    )
    fan_out = 1.0 + 0.4 * dependents

    # Factor 3: Category weight
    category_weights = {
        "requirement": 1.5,  # Can redirect the whole project
        "value": 1.3,        # Determines if the project is worth doing
        "implementation": 1.0,
        "integration": 0.8,
    }
    cat_weight = category_weights.get(hypothesis.category, 1.0)

    return uncertainty * fan_out * cat_weight


def prioritize(registry: HypothesisRegistry) -> List[Tuple[float, DevHypothesis]]:
    """Rank active hypotheses by information value, descending."""
    active = registry.active()
    scored = [
        (information_value(h, registry.hypotheses), h)
        for h in active
    ]
    scored.sort(key=lambda x: x[0], reverse=True)
    return scored


ranked = prioritize(registry)

print("Hypothesis Priority Queue")
print("=" * 70)
print(f"{'Rank':<6} {'ID':<8} {'InfoVal':>8} {'Conf':>6} {'Category':<16} {'Claim'}")
print("-" * 70)
for i, (score, h) in enumerate(ranked, 1):
    print(f"{i:<6} {h.id:<8} {score:>8.3f} {h.confidence:>5.2f} "
          f"{h.category:<16} {h.claim[:35]}...")

5. Stage 4: Skeleton Tests as Falsification Instruments

A central practice in hypothesis-driven development is writing tests before implementation, not as a coding discipline (TDD) but as a scientific discipline. Each test encodes the success criterion of a hypothesis. When the test passes, the hypothesis has survived falsification. When it fails, the hypothesis is either refuted or the implementation is incomplete.

Because a failing test in this framework carries a fundamentally different meaning than in conventional TDD, the tests themselves need a different structure, one designed to encode falsifiable criteria rather than implementation contracts.

Skeleton Tests: Shape Before Substance

We call these skeleton tests because they define the shape of the expected behavior without implementing the internals. They are written at the same time as the hypothesis, before any production code exists. This practice forces the team to make success criteria concrete and measurable before investing in implementation.

A skeleton test is a fully structured test function with imports, assertions, and threshold constants. Its body references modules and classes that do not yet exist. The developer marks it pytest.mark.skip or xfail so the suite stays green while production code is absent. Skeleton tests translate a vague success criterion ("embedding quality should be good enough") into an unambiguous, machine-checkable contract before writing a single line of production code. This eliminates the common failure mode where the team builds first and retrofits acceptance criteria later. When a developer begins work on a hypothesis, they remove the skip marker, run the test, watch it fail (red), implement until it passes (green), and then record the result as evidence in the hypothesis registry. Use skeleton tests whenever the success criterion is quantitative and automatable; for qualitative criteria (such as user interview insights or design review feedback), record evidence manually in the registry instead.

"""
Stage 4: Skeleton tests for each hypothesis.

File: tests/test_hypotheses.py

These tests encode the success criteria from the hypothesis registry.
Each test is initially marked with pytest.mark.skip or xfail because
the implementation does not exist yet. As implementation proceeds,
tests are unskipped and expected to pass.
"""

import pytest
import numpy as np


# --------------- H-002: SPECTER2 vs BM25 ---------------

class TestH002_EmbeddingQuality:
    """Hypothesis: SPECTER2 produces more relevant results than BM25.
    Success criterion: nDCG@10 improvement > 0.05 on TREC-COVID."""

    NDCG_IMPROVEMENT_THRESHOLD = 0.05

    @pytest.fixture
    def trec_covid_queries(self):
        """Load TREC-COVID benchmark queries and relevance judgments."""
        # In production, this loads from the actual dataset
        # Skeleton returns a minimal test fixture
        return {
            "queries": [
                "coronavirus origin",
                "COVID-19 vaccine efficacy",
                "SARS-CoV-2 transmission dynamics",
            ],
            "qrels": {
                "coronavirus origin": {"doc_001": 2, "doc_015": 1, "doc_042": 0},
                "COVID-19 vaccine efficacy": {"doc_007": 2, "doc_023": 1},
                "SARS-CoV-2 transmission dynamics": {"doc_003": 2, "doc_011": 2},
            }
        }

    @pytest.mark.skip(reason="H-002: Awaiting SPECTER2 and BM25 retriever implementations")
    def test_specter2_beats_bm25_ndcg(self, trec_covid_queries):
        """SPECTER2 nDCG@10 must exceed BM25 nDCG@10 by threshold."""
        from paper_recommender.retrievers import SPECTER2Retriever, BM25Retriever
        from paper_recommender.evaluation import compute_ndcg

        specter = SPECTER2Retriever()
        bm25 = BM25Retriever()

        specter_scores = []
        bm25_scores = []

        for query in trec_covid_queries["queries"]:
            qrels = trec_covid_queries["qrels"][query]
            specter_results = specter.retrieve(query, k=10)
            bm25_results = bm25.retrieve(query, k=10)

            specter_scores.append(compute_ndcg(specter_results, qrels, k=10))
            bm25_scores.append(compute_ndcg(bm25_results, qrels, k=10))

        improvement = np.mean(specter_scores) - np.mean(bm25_scores)

        assert improvement > self.NDCG_IMPROVEMENT_THRESHOLD, (
            f"SPECTER2 improvement ({improvement:.4f}) did not exceed "
            f"threshold ({self.NDCG_IMPROVEMENT_THRESHOLD}). "
            f"Hypothesis H-002 may need to be refuted."
        )


# --------------- H-003: FAISS Latency ---------------

class TestH003_RetrievalLatency:
    """Hypothesis: ANN search achieves p95 < 200ms on 50M vectors.
    Success criterion: p95 latency < 200ms with recall@10 > 0.95."""

    LATENCY_P95_MS = 200
    RECALL_THRESHOLD = 0.95

    @pytest.mark.skip(reason="H-003: Awaiting FAISS index with 50M vectors")
    def test_faiss_latency_under_threshold(self):
        """p95 query latency must be under 200ms."""
        import time
        from paper_recommender.index import FAISSIndex

        index = FAISSIndex.load("indices/specter2_50m.faiss")
        query_vectors = np.random.randn(1000, 768).astype(np.float32)

        latencies = []
        for qv in query_vectors:
            start = time.perf_counter()
            index.search(qv.reshape(1, -1), k=10)
            elapsed_ms = (time.perf_counter() - start) * 1000
            latencies.append(elapsed_ms)

        p95 = np.percentile(latencies, 95)
        assert p95 < self.LATENCY_P95_MS, (
            f"p95 latency ({p95:.1f}ms) exceeds threshold ({self.LATENCY_P95_MS}ms). "
            f"H-003 refuted: need to optimize index or reduce corpus size."
        )

    @pytest.mark.skip(reason="H-003: Awaiting FAISS index with 50M vectors")
    def test_faiss_recall_above_threshold(self):
        """Approximate search recall@10 must exceed 0.95 vs exact search."""
        from paper_recommender.index import FAISSIndex

        index = FAISSIndex.load("indices/specter2_50m.faiss")
        # Compare approximate results against exact brute-force results
        query_vectors = np.random.randn(100, 768).astype(np.float32)

        recalls = []
        for qv in query_vectors:
            approx_ids = set(index.search(qv.reshape(1, -1), k=10)[1][0])
            exact_ids = set(index.search_exact(qv.reshape(1, -1), k=10)[1][0])
            recalls.append(len(approx_ids & exact_ids) / 10)

        mean_recall = np.mean(recalls)
        assert mean_recall > self.RECALL_THRESHOLD, (
            f"Mean recall ({mean_recall:.3f}) below threshold ({self.RECALL_THRESHOLD}). "
            f"H-003 partially refuted: latency may be fine but recall is insufficient."
        )


# --------------- H-001: User Engagement ---------------

class TestH001_UserEngagement:
    """Hypothesis: At least 20% of researchers click on a recommendation.
    Success criterion: session CTR >= 0.20."""

    CTR_THRESHOLD = 0.20

    @pytest.mark.skip(reason="H-001: Awaiting beta deployment and 2-week data collection")
    def test_session_ctr_above_threshold(self):
        """Session click-through rate must exceed 20%."""
        from paper_recommender.analytics import load_engagement_data

        data = load_engagement_data("beta_v1", min_days=14)
        sessions_with_clicks = sum(1 for s in data if s["clicked_any"])
        total_sessions = len(data)
        ctr = sessions_with_clicks / total_sessions

        assert ctr >= self.CTR_THRESHOLD, (
            f"Session CTR ({ctr:.2%}) below threshold ({self.CTR_THRESHOLD:.0%}). "
            f"H-001 refuted: users are not engaging with recommendations."
        )

        # Also check statistical significance
        from scipy import stats
        # One-sample proportion test against threshold
        z_stat = (ctr - self.CTR_THRESHOLD) / np.sqrt(
            self.CTR_THRESHOLD * (1 - self.CTR_THRESHOLD) / total_sessions
        )
        p_value = 1 - stats.norm.cdf(z_stat)
        assert p_value < 0.05, (
            f"CTR exceeds threshold but is not statistically significant "
            f"(p={p_value:.4f}). Collect more data before confirming H-001."
        )

Fun Note: Red Means Learning

In traditional TDD, a red test means "something is broken." In hypothesis-driven development, a red test means "we learned something." When test test_specter2_beats_bm25_ndcg fails because the improvement is only 0.03 (below the 0.05 threshold), that is not a bug. It is evidence. The hypothesis is refuted, and the team now knows to explore a different retrieval approach. The test did its job: it falsified a hypothesis cheaply, before the team built an entire product around the wrong embedding model. The most valuable test is often the one that fails.

6. Stage 5: Git as a Lab Notebook

The final stage organizes everything into a Git repository that serves dual duty as a lab notebook (recording what was tried and what was learned) and a production codebase (containing the software that survived testing). The branching strategy directly maps to the hypothesis framework.

"""
Stage 5: Repository structure and Git workflow for hypothesis-driven development.

This module provides utilities for managing the Git-based hypothesis workflow.
It creates branches for experiments, tracks results in commits, and generates
a development log from the Git history.
"""

import subprocess
import json
from pathlib import Path
from typing import Optional


class HypothesisGitWorkflow:
    """Manages Git operations for hypothesis-driven development.

    Branch naming convention:
      - main:                       production-ready, confirmed code
      - experiment/: experimental branch for testing
      - feature/:      confirmed hypothesis, being productionized

    Commit message convention:
      - [H-XXX] hypothesis: 
      - [H-XXX] evidence: 
      - [H-XXX] confirmed: 

      - [H-XXX] refuted: 

    """

    def __init__(self, repo_path: Path):
        self.repo_path = repo_path

    def _git(self, *args: str) -> str:
        """Run a git command and return stdout."""
        result = subprocess.run(
            ["git", *args],
            cwd=self.repo_path,
            capture_output=True, text=True, check=True
        )
        return result.stdout.strip()

    def start_experiment(self, hypothesis_id: str, claim: str) -> str:
        """Create a new experiment branch and commit the hypothesis."""
        branch = f"experiment/{hypothesis_id.lower()}"
        self._git("checkout", "-b", branch)
        self._git("commit", "--allow-empty",
                  "-m", f"[{hypothesis_id}] hypothesis: {claim}")
        return branch

    def record_evidence(self, hypothesis_id: str,
                        observation: str, confirms: bool) -> None:
        """Commit an evidence observation to the experiment branch."""
        verdict = "supports" if confirms else "contradicts"
        self._git("add", "-A")
        self._git("commit", "-m",
                  f"[{hypothesis_id}] evidence ({verdict}): {observation}")

    def conclude_experiment(self, hypothesis_id: str,
                            confirmed: bool, summary: str) -> None:
        """Mark the experiment as concluded with a final commit."""
        status = "confirmed" if confirmed else "refuted"
        self._git("add", "-A")
        self._git("commit", "-m",
                  f"[{hypothesis_id}] {status}: {summary}")

        if confirmed:
            # Merge confirmed experiments into main
            self._git("checkout", "main")
            self._git("merge", f"experiment/{hypothesis_id.lower()}",
                      "--no-ff", "-m",
                      f"Merge confirmed experiment {hypothesis_id}: {summary}")

    def experiment_log(self, hypothesis_id: Optional[str] = None) -> str:
        """Extract the development log for one or all hypotheses."""
        pattern = f"[{hypothesis_id}]" if hypothesis_id else "[H-"
        log = self._git("log", "--oneline", "--all",
                        f"--grep={pattern}")
        return log


# Recommended repository structure
REPO_STRUCTURE = """
paper-recommender/
    hypotheses.json              # Hypothesis registry (committed to Git)
    pyproject.toml               # Project metadata and dependencies
    src/
        paper_recommender/
            __init__.py
            retrievers.py        # BM25, SPECTER2 retriever implementations
            index.py             # FAISS index management
            evaluation.py        # nDCG, recall, latency benchmarks
            analytics.py         # User engagement tracking
    tests/
        test_hypotheses.py       # Skeleton tests encoding success criteria
        test_unit.py             # Standard unit tests
        conftest.py              # Shared fixtures
    experiments/
        h002_specter_vs_bm25/    # Data and notebooks for each experiment
            benchmark_results.json
            analysis.ipynb
        h003_faiss_latency/
            latency_profile.json
    .github/
        workflows/
            hypothesis_ci.yml    # CI that runs hypothesis tests on push
        ISSUE_TEMPLATE/
            hypothesis.md        # GitHub Issue template for new hypotheses
    docs/
        decisions/               # Architecture Decision Records
            001-embedding-choice.md
"""

print(REPO_STRUCTURE)

Notice the branch naming convention: when conclude_experiment confirms a hypothesis, it merges the experiment branch into main. At that point the confirmed hypothesis becomes a user story for production hardening. The team creates a feature/ branch from the merged code, writes conventional unit and integration tests around it, adds documentation, and treats the confirmed behavior as a committed product requirement rather than an open question. This is the step where discovery hands off to delivery: the hypothesis provided the evidence that the feature is worth building, and the feature branch provides the engineering rigor to build it well.

7. Putting It All Together: The CI Pipeline

A well-organized repository captures the structure of the experiment, but without automation the team must remember to run tests manually after every change, and manual discipline erodes within weeks.

The final piece is a continuous integration pipeline that runs the hypothesis tests on every push. This is the automated falsification engine: whenever a developer pushes code to an experiment branch, CI runs the relevant skeleton tests and reports whether the success criteria are met. In short: the repository that tests its own beliefs, not just its own code, is the one that finds the wrong turn before the team has driven a hundred miles down it.

"""
GitHub Actions workflow for hypothesis-driven CI.

File: .github/workflows/hypothesis_ci.yml (shown as Python string for syntax highlighting)

This workflow:
1. Runs all unskipped hypothesis tests on every push
2. Updates the hypothesis registry with test results
3. Posts a comment on the PR with the hypothesis status dashboard
"""

HYPOTHESIS_CI_YAML = """
name: Hypothesis CI

on:
  push:
    branches: ['experiment/**']
  pull_request:
    branches: [main]

jobs:
  test-hypotheses:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Install dependencies
        run: pip install -e ".[test]"

      - name: Run hypothesis tests
        run: |
          pytest tests/test_hypotheses.py -v \\
            --tb=short \\
            --junitxml=hypothesis-results.xml \\
            -k "not skip"

      - name: Generate hypothesis dashboard
        if: always()
        run: |
          python -c "
          from paper_recommender.registry import HypothesisRegistry
          from pathlib import Path
          registry = HypothesisRegistry.load(Path('hypotheses.json'))
          print(registry.dashboard())
          "

      - name: Comment PR with hypothesis status
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const dashboard = fs.readFileSync('hypothesis-dashboard.txt', 'utf8');
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: '## Hypothesis Dashboard\\n```\\n' + dashboard + '\\n```'
            });
"""

print(HYPOTHESIS_CI_YAML)

8. Extending the Discovery Workbench

The hypothesis registry and Git workflow from this section become a permanent component of the Discovery Workbench, the platform that grows throughout this book. Specifically, we add a DevTracker module that integrates with the Workbench's existing architecture from Chapter 6.

"""
Discovery Workbench extension: DevTracker module.

Integrates hypothesis-driven development tracking into the
Discovery Workbench platform. This module will be extended in
subsequent chapters with AI-assisted hypothesis generation (Ch 8),
vibe coding integration (Ch 9), and automated experiment design (Ch 46).
"""

from dataclasses import dataclass, field
from typing import List, Dict, Optional
from pathlib import Path
import json


@dataclass
class DevTracker:
    """Development hypothesis tracker for the Discovery Workbench.

    Manages the lifecycle of development hypotheses, from formulation
    through testing to resolution. Connects to the Workbench's
    experiment registry and knowledge graph components.
    """
    project_name: str
    registry_path: Path
    _registry: Optional[object] = field(default=None, repr=False)

    def initialize(self, idea: str) -> Dict:
        """Start a new project from a vague idea.

        Returns a structured decomposition ready for hypothesis formulation.
        This is Stage 1 of the five-stage pipeline.
        """
        return {
            "project": self.project_name,
            "original_idea": idea,
            "uncertainty_categories": {
                "user_needs": [],
                "technical": [],
                "value": [],
                "integration": [],
            },
            "status": "decomposition_needed"
        }

    def compute_sprint_plan(self, budget_hours: float = 80) -> List[Dict]:
        """Generate a prioritized sprint plan from active hypotheses.

        Allocates the sprint budget across hypothesis experiments,
        balancing exploration (testing uncertain hypotheses) with
        exploitation (building on confirmed ones).
        """
        # Load registry and prioritize
        # (Implementation builds on Listings 7.12 and 7.13)
        plan = []
        explore_budget = budget_hours * 0.35  # 35% starting ratio; adjust per team maturity
        exploit_budget = budget_hours * 0.65

        # Exploration: test highest-information-value hypotheses
        plan.append({
            "phase": "explore",
            "budget_hours": explore_budget,
            "goal": "Test top-priority untested hypotheses",
            "hypotheses": []  # Populated from priority queue
        })

        # Exploitation: build features from confirmed hypotheses
        plan.append({
            "phase": "exploit",
            "budget_hours": exploit_budget,
            "goal": "Implement features from confirmed hypotheses",
            "hypotheses": []  # Populated from confirmed queue
        })

        return plan

    def generate_report(self) -> str:
        """Generate a discovery progress report for the sprint retrospective."""
        return (
            f"Discovery Report: {self.project_name}\n"
            f"Registry: {self.registry_path}\n"
            f"Status: Active\n"
            # Full implementation reads from the hypothesis registry
            # and computes discovery velocity, entropy reduction, etc.
        )

Exercises

1. (Conceptual) The hypothesis registry in Listing 7.12 uses a simple Bayesian update rule. Describe two limitations of this approach compared to a full Bayesian model with proper likelihood functions. How would you design a more rigorous confidence update that accounts for different types of evidence (user interviews vs. A/B tests vs. offline benchmarks)?
2. (Coding) Implement the complete five-stage pipeline (shown in Figure 7.3) for a project of your choice. Start with a vague idea, decompose it into at least six uncertainties, formulate hypotheses, prioritize them, and write skeleton pytest tests. Push the result to a Git repository with proper branching. Run pytest --co (collect only) to verify that all tests are discovered and properly skipped.
3. (Analysis) The prioritization function in Listing 7.13 uses a multiplicative combination of uncertainty, fan-out, and category weight. Design an experiment to test whether this formula produces better outcomes than a simpler "highest uncertainty first" rule. Define your success criterion, describe the simulation setup, and predict which approach will win under high-uncertainty vs. low-uncertainty conditions.