Part II: Discovery Through Software Engineering and Vibe Coding
Chapter 7: Software Development As A Discovery Process

Software Development As A Discovery Process

"Every sprint is an experiment. The backlog is a hypothesis queue. And the retrospective is peer review, except nobody reads the paper."

A Scrum Master Who Discovered the Scientific Method

Overview

Software development is not manufacturing. It is discovery. Every feature begins as an uncertain guess about what users need, how the system should behave, and which technical approach will survive contact with production. The software development life cycle (SDLC) is, at its core, an iterative process of forming hypotheses, building experiments (code), running those experiments (tests, deployments, user feedback), and updating beliefs. This chapter makes that analogy rigorous.

We formalize the SDLC using the search framework from Chapter 1, where the state space $S$ consists of possible software systems, actions $A$ are development activities (writing code, refactoring, deploying), and the objective $f$ is a composite of user satisfaction, reliability, and maintainability. We show that agile methodologies, lean startup principles, and design thinking are all instances of hypothesis-driven search with different exploration strategies. We then quantify discovery velocity, the rate at which a team reduces uncertainty about what to build and how, and derive the expected regret of greedy development (building the first idea that seems plausible without exploring alternatives).

The tools of this chapter are Git, GitHub, and pytest: the instruments that let you version your hypotheses, collaborate on experiments, and automate falsification. By the end, you will have a complete recipe for transforming a vague product idea into falsifiable hypotheses, user stories, and skeleton tests, all tracked in a repository that serves as both a lab notebook and a production system.

Prerequisites

This chapter assumes you have read Chapter 1: Discovery As Search (for the search framework and regret analysis), Chapter 2: Scientific Discovery and Knowledge Creation (for the hypothesis testing paradigm), and Chapter 6: Discovery System Architecture (for the Discovery Workbench scaffold). Familiarity with basic Git operations (commit, branch, merge) and Python testing (assert statements, running pytest) is helpful but not strictly required; we build from first principles.

Learning Outcomes

Sections

7.1 SDLC as Iterative Hypothesis Testing

Formalizing the SDLC within the $(S, A, T, f, C)$ search framework. Requirements as prior distributions, code commits as state transitions, test suites as falsification oracles. Defining discovery velocity and deriving the expected regret of greedy (waterfall) versus exploratory (iterative) development.

7.2 Engineering Discovery and AI

Agile, lean, and design thinking as exploration strategies. How AI reshapes the economics of software discovery. LLM-assisted hypothesis generation, automated testing, and AI pair programming as uncertainty reducers.

7.3 Building a Hypothesis-Driven Dev Workflow

Recipe: from vague idea to falsifiable hypotheses, user stories, skeleton tests, and a living repository. Git as a lab notebook. pytest as an automated falsification engine. Extending the Discovery Workbench with a development tracker.

What's Next

Chapter 8: Foundations of AI Assisted Software Engineering introduces the AI tools that accelerate every phase of the development cycle. Large language models, code generation systems, and AI assistants change the cost structure of exploration, making it feasible to test more hypotheses per sprint than any human team could attempt alone. The discovery velocity metric we define here becomes the primary evaluation criterion for the AI coding tools in Chapters 8 through 24.

Bibliography

Foundational Papers

Boehm, B. W. (2003). Value-based software engineering. ACM SIGSOFT Software Engineering Notes, 28(2), 1-12.

Articulates the value-driven perspective on software engineering that underpins the economic analysis of discovery velocity in Section 7.1.

Boehm, B. W. (1986). A spiral model of software development and enhancement. ACM SIGSOFT Software Engineering Notes, 11(4), 14-24.

The spiral model introduced risk-driven iteration into the SDLC, a direct precursor to hypothesis-driven development.

Fan, A., et al. (2023). Large language models for software engineering: survey and open problems. arXiv:2310.03533.

A comprehensive survey of LLM applications in software engineering, providing context for Section 7.2's discussion of AI-accelerated discovery.

Books

Ries, E. (2011). The Lean Startup. Crown Business.

The canonical reference for build-measure-learn cycles, minimum viable products, and validated learning in product development.

Humble, J. & Farley, D. (2010). Continuous Delivery. Addison-Wesley.

Defines the deployment pipeline as an automated experiment runner, directly supporting the hypothesis-testing view of SDLC.

Brown, T. (2009). Change by Design. Harper Business.

Tim Brown's articulation of design thinking as an iterative, human-centered discovery process for product innovation.

Tools & Libraries

Git

The distributed version control system used throughout this chapter as a hypothesis versioning and branching mechanism.

pytest

Python's most popular testing framework, used here as an automated falsification engine for development hypotheses.

GitHub

The collaboration platform whose pull requests, issues, and actions pipelines provide infrastructure for team-based hypothesis-driven development.

Tutorials & Surveys

Zhang, C., et al. (2023). A survey on large language models for software engineering. arXiv:2312.15231.

Surveys the landscape of LLM-assisted software engineering, covering code generation, testing, debugging, and documentation.

Dybå, T. & Dingsøyr, T. (2008). Empirical studies of agile software development: a systematic review. Information and Software Technology, 50(9-10), 833-859.

A systematic review of empirical evidence for agile methods, relevant to our analysis of agile as a discovery strategy.

Hou, X., et al. (2024). Large language models for software engineering: a systematic literature review. ACM Transactions on Software Engineering and Methodology.

The most comprehensive recent review of LLMs in software engineering, covering 229 papers across code generation, testing, and maintenance.