Chapter 13: Discovery of Requirements

Overview

Requirements are the foundation of every software system, yet they are among the hardest artifacts to get right. Studies consistently show that requirements defects account for 40% to 60% of all defects in delivered software, and that fixing a requirements error after deployment costs 50 to 200 times more than fixing it during elicitation. The problem is fundamentally one of discovery: requirements do not exist as finished artifacts waiting to be collected. They must be excavated from stakeholder conversations, domain documents, support tickets, and existing codebases, then validated for completeness, consistency, and testability.

This chapter treats requirements engineering as a search problem in the framework introduced in Chapter 1. The search space $S$ is the set of all possible requirements for a system. The actions $A$ include interviewing stakeholders, analyzing transcripts, examining existing systems, and reviewing domain literature. The transition function $T$ maps each elicitation action to a refined set of candidate requirements. The objective function $f$ measures requirement quality: completeness, consistency, unambiguity, and testability. The constraints $C$ encode budget, timeline, regulatory mandates, and technical feasibility.

We will build AI assistants that automate the most labor-intensive parts of this search: extracting structured requirements from unstructured text, clustering related requirements, detecting conflicts, and maintaining traceability. By the end of the chapter, you will have a working Requirement Discovery Assistant that ingests stakeholder transcripts, produces validated user stories with acceptance criteria, and flags conflicts using graph-based satisfiability analysis. This assistant becomes a component of the Discovery Workbench, connecting to the MCP servers from Chapter 12 and the architecture discovery pipeline in Chapter 14.

Prerequisites

You should have read Chapter 9: Vibe Coding for the basics of steering LLMs through specification, and Chapter 10: Prompting to Programming for structured output extraction with LLMs. Familiarity with Python dataclasses or Pydantic models is assumed. Chapter 11: Context Engineering provides the context-delivery strategies that make LLM-based extraction reliable at scale. Background in graph theory is helpful for the conflict detection material but not strictly required.

Learning Outcomes

Distinguish user stories, use cases, and acceptance criteria as complementary requirement representations, and know when each is appropriate.
Use LLMs to extract structured requirements from unstructured stakeholder transcripts, support tickets, and domain documents.
Validate requirements for completeness, consistency, unambiguity, and testability using automated checks and LLM-assisted review.
Build and query a traceability matrix that links requirements to their sources, tests, and implementation artifacts.
Model requirement conflicts as a constraint satisfaction problem and detect them using graph analysis with NetworkX.
Implement a complete Requirement Discovery Assistant that processes raw transcripts into validated, conflict-checked user stories.

Sections

13.1 Stakeholders and Requirements

User stories, use cases, and acceptance criteria as structured representations of stakeholder needs. The anatomy of a well-formed requirement. Quality attributes: completeness, consistency, unambiguity, testability. The traceability matrix as a discovery artifact.

13.2 AI-Assisted Requirement Extraction

LLM-based extraction of requirements from transcripts, tickets, and documents. Prompt engineering for structured output. Clustering and deduplication with embeddings. Conflict detection as constraint satisfiability over requirement graphs using NetworkX.

13.3 Building a Requirement Discovery Assistant

Recipe: a complete pipeline that ingests stakeholder transcripts, extracts and clusters requirements, validates quality attributes, detects conflicts, and produces a traceability matrix. Integration with the Discovery Workbench.

What's Next

Requirements tell you what a system should do. Chapter 14: Discovery of Architectures tackles the next question: how should the system be structured to satisfy those requirements? Architecture discovery takes the validated requirements from this chapter as input and explores the design space of components, connectors, and deployment topologies. The traceability matrix you build here will extend into Chapter 14 to link each requirement to the architectural decisions that address it, closing the loop between stakeholder needs and system structure. Together, Chapters 13 and 14 form the "what and how" of AI-assisted software design, grounding the vibe coding practices from Chapter 9 in disciplined engineering methodology.

Bibliography

Foundational Papers

Zave, P. & Jackson, M. (1997). Four dark corners of requirements engineering. ACM Transactions on Software Engineering and Methodology, 6(1), 1-30.

A foundational analysis of the theoretical difficulties in requirements engineering, including the distinction between requirements (about the world) and specifications (about the machine). Essential framing for understanding why requirements discovery is hard.

Zhang, J., et al. (2023). Requirements engineering using large language models: A systematic literature review. arXiv:2310.06135.

A comprehensive survey of LLM applications in requirements engineering: extraction, classification, quality assessment, and conflict detection. Maps the current state of AI-assisted RE research.

Dalpiaz, F. & Brinkkemper, S. (2020). Agile requirements engineering. IEEE Software, 37(4), 56-61.

Connects agile user-story practices with traditional requirements engineering, arguing that the two complement rather than replace each other. Motivates the hybrid approach used in this chapter.

Books

Wiegers, K. & Beatty, J. (2013). Software Requirements, 3rd Edition. Microsoft Press.

The standard practitioner reference for requirements engineering. Covers elicitation techniques, specification formats, validation, and management. The user-story and use-case templates in this chapter follow Wiegers's conventions.

Pohl, K. (2010). Requirements Engineering: Fundamentals, Principles, and Techniques. Springer.

A rigorous academic treatment of requirements engineering with formal definitions of completeness, consistency, and traceability. Provides the theoretical grounding for the quality attributes we automate.

Tools & Libraries

Hagberg, A., Schult, D. & Swart, P. (2008). NetworkX: Network analysis in Python.

The graph library used throughout this chapter for building requirement dependency graphs, detecting conflicts via cycle analysis, and computing traceability metrics.

Pydantic. (2024). Pydantic: Data validation using Python type annotations.

Used for defining structured requirement schemas with automatic validation. Ensures that extracted requirements conform to well-formedness rules before entering the traceability matrix.

Pedregosa, F., et al. (2011). Scikit-learn: Machine learning in Python. JMLR, 12, 2825-2830.

Provides the clustering algorithms (DBSCAN, agglomerative clustering) used for requirement deduplication and grouping in Section 13.2.

Tutorials & Standards

ISO/IEC/IEEE 29148:2018. Systems and software engineering: Life cycle processes, Requirements engineering.

The international standard for requirements engineering processes. Defines the quality attributes (completeness, consistency, feasibility, testability) that our automated validators check.

OpenAI. (2024). Structured Outputs in the API.

Documents the JSON-mode and structured-output capabilities that make reliable requirement extraction from LLMs possible. The techniques generalize across providers.

Pydantic. (2024). PydanticAI: Agent framework with structured outputs.

An agent framework that combines Pydantic validation with LLM tool calling. Provides an alternative integration path for the requirement extraction pipeline.