Part VII: Autonomous Discovery Systems
Chapter 57: Responsible Discovery AI

57.3 Governance and Accountability

"I submitted my research proposal to six review committees, three regulatory bodies, and one ethics board. They all approved it in parallel. Unfortunately, none of them talked to each other."

An Approval Workflow With a Concurrency Bug

Prerequisites

This section builds on the risk register and dual-use screening from Section 57.1 and the integrity checks from Section 57.2. It references the multi-agent architecture from Chapter 54 and the experiment provenance system from Chapter 47. The code uses Python dataclasses and enumerations.

The Big Picture

Safety tools (Section 57.1) identify what can go wrong. Integrity checks (Section 57.2) ensure that research outputs are honest. Governance is the organizational machinery that connects these technical safeguards to decisions: who approves a research plan, under what conditions, with what oversight, and how each decision is recorded for future accountability. This section builds a complete governance workflow for autonomous discovery, from proposal submission through safety review, ethics screening, approval, execution, and audit. We formalize each stage as a state machine, define the approval hierarchies that different risk levels require, survey the regulatory frameworks that constrain autonomous research (the EU AI Act, NIST AI RMF, biosafety regulations), and construct an immutable audit trail that satisfies both internal accountability and external regulatory requirements.

1. The Governance Workflow

In 2023, an autonomous chemistry agent (Boiko et al., Nature) went from research objective to synthesized compound in under four minutes, faster than any review committee could convene, and without a single institutional checkpoint recording why that particular molecule was chosen. What organizational machinery would have caught that gap?

That gap is not hypothetical. Every autonomous system that can design, order reagents, and trigger a robotic synthesis without structured oversight can convert speed into risk: the faster it runs, the wider the blast radius of any unchecked decision. Governance workflows exist precisely to make that speed safe.

A governance workflow is a formally specified sequence of checkpoints that a research proposal must clear before any experiment runs, during execution, and after completion. Autonomous systems can initiate experiments faster than any human can review them. Without a programmatic gate, a system could launch hundreds of unchecked studies overnight. The mechanism is a finite state machine (a model of computation with a fixed set of states, where transitions between states are triggered only when specified conditions are met): each stage holds a proposal until a set of boolean predicates (entry conditions) evaluate to true. The proposal then advances to the next stage or returns for revision. Use a governance workflow whenever the research involves any external resource, physical actuation, or result that will be published. For purely exploratory computational sketches with no side effects, a lightweight logging checkpoint suffices. In short: governance is the code that turns an autonomous system's velocity from a threat into an asset. Figure 57.3.1 illustrates the governance workflow state machine.

Governance workflow state machine
Figure 57.3.1: The six-stage governance state machine for autonomous discovery, showing the sequential pipeline from proposal to audit, revision and rejection branches, and the tiered approval hierarchy that routes proposals to the appropriate authority based on risk level.

The six stages are shown in Figure 57.3 and described below:

Governance Workflow State Machine 1. Proposal 2. Safety Review 3. Ethics Screening 4. Approval 5. Execution 6. Audit Revision Required Rejected Forward transition Revision loop Rejection
Figure 57.3: The governance workflow as a finite state machine. Solid arrows show the forward path from Proposal through Audit. Dashed yellow arrows show the revision loop (stages 2, 3, or 4 can send a proposal back through Revision Required to Proposal for resubmission). Dashed red arrows show that any stage except Audit can reject a proposal outright. Audit and Rejected are terminal states with no outgoing transitions.
  1. Proposal: The discovery system (or a human researcher) submits a structured research plan specifying objectives, methods, expected outcomes, and resource requirements.
  2. Safety Review: The risk register (Section 57.1) is populated, dual-use screening (where "dual-use" refers to research that could be repurposed for both beneficial and harmful applications) runs, and the proposal's risk-adjusted expected value is computed.
  3. Ethics Screening: The integrity assessment (Section 57.2) runs, checking authorship compliance, data provenance, and potential fabrication risks. Human subjects protocols are verified if applicable.
  4. Approval: Based on the risk level determined in stages 2 and 3, the appropriate authority approves or rejects the proposal. Low-risk proposals may be auto-approved; high-risk proposals require committee review.
  5. Execution: The approved research plan executes within the constraints specified in the approval. Runtime monitoring checks that the system stays within approved parameters.
  6. Audit: All decisions, actions, and outcomes are recorded in an immutable audit log with cryptographic integrity guarantees.
from enum import Enum, auto
from dataclasses import dataclass, field
import datetime


class WorkflowStage(Enum):
    """Stages of the governance workflow."""
    PROPOSAL = auto()
    SAFETY_REVIEW = auto()
    ETHICS_SCREENING = auto()
    APPROVAL = auto()
    EXECUTION = auto()
    AUDIT = auto()
    REJECTED = auto()
    REVISION_REQUIRED = auto()


class ApprovalLevel(Enum):
    """Approval authority levels based on risk classification."""
    AUTO = auto()          # System auto-approves (low risk)
    PI = auto()            # Principal investigator (medium risk)
    COMMITTEE = auto()     # Safety committee (high risk)
    INSTITUTIONAL = auto() # Institutional review board (critical)
    REGULATORY = auto()    # External regulatory body


@dataclass
class WorkflowTransition:
    """A transition between governance workflow stages."""
    from_stage: WorkflowStage
    to_stage: WorkflowStage
    timestamp: datetime.datetime
    actor: str           # Who or what triggered the transition
    reason: str          # Why the transition occurred
    conditions_met: list[str]  # What conditions were satisfied

    @property
    def log_entry(self) -> str:
        return (
            f"[{self.timestamp.isoformat()}] "
            f"{self.from_stage.name} -> {self.to_stage.name} "
            f"by {self.actor}: {self.reason}"
        )


@dataclass
class GovernanceWorkflow:
    """State machine for research governance.

    Enforces the six-stage pipeline with appropriate
    approval levels based on risk classification.
    """
    proposal_id: str
    current_stage: WorkflowStage = WorkflowStage.PROPOSAL
    transitions: list[WorkflowTransition] = field(
        default_factory=list
    )
    risk_level: str = "UNKNOWN"
    required_approval: ApprovalLevel = ApprovalLevel.AUTO

    # Valid transitions
    VALID_TRANSITIONS = {
        WorkflowStage.PROPOSAL: {
            WorkflowStage.SAFETY_REVIEW,
            WorkflowStage.REJECTED,
        },
        WorkflowStage.SAFETY_REVIEW: {
            WorkflowStage.ETHICS_SCREENING,
            WorkflowStage.REVISION_REQUIRED,
            WorkflowStage.REJECTED,
        },
        WorkflowStage.ETHICS_SCREENING: {
            WorkflowStage.APPROVAL,
            WorkflowStage.REVISION_REQUIRED,
            WorkflowStage.REJECTED,
        },
        WorkflowStage.APPROVAL: {
            WorkflowStage.EXECUTION,
            WorkflowStage.REVISION_REQUIRED,
            WorkflowStage.REJECTED,
        },
        WorkflowStage.EXECUTION: {
            WorkflowStage.AUDIT,
            WorkflowStage.REJECTED,  # halted during execution
        },
        WorkflowStage.REVISION_REQUIRED: {
            WorkflowStage.PROPOSAL,  # resubmit
        },
        WorkflowStage.AUDIT: set(),       # terminal
        WorkflowStage.REJECTED: set(),    # terminal
    }

    def transition(
        self,
        to_stage: WorkflowStage,
        actor: str,
        reason: str,
        conditions_met: list[str] = None,
    ) -> bool:
        """Attempt a state transition.

        Returns True if transition was valid and executed,
        False if transition was invalid.
        """
        if conditions_met is None:
            conditions_met = []

        valid_targets = self.VALID_TRANSITIONS.get(
            self.current_stage, set()
        )
        if to_stage not in valid_targets:
            return False

        transition = WorkflowTransition(
            from_stage=self.current_stage,
            to_stage=to_stage,
            timestamp=datetime.datetime.now(),
            actor=actor,
            reason=reason,
            conditions_met=conditions_met,
        )
        self.transitions.append(transition)
        self.current_stage = to_stage
        return True

    def set_risk_level(self, risk_level: str) -> None:
        """Set risk level and determine required approval authority."""
        self.risk_level = risk_level
        approval_map = {
            "LOW": ApprovalLevel.AUTO,
            "MEDIUM": ApprovalLevel.PI,
            "HIGH": ApprovalLevel.COMMITTEE,
            "CRITICAL": ApprovalLevel.INSTITUTIONAL,
            "BLOCKED": ApprovalLevel.REGULATORY,
        }
        self.required_approval = approval_map.get(
            risk_level, ApprovalLevel.INSTITUTIONAL
        )

    def history(self) -> str:
        """Return formatted transition history."""
        lines = [f"Workflow history for {self.proposal_id}:"]
        for t in self.transitions:
            lines.append(f"  {t.log_entry}")
        lines.append(f"  Current stage: {self.current_stage.name}")
        lines.append(f"  Risk level: {self.risk_level}")
        lines.append(
            f"  Required approval: {self.required_approval.name}"
        )
        return "\n".join(lines)


# Walk a proposal through the governance pipeline
wf = GovernanceWorkflow(proposal_id="PROP-2026-SolarCell")

# Stage 1 -> 2: Submit for safety review
wf.transition(
    WorkflowStage.SAFETY_REVIEW,
    actor="Discovery Workbench v3.2",
    reason="Proposal submitted for safety review",
    conditions_met=["proposal_complete", "objectives_defined"],
)

# Safety review determines risk level
wf.set_risk_level("MEDIUM")

# Stage 2 -> 3: Pass safety review
wf.transition(
    WorkflowStage.ETHICS_SCREENING,
    actor="SafetyReviewModule",
    reason="Risk register populated; no CRITICAL or BLOCKED hazards",
    conditions_met=["risk_register_complete",
                     "dual_use_screen_passed",
                     "raev_positive"],
)

# Stage 3 -> 4: Pass ethics screening
wf.transition(
    WorkflowStage.APPROVAL,
    actor="IntegrityCheckModule",
    reason="All integrity checks passed",
    conditions_met=["authorship_compliant",
                     "provenance_valid",
                     "no_fabrication_risk"],
)

# Stage 4 -> 5: PI approves (MEDIUM risk requires PI)
wf.transition(
    WorkflowStage.EXECUTION,
    actor="Dr. Sarah Chen (PI)",
    reason="Approved by PI; risk level MEDIUM within PI authority",
    conditions_met=["pi_review_complete",
                     "approval_level_satisfied"],
)

# Stage 5 -> 6: Execution complete, audit
wf.transition(
    WorkflowStage.AUDIT,
    actor="ExecutionMonitor",
    reason="Experiment completed within approved parameters",
    conditions_met=["no_safety_violations",
                     "results_within_bounds",
                     "resources_within_budget"],
)

print(wf.history())
Listing 57.12: A governance workflow state machine tracking a solar cell research proposal through all six stages, with the VALID_TRANSITIONS dictionary enforcing that no stage can be skipped. Each transition records the actor, reason, and conditions satisfied, creating an auditable decision trail.
Workflow history for PROP-2026-SolarCell:
  [2026-07-02T14:30:01.234567] PROPOSAL -> SAFETY_REVIEW by Discovery Workbench v3.2: Proposal submitted for safety review
  [2026-07-02T14:30:01.234789] SAFETY_REVIEW -> ETHICS_SCREENING by SafetyReviewModule: Risk register populated; no CRITICAL or BLOCKED hazards
  [2026-07-02T14:30:01.234890] ETHICS_SCREENING -> APPROVAL by IntegrityCheckModule: All integrity checks passed
  [2026-07-02T14:30:01.234901] APPROVAL -> EXECUTION by Dr. Sarah Chen (PI): Approved by PI; risk level MEDIUM within PI authority
  [2026-07-02T14:30:01.234912] EXECUTION -> AUDIT by ExecutionMonitor: Experiment completed within approved parameters
  Current stage: AUDIT
  Risk level: MEDIUM
  Required approval: PI
Output of Listing 57.12: The complete transition history for the solar cell proposal, showing its path through all six stages with timestamps, actors, and conditions at each step.
Key Insight: The Workflow Is a Contract, Not a Suggestion

The governance workflow is implemented as a state machine with strict transition rules, as illustrated in Figure 57.3. The system cannot move from PROPOSAL directly to EXECUTION; it must pass through SAFETY_REVIEW, ETHICS_SCREENING, and APPROVAL first. This is not a guideline that relies on good faith; it is a programmatic constraint that the Discovery Workbench enforces at the API level. An autonomous agent that attempts to execute a research plan without approval receives an error, not a warning. The constraint is analogous to a type system in programming: it makes entire categories of error structurally impossible.

With the workflow stages defined as a strict state machine, the next question is who holds the authority to approve or reject a proposal at each gate, and how that authority scales with the level of risk involved.

2. Approval Hierarchies

Risk level determines approval authority. Low-risk computational studies (no physical experiments, no dual-use concerns, no human subjects) receive automatic approval. High-risk experiments with hazardous materials escalate to a safety committee, and dual-use projects escalate further to institutional or regulatory review. The approval hierarchy encodes these escalation rules.

Common Misconception

A common misconception is that "auto-approved" means "ungoverned." When a low-risk proposal receives automatic approval, it has still passed through every prior stage of the governance pipeline (proposal validation, safety review, ethics screening); the only thing that is automated is the final sign-off, because the risk level does not warrant a human reviewer's time. The proposal's full audit trail, boundary conditions, and runtime monitoring remain active and identical to those of a committee-reviewed project. Auto-approval is the output of governance, not the absence of it.

from dataclasses import dataclass


@dataclass
class ApprovalAuthority:
    """An entity authorized to approve research at a given level."""
    authority_id: str
    name: str
    level: ApprovalLevel
    can_approve_levels: list[ApprovalLevel]
    requires_quorum: bool = False
    quorum_size: int = 1

    def can_approve(self, required_level: ApprovalLevel) -> bool:
        """Check if this authority can approve the given level."""
        return required_level in self.can_approve_levels


# Define the approval hierarchy
APPROVAL_HIERARCHY = [
    ApprovalAuthority(
        authority_id="AUTH-SYSTEM",
        name="Discovery Workbench (auto-approve)",
        level=ApprovalLevel.AUTO,
        can_approve_levels=[ApprovalLevel.AUTO],
    ),
    ApprovalAuthority(
        authority_id="AUTH-PI",
        name="Principal Investigator",
        level=ApprovalLevel.PI,
        can_approve_levels=[ApprovalLevel.AUTO, ApprovalLevel.PI],
    ),
    ApprovalAuthority(
        authority_id="AUTH-SAFETY",
        name="Institutional Safety Committee",
        level=ApprovalLevel.COMMITTEE,
        can_approve_levels=[ApprovalLevel.AUTO, ApprovalLevel.PI,
                            ApprovalLevel.COMMITTEE],
        requires_quorum=True,
        quorum_size=3,
    ),
    ApprovalAuthority(
        authority_id="AUTH-IRB",
        name="Institutional Review Board",
        level=ApprovalLevel.INSTITUTIONAL,
        can_approve_levels=[ApprovalLevel.AUTO, ApprovalLevel.PI,
                            ApprovalLevel.COMMITTEE,
                            ApprovalLevel.INSTITUTIONAL],
        requires_quorum=True,
        quorum_size=5,
    ),
    ApprovalAuthority(
        authority_id="AUTH-REG",
        name="Regulatory Authority (external)",
        level=ApprovalLevel.REGULATORY,
        can_approve_levels=[ApprovalLevel.AUTO, ApprovalLevel.PI,
                            ApprovalLevel.COMMITTEE,
                            ApprovalLevel.INSTITUTIONAL,
                            ApprovalLevel.REGULATORY],
        requires_quorum=True,
        quorum_size=1,  # varies by jurisdiction
    ),
]


def find_required_authority(
    required_level: ApprovalLevel,
    hierarchy: list[ApprovalAuthority] = None,
) -> ApprovalAuthority:
    """Find the lowest-level authority that can approve."""
    if hierarchy is None:
        hierarchy = APPROVAL_HIERARCHY
    for authority in hierarchy:
        if authority.can_approve(required_level):
            return authority
    raise ValueError(
        f"No authority found for level {required_level.name}"
    )


# Demonstrate approval routing
for risk, level in [
    ("LOW", ApprovalLevel.AUTO),
    ("MEDIUM", ApprovalLevel.PI),
    ("HIGH", ApprovalLevel.COMMITTEE),
    ("CRITICAL", ApprovalLevel.INSTITUTIONAL),
]:
    auth = find_required_authority(level)
    quorum_note = (f" (quorum: {auth.quorum_size})"
                   if auth.requires_quorum else "")
    print(f"  {risk:>8} risk -> {auth.name}{quorum_note}")
Listing 57.13: Approval hierarchy with ApprovalAuthority dataclasses mapping risk levels to authorities. The requires_quorum flag enforces that committee and institutional decisions need a minimum number of concurrent approvers (where a quorum is the minimum number of members who must participate for a decision to be valid).
      LOW risk -> Discovery Workbench (auto-approve)
   MEDIUM risk -> Principal Investigator
     HIGH risk -> Institutional Safety Committee (quorum: 3)
 CRITICAL risk -> Institutional Review Board (quorum: 5)
Output of Listing 57.13: Each risk level routes to the lowest authority empowered to approve it. Low-risk projects are auto-approved by the system; critical projects require a five-member institutional review board.

Approval hierarchies determine who decides within an organization, but those internal policies do not exist in a vacuum; external regulations constrain what any authority, human or automated, is permitted to approve.

3. Regulatory Frameworks

Autonomous discovery systems operate within a web of regulations that vary by jurisdiction, domain, and risk level. As of 2026, three frameworks are particularly relevant:

The EU AI Act (2024) classifies AI systems into four risk tiers: unacceptable, high, limited, and minimal. Autonomous scientific discovery systems typically fall into the "high-risk" category when they involve critical infrastructure, education, employment, or health applications. High-risk systems must satisfy requirements for risk management, data governance, technical documentation, transparency, human oversight, accuracy, robustness, and cybersecurity. This governance workflow directly addresses several of these: risk management (safety review stage), human oversight (approval stage), and technical documentation (audit stage).

The NIST AI Risk Management Framework (2023) organizes risk management into four functions: Govern (policies and accountability), Map (context and risk identification), Measure (risk analysis and tracking), and Manage (risk response). Our governance workflow maps onto all four. The workflow structure is Govern. The risk register is Map. Risk-adjusted expected value (RAEV) computation is Measure. The approval hierarchy is Manage.

Biosafety regulations (the Biological Weapons Convention, select agent regulations, institutional biosafety committees) impose specific requirements on research involving pathogens, toxins, and gain-of-function experiments. Autonomous discovery systems in biology must comply with these regulations in addition to general AI governance frameworks.

Checkpoint

So far: three regulatory frameworks constrain autonomous discovery systems: the EU AI Act (risk tiers and compliance requirements), the NIST AI RMF (four functions: Govern, Map, Measure, Manage), and domain-specific biosafety regulations; the governance workflow maps onto all three.


from dataclasses import dataclass


@dataclass(frozen=True)
class RegulatoryRequirement:
    """A specific regulatory requirement applicable to the project."""
    requirement_id: str
    framework: str         # "EU AI Act", "NIST AI RMF", etc.
    article: str           # Specific article or section
    description: str
    applies_when: str      # Condition that triggers applicability
    evidence_required: str # What must be documented
    governance_stage: str  # Which workflow stage addresses this


# Key regulatory requirements for autonomous discovery
REGULATORY_REQUIREMENTS = [
    RegulatoryRequirement(
        requirement_id="REG-EU-001",
        framework="EU AI Act",
        article="Article 9: Risk Management System",
        description="High-risk AI systems shall have a risk "
                    "management system established, implemented, "
                    "documented, and maintained",
        applies_when="System classified as high-risk AI",
        evidence_required="Risk register with identified hazards, "
                          "likelihood, severity, and mitigations",
        governance_stage="SAFETY_REVIEW",
    ),
    RegulatoryRequirement(
        requirement_id="REG-EU-002",
        framework="EU AI Act",
        article="Article 14: Human Oversight",
        description="High-risk AI systems shall be designed to "
                    "allow effective oversight by natural persons",
        applies_when="System classified as high-risk AI",
        evidence_required="Documentation of human oversight points, "
                          "approval authorities, and override "
                          "mechanisms",
        governance_stage="APPROVAL",
    ),
    RegulatoryRequirement(
        requirement_id="REG-NIST-001",
        framework="NIST AI RMF",
        article="GOVERN 1.1",
        description="Legal and regulatory requirements are "
                    "identified and addressed",
        applies_when="Any AI system in regulated domain",
        evidence_required="Regulatory mapping document linking "
                          "requirements to system controls",
        governance_stage="ETHICS_SCREENING",
    ),
    RegulatoryRequirement(
        requirement_id="REG-BIO-001",
        framework="Select Agent Regulations (42 CFR 73)",
        article="Section 73.1",
        description="Registration and oversight requirements for "
                    "research involving select agents and toxins",
        applies_when="Research involves organisms or toxins on "
                     "the Federal Select Agent Program list",
        evidence_required="IBC (Institutional Biosafety Committee) "
                          "approval, select agent registration, "
                          "biosafety plan, training records",
        governance_stage="SAFETY_REVIEW",
    ),
]


def check_regulatory_compliance(
    project_domains: set[str],
    project_risk_level: str,
    available_evidence: set[str],
    requirements: list[RegulatoryRequirement] = None,
) -> dict:
    """Check project against applicable regulatory requirements.

    Returns compliance status and any gaps.
    """
    if requirements is None:
        requirements = REGULATORY_REQUIREMENTS

    applicable = []
    compliant = []
    gaps = []

    for req in requirements:
        # Simplified applicability check
        is_applicable = False
        if "high-risk" in req.applies_when.lower() and \
           project_risk_level in ("HIGH", "CRITICAL"):
            is_applicable = True
        if "any ai" in req.applies_when.lower():
            is_applicable = True
        if "select agent" in req.applies_when.lower() and \
           "biosafety" in project_domains:
            is_applicable = True

        if is_applicable:
            applicable.append(req)
            # Check if required evidence exists
            evidence_keywords = set(
                req.evidence_required.lower().split()
            )
            if evidence_keywords & available_evidence:
                compliant.append(req)
            else:
                gaps.append(req)

    return {
        "applicable_count": len(applicable),
        "compliant_count": len(compliant),
        "gap_count": len(gaps),
        "gaps": [
            f"{r.framework} {r.article}: {r.description}"
            for r in gaps
        ],
        "fully_compliant": len(gaps) == 0,
    }


# Check compliance for a high-risk chemistry project
result = check_regulatory_compliance(
    project_domains={"chemistry", "drug_discovery"},
    project_risk_level="HIGH",
    available_evidence={
        "risk", "register", "hazards", "mitigations",
        "documentation", "oversight", "approval",
    },
)

print(f"Applicable requirements: {result['applicable_count']}")
print(f"Compliant: {result['compliant_count']}")
print(f"Gaps: {result['gap_count']}")
for gap in result["gaps"]:
    print(f"  Missing: {gap}")
Listing 57.14: Regulatory compliance checker that maps project characteristics (domain, risk level, available evidence) to applicable requirements from the EU AI Act, NIST AI RMF, and biosafety regulations, then identifies gaps where the project lacks required documentation.
Applicable requirements: 3
Compliant: 3
Gaps: 0
Output of Listing 57.14: The high-risk chemistry project satisfies all three applicable requirements. The biosafety select agent regulation does not apply because the project domain is chemistry, not biosafety.
Practical Example: Navigating the EU AI Act for a Self-Driving Lab

Consider a self-driving laboratory (Chapter 55) that autonomously designs and synthesizes candidate drug molecules. Under the EU AI Act, this system is likely classified as high-risk because it contributes to a medical product pipeline. The lab must therefore implement: (1) a risk management system (our risk register and RAEV computation), (2) data governance (our provenance chain from Section 57.2), (3) technical documentation (our audit trail), (4) transparency (our AI contribution statements), (5) human oversight (our approval hierarchy), (6) accuracy and robustness testing (evaluation from Chapter 56), and (7) cybersecurity measures (access controls and tamper detection). The governance workflow we built addresses requirements 1, 3, 4, and 5 directly. Requirements 2 and 6 are covered by other chapters. Requirement 7 is partially addressed by cryptographic hashing in the audit trail, though a full cybersecurity assessment would require additional measures beyond the scope of this book.

4. Aligning Discovery with Human Values

Governance frameworks enforce compliance with rules. Value alignment asks a deeper question: are the rules themselves the right ones? An autonomous discovery system that satisfies every regulatory requirement may still pursue research that conflicts with human values if those values are not encoded in the system's objectives.

Value alignment for discovery systems takes the form of value constraints that the optimization objective must satisfy, independent of regulatory requirements. The dimensions used here (beneficence, non-maleficence, autonomy, justice, and others) derive from the Belmont Report and the principlist tradition in bioethics, adapted to the context of automated scientific research.

Real-World Application: Emerald Cloud Lab
Real-World Application: Emerald Cloud Lab
from dataclasses import dataclass
from enum import Enum, auto


class ValueDimension(Enum):
    """Dimensions of human values relevant to scientific research."""
    BENEFICENCE = auto()        # Does the research benefit people?
    NON_MALEFICENCE = auto()    # Does it avoid causing harm?
    AUTONOMY = auto()           # Does it respect individual choice?
    JUSTICE = auto()            # Are benefits and risks distributed fairly?
    TRANSPARENCY = auto()       # Can the research be understood/audited?
    SUSTAINABILITY = auto()     # Is the environmental cost acceptable?
    DIGNITY = auto()            # Does it respect human dignity?


@dataclass
class ValueAssessment:
    """Assessment of a research proposal against value dimensions."""
    dimension: ValueDimension
    score: float         # -1.0 (violates) to +1.0 (strongly supports)
    justification: str
    concerns: list[str]


def assess_value_alignment(
    proposal_description: str,
    beneficiaries: list[str],
    potential_harms: list[str],
    environmental_cost: str,
    transparency_level: str,
) -> list[ValueAssessment]:
    """Assess a proposal against core value dimensions.

    This is a structured checklist, not an automated judgment.
    Each assessment requires human validation.
    """
    assessments = []

    # Beneficence: who benefits?
    beneficence_score = min(1.0, len(beneficiaries) * 0.25)
    assessments.append(ValueAssessment(
        dimension=ValueDimension.BENEFICENCE,
        score=beneficence_score,
        justification=f"Identified {len(beneficiaries)} "
                      f"beneficiary group(s)",
        concerns=[] if beneficiaries else
                 ["No beneficiaries identified"],
    ))

    # Non-maleficence: what harm is possible?
    harm_score = max(-1.0, 1.0 - len(potential_harms) * 0.3)
    assessments.append(ValueAssessment(
        dimension=ValueDimension.NON_MALEFICENCE,
        score=harm_score,
        justification=f"Identified {len(potential_harms)} "
                      f"potential harm(s)",
        concerns=potential_harms,
    ))

    # Transparency
    transparency_scores = {
        "full": 1.0, "partial": 0.5,
        "limited": 0.0, "opaque": -0.5,
    }
    t_score = transparency_scores.get(transparency_level, 0.0)
    assessments.append(ValueAssessment(
        dimension=ValueDimension.TRANSPARENCY,
        score=t_score,
        justification=f"Transparency level: {transparency_level}",
        concerns=[] if t_score >= 0.5 else
                 ["Research process not fully transparent"],
    ))

    return assessments


# Assess value alignment for a drug discovery project
assessments = assess_value_alignment(
    proposal_description="Discover novel antibiotics using "
                         "AI-guided molecular generation",
    beneficiaries=["patients with resistant infections",
                   "healthcare systems",
                   "global public health"],
    potential_harms=["environmental release of novel compounds",
                     "antimicrobial resistance acceleration"],
    environmental_cost="moderate",
    transparency_level="full",
)

print("Value Alignment Assessment:")
for a in assessments:
    status = "PASS" if a.score > 0 else "CONCERN"
    print(f"  {a.dimension.name:20s} {a.score:+.1f}  [{status}] "
          f"{a.justification}")
    for c in a.concerns:
        print(f"    Warning: {c}")
Listing 57.15: Value alignment assessment scoring a research proposal against ethical dimensions (beneficence, non-maleficence, transparency) using the ValueDimension enum and ValueAssessment dataclass, with structured concerns for human review.
Value Alignment Assessment:
  BENEFICENCE          +0.8  [PASS] Identified 3 beneficiary group(s)
  NON_MALEFICENCE      +0.4  [PASS] Identified 2 potential harm(s)
    Warning: environmental release of novel compounds
    Warning: antimicrobial resistance acceleration
  TRANSPARENCY         +1.0  [PASS] Transparency level: full
Output of Listing 57.15: The antibiotic discovery project scores positively on all three assessed dimensions but flags two non-maleficence warnings (environmental release, resistance acceleration) that require mitigation plans before proceeding.

Mental Model

Value alignment for a discovery system works like a building code for a construction project. The architect (the AI) can design any structure it wants, but the design must satisfy non-negotiable constraints: minimum ceiling height, fire exits on every floor, load-bearing walls rated for earthquakes. The architect did not write these codes; city councils and safety engineers did, based on decades of experience with what goes wrong when buildings are optimized purely for cost or aesthetics. Similarly, the discovery system optimizes for scientific novelty and impact, but every candidate experiment must pass through value constraints (beneficence, non-maleficence, justice) that humans defined based on centuries of ethical reasoning. The building code does not tell the architect what to build; it tells the architect what not to build. Value constraints work the same way: they bound the optimization space without dictating the solution within it.

Key Insight: Values Are Inputs, Not Outputs

An autonomous discovery system does not determine what is ethical; it enforces value constraints that humans define. The value dimensions above (beneficence, non-maleficence, justice, transparency, and others) encode centuries of ethical thinking into a computational framework, but the weights assigned to each dimension, the thresholds that determine acceptability, and the trade-offs between competing values are human decisions. The system's role is to make these decisions explicit, structured, and auditable, not to automate them away. When a governance workflow flags a conflict between beneficence and non-maleficence, the right response is not algorithmic resolution but human deliberation, informed by the structured assessment the system provides.

Value constraints and regulatory requirements shape what gets approved, but approval is a single moment in time; the system must also verify that an experiment, once running, continues to respect those constraints throughout its execution.

5. Runtime Monitoring and Emergency Stops

Governance does not end at approval. During execution, the system must monitor whether the autonomous discovery process stays within the boundaries defined in the approval. If it deviates, an emergency stop mechanism must halt the process before harm occurs.

from dataclasses import dataclass, field
from enum import Enum, auto


class MonitoringAlert(Enum):
    """Alert levels for runtime monitoring."""
    INFO = auto()       # Normal operation, logged but no action
    WARNING = auto()    # Approaching boundary, human notified
    CRITICAL = auto()   # Boundary exceeded, execution paused
    EMERGENCY = auto()  # Safety threat, immediate shutdown


@dataclass
class ExecutionBoundary:
    """A boundary condition that must be maintained during execution."""
    boundary_id: str
    parameter: str       # What is being monitored
    lower_bound: float
    upper_bound: float
    unit: str
    alert_on_breach: MonitoringAlert


@dataclass
class RuntimeMonitor:
    """Monitor that checks execution against approved boundaries."""
    boundaries: list[ExecutionBoundary] = field(
        default_factory=list
    )
    alerts: list[dict] = field(default_factory=list)
    halted: bool = False

    def check(self, parameter: str, value: float) -> MonitoringAlert:
        """Check a parameter value against its boundary.

        Returns the highest alert level triggered.
        """
        highest_alert = MonitoringAlert.INFO
        for b in self.boundaries:
            if b.parameter != parameter:
                continue
            if value < b.lower_bound or value > b.upper_bound:
                alert = b.alert_on_breach
                self.alerts.append({
                    "parameter": parameter,
                    "value": value,
                    "boundary": f"[{b.lower_bound}, {b.upper_bound}]"
                               f" {b.unit}",
                    "alert": alert.name,
                })
                if alert.value > highest_alert.value:
                    highest_alert = alert
                if alert == MonitoringAlert.EMERGENCY:
                    self.halted = True

        return highest_alert


# Configure monitoring for a synthesis experiment
monitor = RuntimeMonitor(boundaries=[
    ExecutionBoundary(
        boundary_id="BND-001",
        parameter="temperature",
        lower_bound=15.0, upper_bound=85.0,
        unit="C",
        alert_on_breach=MonitoringAlert.CRITICAL,
    ),
    ExecutionBoundary(
        boundary_id="BND-002",
        parameter="pressure",
        lower_bound=0.8, upper_bound=2.5,
        unit="atm",
        alert_on_breach=MonitoringAlert.EMERGENCY,
    ),
    ExecutionBoundary(
        boundary_id="BND-003",
        parameter="toxicity_score",
        lower_bound=0.0, upper_bound=0.3,
        unit="normalized",
        alert_on_breach=MonitoringAlert.EMERGENCY,
    ),
])

# Simulate readings
readings = [
    ("temperature", 72.0),   # normal
    ("pressure", 1.5),       # normal
    ("temperature", 88.0),   # over limit -> CRITICAL
    ("toxicity_score", 0.65),# over limit -> EMERGENCY
]

for param, value in readings:
    alert = monitor.check(param, value)
    status = "HALTED" if monitor.halted else "running"
    if alert != MonitoringAlert.INFO:
        print(f"  [{alert.name}] {param}={value} "
              f"(system: {status})")

print(f"\nTotal alerts: {len(monitor.alerts)}")
print(f"System halted: {monitor.halted}")
Listing 57.16: Runtime monitor with ExecutionBoundary definitions for temperature, pressure, and toxicity. The check() method compares each reading against its boundary and triggers an emergency halt when the toxicity score exceeds 0.3.
  [CRITICAL] temperature=88.0 (system: running)
  [EMERGENCY] toxicity_score=0.65 (system: HALTED)

Total alerts: 2
System halted: True
Output of Listing 57.16: The temperature breach (88.0 > 85.0) triggers a CRITICAL alert that is logged for human review (the system remains running), while the toxicity score breach (0.65 > 0.3) triggers an EMERGENCY stop that halts the system entirely.
Right Tool: Guardrails AI for LLM Safety Monitoring

Our runtime monitor checks numerical parameters. For monitoring the text outputs of large language model (LLM)-based discovery agents, Guardrails AI provides real-time validation:

# Guardrails AI: LLM output validation
# pip install guardrails-ai
from guardrails import Guard
from guardrails.hub import ToxicLanguage, RestrictToTopic

guard = Guard().use_many(
    ToxicLanguage(threshold=0.8, on_fail="exception"),
    RestrictToTopic(
        valid_topics=["chemistry", "materials_science"],
        invalid_topics=["weapons", "explosives"],
        on_fail="exception",
    ),
)

# Validate LLM output before acting on it
try:
    validated = guard.validate(
        "Synthesize compound X using standard Suzuki coupling "
        "at 80C for 12 hours"
    )
    print(f"Output validated: {validated.validated_output}")
except Exception as e:
    print(f"Output blocked: {e}")
Guardrails AI integration using ToxicLanguage and RestrictToTopic validators to screen LLM-generated synthesis instructions for harmful content and off-topic outputs before the discovery agent acts on them.

Guardrails AI provides 50+ pre-built validators (toxicity, personally identifiable information (PII) detection, topic restriction, factual consistency) that plug into any LLM pipeline. Our numerical boundary monitor is 40 lines; Guardrails replaces the text-monitoring portion with 5 lines and covers natural language risks that numerical monitors cannot detect.

Research Frontier

The NIST AI 600-1 "Generative AI Profile" (2024) extends the AI Risk Management Framework specifically to generative and foundation models, adding risk categories (confabulation (generating plausible but fabricated outputs), data privacy, information integrity, harmful bias) that earlier frameworks did not address. For autonomous discovery systems built on large language models, NIST AI 600-1 introduces the concept of "pre-deployment" vs. "post-deployment" risk measurement, requiring that governance workflows reassess risk not just at proposal time but continuously as a generative model's behavior drifts through fine-tuning or retrieval updates. The companion NIST AI 100-2 (2024) report on adversarial machine learning further formalizes threat taxonomies (evasion, poisoning, privacy attacks) that governance workflows should screen for. Together, these 2024 documents push governance beyond the static approve-or-reject model presented in this section toward continuous, adaptive risk monitoring that recalibrates approval thresholds as the underlying model changes.

Try It: Build a Governance Audit Trail in 30 Minutes

Build a working governance state machine with a tamper-evident audit log using only Python's standard library.

  1. Define the state machine. Create a GovernanceWorkflow class with the six stages from this section (PROPOSAL through AUDIT) and a transition() method that validates each move against a dictionary of allowed transitions. Raise a ValueError for illegal transitions and confirm that attempting PROPOSAL to EXECUTION fails.
  2. Add cryptographic chaining. After each valid transition, compute a SHA-256 hash (a cryptographic digest that maps any input to a fixed-length 256-bit fingerprint, making it computationally infeasible to find two inputs with the same hash) of the transition record concatenated with the previous hash (use hashlib.sha256). Store the hash chain in a list. This creates an append-only log where tampering with any entry invalidates all subsequent hashes.
  3. Implement a compliance checker. Write a function that accepts a list of required evidence tags (e.g., ["risk_register", "ethics_review", "pi_approval"]) and a set of provided evidence tags, then returns which requirements are satisfied and which have gaps.
  4. Run two proposals through the pipeline. Walk a low-risk proposal (auto-approved) and a high-risk proposal (committee-approved, requiring quorum of 3 approval records) through the full workflow. Print the hash chain for each and verify integrity by recomputing all hashes from the stored transitions.
  5. Simulate tampering. Modify one transition record in the middle of the chain (change the actor field) and run the integrity check again. Confirm that the verification function detects the break and reports exactly which entry was altered.

Exercise 57.3.1

A research proposal rated MEDIUM risk is submitted to the governance workflow. The principal investigator is on leave, and the only available approver is the Institutional Safety Committee (approval level COMMITTEE). Using the ApprovalAuthority hierarchy from Listing 57.13, determine: (a) Can the Safety Committee approve a MEDIUM-risk proposal? (b) Does the committee's quorum requirement (3 members) change for a MEDIUM-risk proposal it is overseeing? (c) If only two committee members are available, what should the workflow do?

Hint

Look at the can_approve_levels list for AUTH-SAFETY. The committee can approve any level up to and including COMMITTEE. Quorum is a property of the authority, not the risk level, so the quorum of 3 still applies regardless of the proposal's risk classification. If quorum is not met, the workflow should remain in the APPROVAL stage until quorum is satisfied or route to a different authority at the same or higher level.

Step-Through: Governance State Machine Transitions

Trace the governance workflow for a proposal rated HIGH risk, where the ethics screening fails on the first attempt. Each step corresponds to a transition arrow in Figure 57.3.

Step 1. State = PROPOSAL. Entry conditions met (objectives defined, proposal complete). Transition to SAFETY_REVIEW. Transition list: 1 entry.
Step 2. State = SAFETY_REVIEW. Risk register populated, risk level set to HIGH, required_approval = COMMITTEE. No BLOCKED hazards. Transition to ETHICS_SCREENING. Transition list: 2 entries.
Step 3. State = ETHICS_SCREENING. Integrity check finds missing provenance for one dataset. Transition to REVISION_REQUIRED. Transition list: 3 entries.
Step 4. State = REVISION_REQUIRED. Only valid target is PROPOSAL (resubmit). Researcher adds provenance documentation. Transition to PROPOSAL. Transition list: 4 entries.
Step 5. State = PROPOSAL (second pass). Re-enter SAFETY_REVIEW. Risk level remains HIGH. Transition list: 5 entries.
Step 6. State = SAFETY_REVIEW. Pass. Transition to ETHICS_SCREENING. Transition list: 6 entries.
Step 7. State = ETHICS_SCREENING. All checks pass. Transition to APPROVAL. Transition list: 7 entries.
Step 8. State = APPROVAL. Committee review with quorum of 3. Three members vote approve. Transition to EXECUTION. Transition list: 8 entries.
Step 9. State = EXECUTION. No boundary violations. Transition to AUDIT. Transition list: 9 entries. Final state: AUDIT.

The proposal traversed 9 transitions instead of the minimum 5 because the ethics failure added a REVISION_REQUIRED detour (2 extra transitions) plus a repeated SAFETY_REVIEW and ETHICS_SCREENING pass (2 more). The full history is preserved in the audit trail.

Real-World Application: Emerald Cloud Lab

Emerald Cloud Lab (ECL) is a cloud-hosted robotic laboratory where researchers submit experiment protocols remotely and robotic systems execute them. ECL enforces a governance layer that validates every protocol against safety constraints (chemical compatibility, equipment limits, reagent quantities) before scheduling execution. Protocols that exceed safety thresholds are flagged for human review, mirroring the approval hierarchy in this section where risk level determines whether a proposal is auto-approved or escalated to committee oversight.

The Audit Trail That Solved a Nobel Prize Dispute

When the 2008 Nobel Prize in Chemistry was awarded for the discovery of green fluorescent protein (GFP), the committee initially overlooked Douglas Prasher and other early contributors. What ultimately clarified each researcher's role was the paper trail: dated lab notebooks, signed witness statements, and institutional review records. Osamu Shimomura's meticulous 1962 notebook entries proved he had first isolated GFP from jellyfish. Modern cryptographic audit trails, like the hash-chained logs in this section, serve exactly the same purpose but with tamper evidence that pen and paper cannot provide. A single flipped bit in a forged record breaks every subsequent hash in the chain.

Lab: Build and Break a Hash-Chained Governance Log

Goal: Construct a tamper-evident audit trail for a governance workflow and verify that tampering is detectable.
Tools needed: Python 3.10+, only the standard library (hashlib, dataclasses, json, datetime). No external packages required.
Procedure (25 minutes):

  1. (5 min) Implement the GovernanceWorkflow class from Listing 57.12. After each transition, compute hashlib.sha256(previous_hash + json.dumps(transition_record)) and store the hex digest alongside the transition.
  2. (5 min) Walk two proposals through the full pipeline: one LOW-risk (auto-approved, 5 transitions) and one HIGH-risk with a revision loop (9 transitions). Print the hash chain for each.
  3. (5 min) Write a verify_chain() function that recomputes every hash from the stored transitions and checks it against the stored digest. Confirm both chains pass.
  4. (5 min) Tamper with one transition in the middle of the HIGH-risk chain (change the actor from "SafetyReviewModule" to "AttackerBot"). Run verify_chain() again and observe that it reports the exact index where the chain breaks.
  5. (5 min) Vary the hash algorithm (switch to SHA-512 or BLAKE2b) and measure whether verification time changes for chains of 100 and 10,000 entries using time.perf_counter().

What to observe: Tampering always breaks the chain at the modified entry and every entry after it. The number of broken links equals the distance from the tampered entry to the end of the chain. Hash algorithm choice typically has negligible impact on verification time for chains under 100,000 entries.

What's Next

The governance machinery is now complete in concept: risk registers, dual-use screening, integrity checks, approval workflows, regulatory compliance, value alignment, and runtime monitoring. In Section 57.4: Building a Governance Layer, we integrate all of these components into a single governance layer for the Discovery Workbench, with a unified API, cryptographically signed audit logs, and a working end-to-end demonstration.

Bibliography

European Union AI Act (2024). Regulation (EU) 2024/1689.

The comprehensive EU AI regulation establishing risk tiers and compliance requirements for high-risk AI systems.

NIST AI Risk Management Framework (2023).

The four-function (Govern, Map, Measure, Manage) framework for AI risk management that structures our governance workflow.

Boiko, D. A., et al. (2023). Autonomous chemical research with large language models. Nature, 624, 570-578.

Coscientist demonstrates the real-world governance challenges of autonomous laboratory agents.

Guardrails AI

Open-source framework for LLM output validation with 50+ pre-built validators for safety, toxicity, and topic restriction.

Federal Select Agent Program. U.S. CDC/APHIS.

The regulatory program governing research with biological select agents and toxins.