Multi-Agent Resilience

The resilience module provides the database layer and core types for multi-agent resilience. It tracks agent tasks, events, traces, and subagent sessions in SQLite for recovery and observability.

Design Philosophy

1. Persistent state — All agent state survives restarts via SQLite
2. Structured traces — Task execution traces enable shadow replay for debugging
3. Session lifecycle — Subagent sessions track creation, idle, and swap states
4. Tiered events — Skeleton events for structure, Pulse events for detail

Core Types

TaskStatus

Tasks progress through a state machine:

pub enum TaskStatus {
    Pending,      // Waiting to execute
    Running,      // Currently executing
    Completed,    // Success
    Failed,       // Error occurred
    Interrupted,  // System restart
    Idle,         // Paused (Session-as-a-Service)
    Swapped,      // Context swapped to disk
}

Lane

Execution lane for resource isolation — not Sequential/Parallel:

pub enum Lane {
    Main,         // User interactions, abort commands (high priority)
    Subagent,     // Background work (normal priority)
}

RiskLevel

Risk level for task-recovery decisions — two tiers only:

pub enum RiskLevel {
    Low,          // Read-only operations; safe to auto-resume
    High,         // Write operations; requires user confirmation
}

AgentTask

A task with recovery checkpoints:

pub struct AgentTask {
    pub id: String,                            // task_id
    pub parent_session_id: String,
    pub agent_id: String,
    pub task_prompt: String,
    pub status: TaskStatus,
    pub risk_level: RiskLevel,
    pub lane: Lane,
    pub checkpoint_snapshot_path: Option<String>,  // Shadow Replay snapshot path
    pub last_tool_call_id: Option<String>,
    pub recursion_depth: u32,
    pub parent_task_id: Option<String>,
    pub created_at: i64,
    pub updated_at: i64,
    pub started_at: Option<i64>,
    pub completed_at: Option<i64>,
    pub metadata_json: Option<String>,
}

TaskTrace

Structured execution traces for shadow replay:

pub struct TaskTrace {
    pub id: i64,
    pub task_id: String,
    pub step_index: u32,
    pub event: AgentTraceEvent,
    pub timestamp: i64,
}

TaskTrace::with_timestamp lets an ordered fixture override the Utc::now() stamp that new() writes, since the trace-listing cursor paginates on timestamp with strict less-than semantics.

AgentEvent

Tiered persistence (Skeleton / Pulse):

pub struct AgentEvent {
    pub id: i64,
    pub task_id: String,
    pub seq: u64,                       // ordering for Gap-Fill
    pub event_type: String,
    pub payload_json: String,
    pub is_structural: bool,            // true = skeleton, persisted immediately
    pub timestamp: i64,
}

Long-lived subagent session management lives in src/teams/sessions/store.rs (SessionStatus / CollaborativeSession / SessionTurn), not in resilience.

StateDatabase

SQLite database providing CRUD operations for:

Schema

The schema is versioned with migration utilities in migration.rs. Key indexes:

• tasks(task_id, status) — Task lookup by status
• events(event_id, created_at) — Event time-range queries
• traces(trace_id, task_id) — Trace-to-task mapping

Safety

• Integer overflow prevention — usize to i64 conversions use try_from with i64::MAX fallback
• Lock safety — All lock() calls use unwrap_or_else(|e| e.into_inner())
• Parameterized queries — All SQL uses params![], no string interpolation
• No static mut — AtomicBool for flags

Key Source Files

• src/resilience/mod.rs — Module overview
• src/resilience/types.rs — Core types (AgentTask, TaskTrace, etc.)
• src/resilience/database/state_database.rs — SQLite CRUD operations
• src/resilience/database/migration.rs — Schema versioning

See Also

• Agent Runtime — Agent execution loop
• Task Scheduling — Task queue and scheduling
• Event System — Event bus architecture

26.7.x Addendum

Guardian Judge Payload Masks Secrets

26.7.21+: GuardianApprovalRequester (the LLM judge in src/approval/guardian_requester.rs) runs the action description it sends to the judge LLM through SecretMasker (src/exec/masker) — both the summary and every parsed command segment pass through the same masker, so the redacted summary cannot leak what the per-segment raw text (built from the original argv by the shell parser) still carries (bearer tokens, URL basic-auth, generic password assignments). A circuit breaker escalates straight to the human without a judge call after sustained provider failures.

Heartbeat Refuses Dangerous Tools

26.7.21+: DefaultProbeExecutor::execute (src/tasks/heartbeat/probe.rs) calls is_denied_on_gateway_surface(tool_name). Probes run on a timer with no LLM in the loop, so any RCE / host-mutation / control-plane tool or confirmation-gated tool is rejected with an error pointing at GATEWAY_TOOLS_ALLOW_ENV (DANGEROUS only). The gate mirrors the one tools.invoke uses.

Provider Error Reads Bounded

26.7.15+: provider error response body reads are now bounded — previously they could hang on a stalled proxy.

Shadow-Mode System State Bus

26.7.x shadow mode: the System State Bus (SSB) with PII filtering; a robustness and privacy layer; an action dispatcher for automated responses to system events; a Platform Abstraction Layer (PAL) with macOS-specific sensors.

Dreaming Daemon No Longer Burns Quota

26.7.15 fix: the nightly dream run's retry storm is now bounded — should_skip_scheduled_run treats error and a leftover running row as "spent for today"; only cancelled earns a retry. The daemon no longer exhausts the provider quota.

Fail-Closed

26.7.21 hardening: when a policy file is missing, unreadable, or fails to parse, authorization is refused.

See Also

• Daemon — Heartbeat driver
• Error Handling — fail-closed
• Security Overview — the fail-closed principle