Resilience System

SQLite persistence core, StateDatabase, skeleton/pulse event model, and task/trace types — the storage substrate that backs Shadow Replay and risk-aware recovery.

The resilience system is Aleph's persistence and state layer. After the agent-loop migration, src/resilience/ only retains the database layer (StateDatabase) and shared types — governance, collaboration, perception, and recovery middleware have been removed; higher-level recovery is now orchestrated by the harness in src/agents/ together with the gateway event bus.

src/resilience/mod.rs after the 26.7.x migration only exports database and types:
pub mod database;
pub mod types;
The historical recovery/, collaboration/, and shadow-replay submodules no longer exist. Their responsibilities now live in src/agents/swarm/tasks/, src/harness/trace.rs, and src/gateway/event_bus.rs.

Current layout

src/resilience/
├── mod.rs        # entry; only re-exports database + types
├── types.rs      # AgentTask / TaskTrace / AgentEvent / Lane / RiskLevel / TaskStatus
└── database/
    ├── mod.rs                 # module entry; DEFAULT_EMBEDDING_DIM etc.
    ├── state_database/        # StateDatabase implementation + schema migrations
    ├── channel_offsets.rs     # channel offset persistence
    ├── events.rs              # AgentEvent persistence
    ├── group_chat.rs          # group-chat history persistence
    ├── memory_events.rs       # memory-system event log
    ├── migration.rs           # schema migrations
    ├── paired_users.rs        # paired-user records
    ├── tasks.rs               # AgentTask CRUD
    └── traces.rs              # TaskTrace CRUD

StateDatabase

StateDatabase is the persistence backbone of the resilience system. SQLite via rusqlite:

Table / entity	Purpose	Key fields
AgentEvent	Tiered event persistence (Skeleton + Pulse)	`task_id`, `seq`, `event_type`, `is_structural`
AgentTask	Task state + recovery metadata	`id`, `status`, `risk_level`, `lane`, `checkpoint_snapshot_path`
TaskTrace	Shadow Replay execution trace	`task_id`, `step_index`, `event`, `timestamp`
group_chat	Group-chat history	per-team persistence
memory_events	Memory-system events	write log
paired_users	Paired-user records	pairing
channel_offsets	Channel read offsets	webhook / event-source resync

AgentUsageTotal and MemoryStats are also exported and reused by recall and experience-store submodules in src/memory/.

Event tiers: skeleton and pulse

Events follow a two-tier persistence model:

impl AgentEvent {
    // Structural events (Skeleton) — persisted immediately
    pub const TYPE_TASK_STARTED:        &str = "task_started";
    pub const TYPE_TOOL_CALL_STARTED:    &str = "tool_call_started";
    pub const TYPE_TOOL_CALL_COMPLETED:  &str = "tool_call_completed";
    pub const TYPE_ARTIFACT_CREATED:     &str = "artifact_created";
    pub const TYPE_TASK_COMPLETED:       &str = "task_completed";
    pub const TYPE_TASK_FAILED:          &str = "task_failed";

    // Streaming events (Pulse) — batched before persistence
    pub const TYPE_AI_STREAMING:         &str = "ai_streaming";

    pub fn structural(task_id, seq, etype, payload) -> Self { /* is_structural = true */ }
    pub fn pulse(task_id, seq, etype, payload)        -> Self { /* is_structural = false */ }
}

Skeleton events represent state transitions that must survive crashes. They are written synchronously and used for recovery. Pulse events are high-frequency streaming data (token-by-token output) that is batched and can be dropped without affecting correctness.

Core types (`src/resilience/types.rs`)

`AgentTask`

pub struct AgentTask {
    pub id: String,
    pub parent_session_id: String,
    pub agent_id: String,              // "explorer" / "coder" / …
    pub task_prompt: String,
    pub status: TaskStatus,
    pub risk_level: RiskLevel,
    pub lane: Lane,
    pub checkpoint_snapshot_path: Option<String>,
    pub last_tool_call_id: Option<String>,
    pub recursion_depth: u32,
    pub parent_task_id: Option<String>,
    pub created_at: i64,
    pub updated_at: i64,
    pub started_at: Option<i64>,
    pub completed_at: Option<i64>,
    pub metadata_json: Option<String>,
}

Helpers: AgentTask::new (defaults: Pending + Subagent lane), with_parent_task (recursion depth), with_lane, should_auto_resume (only RiskLevel::Low), needs_resume_confirmation (only RiskLevel::High).

`TaskStatus`

#[serde(rename_all = "snake_case")]
pub enum TaskStatus {
    Pending,
    Running,
    Completed,
    Failed,
    Interrupted,   // system shutdown during execution
    Idle,          // Session-as-a-Service pause
    Swapped,       // context swapped to disk
}

impl TaskStatus {
    pub const fn is_recoverable(&self) -> bool {
        matches!(self, Self::Running | Self::Interrupted)
    }
}

`RiskLevel`

pub enum RiskLevel { Low, High }

Low — read-only operations; safe to auto-resume on restart
High — write operations; requires user confirmation

`Lane`

pub enum Lane { Main, Subagent }

Main — user interactions, abort commands (high priority)
Subagent — background work (normal priority)

The Main lane is never starved by subagent work. User-facing requests (conversation, abort commands) always get priority scheduling.

`TaskTrace`

pub struct TaskTrace {
    pub id: i64,                // assigned by the database
    pub task_id: String,
    pub step_index: u32,        // 0-based
    pub event: AgentTraceEvent, // from aleph_protocol
    pub timestamp: i64,
}

impl TaskTrace {
    pub fn new(task_id, step_index, event) -> Self;
    pub fn with_timestamp(self, ts: i64) -> Self;   // override the wall-clock stamp; for ordered fixtures
    pub const fn event_kind(&self) -> &'static str;
}

AgentTraceEvent comes from the aleph_protocol crate (unified protocol); task_id and step_index are written by the dispatcher; event_kind() returns a stable string for indexing and filtering.

The integration test in src/routing/integration_tests.rs::observer_* verifies the full trace path: RoutingAttribution → OutcomeObserver::on_trace → RoutingExperienceStore::recall round-trips correctly.

Shadow Replay & state recovery

State machine

                 ┌──────────┐
                 │ Pending  │
                 └────┬─────┘
                      │ start
                 ┌────▼─────┐
         ┌──────│ Running  │──────┐
         │      └────┬─────┘      │
     SIGTERM       │        │   error
         │         │ success │     │
     ┌────▼──────┐  │  ┌────▼────┐ │
     │Interrupted│  │  │Completed│ │
     └───────────┘  │  └─────────┘ │
                    │              │
                    └─► Failed ◄───┘

is_recoverable() is true only for Running and Interrupted — those two statuses drive recovery decisions combined with RiskLevel and trace availability.

Decision logic (orchestrated by `agents/swarm/tasks/` + `harness/trace.rs`)

For each task where status = Running | Interrupted:
    │
    ├── No traces available?
    │   └── Skip ("no execution traces available")
    │
    ├── RiskLevel::Low?
    │   └── AutoResume (Shadow Replay + continue)
    │
    └── RiskLevel::High?
        └── PendingConfirmation (wait for user approval)

Read-only operations (file search, code analysis) are safe to retry automatically; write operations (file modification, shell commands) need human confirmation because the system state may have changed since the interrupt.

Trace data flow

Execution: every step writes a TaskTrace
    │
    ▼
harness::trace::TraceSink::on_trace(SessionCompleted { … })
    │
    ▼
gateway event bus routes to OutcomeObserver (src/routing/observer.rs)
    │
    ▼
RoutingExperienceStore.record (src/routing/experience_store.rs, sqlite-vec k-NN)

OutcomeObserver is the VESR write side; RoutingRecall::build_routing_experience_message is the read side (once per run, at run-start). They are paired through RoutingAttribution.task_emb (OnceLock<Vec<f32>>).

Fail-closed

26.7.21 hardening: when a policy file is missing or permissive_default disagrees with Default::default(), authorization is refused (never silently allowed). DiminishingReturnsDetector was removed from harness/tests/budget.rs — it was a deterministic completion judgment inside the loop, which R10 forbids. Hard stops are now bounded by max_iterations, the tool-loop verifier, and the model's own stop instead.

Guardian Judge & Heartbeat

26.7.21+: the LLM judge for heartbeat probes — payload masks secrets. Heartbeat probes refuse to use dangerous tools or confirmation-gated tools. See Heartbeat for the full spec.

Module layout

src/resilience/
├── mod.rs                       # only re-exports database + types
├── types.rs                     # AgentTask / TaskTrace / AgentEvent / Lane / RiskLevel / TaskStatus
└── database/
    ├── mod.rs                   # module entry
    ├── state_database/          # StateDatabase + schema migrations
    ├── channel_offsets.rs
    ├── events.rs                # AgentEvent persistence
    ├── group_chat.rs
    ├── memory_events.rs
    ├── migration.rs
    ├── paired_users.rs
    ├── tasks.rs                 # AgentTask CRUD
    └── traces.rs                # TaskTrace CRUD

The historical stub RPCs (heartbeat.*_stub / cron.*_stub) have been realized; the full RPC is driven by the Daemon, backed by src/tasks/cron/ and src/tasks/heartbeat/ and wired at startup in src/bin/aleph-server/commands/start/builder/handlers/agents.rs::register_cron_handlers / register_heartbeat_handlers.