AI Providers

Aleph exposes model backends through the object-safe AiProvider trait. Protocol adapters translate unified message history, tools, media, and generation settings to each upstream API, while decorators add model overrides, thinking level, metering, MoA, and provider failover without putting recovery policy in the agent loop.

Provider Contract

The core call is AiProvider::process(RequestPayload) -> ProviderResponse. Providers also report protocol and serving provider/model hints so behavior selection, pricing, and attribution use the endpoint that actually served the request rather than a wrapper name.

Streaming uses execute_streaming_dyn and supports_streaming. The outermost decorator must preserve the streaming path. The default implementation replays a completed response to the sink, so a missing specialization degrades from live streaming to batched delivery rather than producing silence.

Configuration

Providers are entries in the plural top-level table:

[providers.primary]
protocol = "anthropic"
models = ["claude-sonnet-4-6"]
enabled = true

[providers.local]
protocol = "ollama"
models = ["qwen3"]
base_url = "http://127.0.0.1:11434"
enabled = true

[general]
default_provider = "primary"
fallback_providers = ["local"]

models is ordered and its first entry is the default. model = "..." remains a backward-compatible input alias. Common fields include protocol, models, base_url, enabled, request and stream-idle timeouts, context-window override, sampling controls, cache retention, model behavior, and protocol-specific options.

API keys are runtime-only values loaded from the encrypted vault and are not persisted by ProviderConfig. Use provider setup or secret-management surfaces rather than writing credentials into the provider table.

The singular [provider.<name>] path is not valid. Provider entries also do not have a max_failures field; failover membership and policy are configured separately.

Failover Walk

FailoverProvider is a decorator around a live primary and an ordered provider/model candidate chain. The harness sees one provider and does not implement retries or select fallbacks.

For each request, the provider layer:

1. Resolves the live primary and effective fallback membership.
2. Applies explicit local/cloud route policy and capability requirements.
3. Orders eligible candidates using configured order or load-balancing policy.
4. Skips open circuits and active provider/model cooldowns where possible.
5. Retries or advances according to the typed failure decision.
6. Records health, cooldown, load, usage, serving provider, and serving model.

Configured fallback names reference keys in [providers]. If no explicit chain is configured, the production chain can derive fallback membership from the live provider registry; provider additions and removals then affect the next request. effective_fallback_names is shared by the walk and route_status, so diagnostics describe the chain that is actually used.

A streaming candidate may fail over only before user-visible content has been emitted. Once content has reached the sink, the emission guard stops the walk rather than appending a second provider's answer to a partial first answer.

Live Route State

[route] controls local/cloud preference, optional cloud escalation, load balancing, provider targets, and rate ceilings. RouteHandle stores the entire route state in one ArcSwap: a write publishes one generation atomically, and each request reads one coherent snapshot. The next request therefore sees either all old fields or all new fields, never a torn mixture.

Candidate slot and endpoint tier are separate concepts. The primary slot owns an explicitly selected model; fallback slots use their own model catalogs. Local/cloud tier is used only for route policy and cost ordering.

Reliability and Accounting

• Transient overload, timeout, and rate-limit failures use bounded retry and backoff decisions.
• Circuit-breaker, provider cooldown, and model cooldown state is shared across global and per-agent chains.
• Provider response body reads and streaming idle gaps are bounded.
• Cache read and creation tokens are included when enforcing throughput windows.
• Metering attributes cost and usage to the actual serving provider/model.
• route_status exposes chain membership, health, cooldown, load, and route decisions.

The failover hot path borrows the request conversation instead of cloning the complete history for every attempt. The provider refactor also removed roughly 2,000 lines of unreachable parallel provider code, leaving the AiProvider/protocol-adapter/failover path as the maintained runtime.

Code Locations

• src/providers/mod.rs — AiProvider and provider construction
• src/providers/protocols/ — protocol adapters
• src/providers/http_provider.rs — HTTP execution and live streaming
• src/providers/failover/ — candidate walk, decisions, circuits, and cooldowns
• src/providers/route_policy.rs — candidate gating and ordering
• src/providers/route_handle.rs — atomic live route snapshots
• src/providers/route_observe.rs — route_status rendering
• src/config/types/provider.rs — [providers.<name>] entry schema

See Also

• Execution Engine — how provider calls enter a run
• Secret Management — credential storage
• Generation — image, audio, and video providers