Sep 8, 2026

Building the Agent Runtime: What the Model Doesn't Do

Every agent demo shows the prompt. The prompt is the tip of the iceberg. Underneath sits the runtime — the host process that loads the model, assembles context, executes tools, persists memory, enforces budgets, checkpoints state, and decides when to stop.

Xu et al. (2026) survey 200+ papers and conclude the unsolved problems are scheduling, recovery, state consistency, and cost control — all runtime problems. This post is a checklist for building a runtime that doesn’t embarrass you in production.

The Runtime Responsibilities (Exhaustive)

Subsystem	Responsibility	Failure Mode If Missing
Model Loader	Load weights, apply chat template, manage KV cache	OOM, wrong format, silent degradation
Context Manager	Assemble prompt: system + working memory + retrieved + history + headroom	Context rot, lost facts, budget overflow
Tool Executor	Validate schemas, run code, timeout, retry, compress results	Hanging calls, schema errors, token explosion
Memory System	Extraction (𝒮), Storage (ℛ), Retrieval (𝒬), Maintenance (𝒰) Zhou et al. 2026	Amnesia, stale facts, contradictions
Scheduler	Parallelize independent calls, sequence dependent, enforce concurrency limits	Deadlocks, latency spikes, budget blowout
Budget Enforcer	Token counter per turn, per session, per user; hard stops	Cost surprises, degraded quality
Checkpointing	Snapshot state (context + memory + tool results) every N turns	Unrecoverable failures, no replay
Eval Harness	Runs snapshot tests Sun et al. 2025, regression suite, canary deployments	Silent regressions, hallucination drift
Observability	Logs: tokens/category, tool latency, retrieval precision, US/HR, cost	Blind debugging, no capacity planning
Deployment	Blue/green, rollback, feature flags, config versioning	Bad deploy = downtime

Context Manager: The Heart

class ContextManager:
    def __init__(self, budget: TokenBudget):
        self.budget = budget
    
    def assemble(self, turn: Turn) -> Prompt:
        sections = [
            ("system", turn.system_prompt, Priority.CRITICAL),
            ("task_spec", turn.task_spec, Priority.CRITICAL),
            ("working_memory", turn.working_memory, Priority.HIGH),
            ("retrieved", self.retrieve(turn.query), Priority.HIGH),
            ("history", self.compress_history(turn.history), Priority.MEDIUM),
            ("few_shot", turn.few_shot, Priority.LOW),
        ]
        return self.pack(sections, self.budget)
    
    def pack(self, sections, budget):
        # 1. Sort by priority
        # 2. Fill from highest priority, truncate lowest
        # 3. Preserve primacy/recency for retrieved + history
        # 4. Return tokens_used for observability

Key invariants:

System prompt never truncated
Task spec never truncated
Retrieved context sorted by rerank score
History compressed via summarization (not truncation)
Primacy/recency placement: critical evidence at very top (after system) or very bottom (before generation)

Tool Executor: The Muscle

class ToolExecutor:
    def __init__(self, registry: ToolRegistry, compressor: Compressor):
        self.registry = registry
        self.compressor = compressor
    
    async def execute(self, calls: list[ToolCall]) -> list[ToolResult]:
        # 1. Validate all schemas in parallel (fail fast)
        # 2. Group: independent → parallel; dependent → sequence
        # 3. Run with timeout + retry (exponential backoff)
        # 4. Compress each result BEFORE returning to context
        # 5. Log: call_id, tool, latency, tokens_in, tokens_out, status

Non-negotiable: Every tool has a compress(result, query, max_tokens) method. Raw tool output never enters context.

Parallelism cap: Budget for the sum of parallel results. Either limit parallelism (max 2–3 retrieval calls/turn) or compress before injecting.

Memory System: The Four Modules

Following Zhou et al. (2026):

class MemorySystem:
    def __init__(self):
        self.R = HybridStorage(vector=pgvector, graph=kuzu, keyword=sqlite_fts)
        self.S = SchemaExtractor(schema=FactSchema)
        self.Q = HybridRouter(indices=["vector", "graph", "keyword"])
        self.U = MaintenanceScheduler(jobs=[consolidate, dedupe, version])
    
    def extract(self, dialogue: Dialogue) -> list[MemoryUnit]:
        return self.S.extract(dialogue)
    
    def store(self, units: list[MemoryUnit]):
        self.R.write(units)
    
    def retrieve(self, query: Query) -> list[MemoryUnit]:
        return self.Q.route_and_search(query)
    
    def maintain(self):
        self.U.run_due_jobs()

Deployment reality: Start with vector + keyword. Add graph when temporal/entity queries exceed 20% of workload. Run U nightly.

Scheduler: Concurrency Without Chaos

class Scheduler:
    def __init__(self, max_parallel: int = 3):
        self.semaphore = asyncio.Semaphore(max_parallel)
        self.dependency_graph = DependencyGraph()
    
    async def run(self, plan: Plan) -> list[Result]:
        # 1. Build DAG from plan (explicit deps or infer from tool signatures)
        # 2. Execute ready nodes in parallel (semaphore-limited)
        # 3. On failure: mark subgraph failed, continue independent branches
        # 4. Support checkpoint: save DAG state + completed results
        # 5. On retry: resume from checkpoint, re-run failed subgraph only

Key metric: Parallel efficiency = (sequential time) / (parallel time). Target > 2.5×.

Budget Enforcer: Tokens Are Money

class BudgetEnforcer:
    def __init__(self, limits: Limits):
        self.limits = limits  # per_turn, per_session, per_user, per_day
        self.usage = UsageTracker()
    
    def check(self, scope: Scope, estimated: int) -> bool:
        return self.usage.get(scope) + estimated <= self.limits.get(scope)
    
    def record(self, scope: Scope, actual: int):
        self.usage.add(scope, actual)
        if self.usage.get(scope) > self.limits.get(scope) * 0.9:
            alert("budget_near_limit", scope=scope, usage=self.usage.get(scope))

Hard stops: Per-turn (model max), per-session (user expectation), per-day (cost control). Soft alerts at 80%.

Checkpointing: Replayability

Every N turns (or on tool error), persist:

{
  "turn": 12,
  "context_hash": "sha256(...)",
  "memory_snapshot_id": "mem_abc123",
  "tool_results": [...],
  "model_output": "...",
  "budget_used": 45231
}

Enables: replay from any turn, A/B prompt testing on real trajectories, forensic debugging.

Eval Harness: Gate Every Deploy

# eval/gate.yml
stages:
  - name: snapshot_regression
    dataset: eval/snapshots/v3.jsonl
    metric: utility_score
    threshold: 0.70
  - name: hallucination_rate
    dataset: eval/snapshots/v3.jsonl
    metric: hallucination_rate
    threshold: 0.15
  - name: memory_update
    dataset: eval/memory/v1.jsonl
    metric: update_accuracy
    threshold: 0.85
  - name: cost_per_task
    dataset: eval/production_traces/last_week.jsonl
    metric: p95_tokens
    threshold: 8000

CI blocks on failure. Canary: 5% traffic → full metrics → promote/rollback.

Observability: The Dashboard You Need

Panel	Query
Tokens per turn (p50/p95)	`sum(tokens) by turn`
Tool latency (p99)	`histogram(tool_latency_ms) by tool`
Retrieval precision@k	`judge_score(retrieved, relevant)`
Utility Score / HR	`eval.mirage.us`, `eval.mirage.hr`
Cost per session	`sum(tokens * $/token) by session_id`
Error rate by type	`count(error_type) / total_turns`
Checkpoint lag	`now - last_checkpoint_ts`

The Runtime Checklist Before Launch

The Model Is Interchangeable

Swap GPT-4o → Claude 3.5 → Llama-3.1-405B. The runtime stays. The prompt templates adapt. The tool schemas stay. The memory system stays. The eval harness stays.

The runtime is your product. The model is your supplier. Negotiate accordingly.