Agent Observability: Logging, Tracing, and Monitoring

You can’t improve what you can’t measure. Here’s how to build observability into your agent systems from day one.

The Three Pillars of Agent Observability

Pillar What It Shows Tools Agent-Specific Considerations
Logs Discrete events Structured JSON logs Tool calls, LLM prompts/outputs, decisions
Metrics Aggregated data Counters, histograms, gauges Task success rate, latency, token usage, cost
Traces Request journeys Distributed tracing (OpenTelemetry) Multi-tool workflows, agent-to-agent calls

1. Structured Logging: The Foundation

What to Log

import structlog
from datetime import datetime
from typing import Any, Dict

logger = structlog.get_logger()

class AgentLogger:
    def log_llm_call(self, session_id: str, turn: int, model: str, 
                     prompt: str, completion: str, 
                     prompt_tokens: int, completion_tokens: int,
                     latency_ms: int, cost_usd: float):
        logger.info("llm_call",
            session_id=session_id,
            turn=turn,
            model=model,
            prompt_tokens=prompt_tokens,
            completion_tokens=completion_tokens,
            latency_ms=latency_ms,
            cost_usd=cost_usd,
            prompt_length=len(prompt),
            completion_length=len(completion)
        )
    
    def log_tool_call(self, session_id: str, turn: int, tool: str,
                      args: Dict[str, Any], result: Any,
                      latency_ms: int, success: bool):
        logger.info("tool_call",
            session_id=session_id,
            turn=turn,
            tool=tool,
            args_keys=list(args.keys()),
            result_type=type(result).__name__,
            latency_ms=latency_ms,
            success=success,
            result_size=len(str(result)) if result else 0
        )
    
    def log_decision(self, session_id: str, turn: int, 
                     decision: str, reasoning: str,
                     confidence: float, alternatives: list):
        logger.info("agent_decision",
            session_id=session_id,
            turn=turn,
            decision=decision,
            reasoning=reasoning[:200],  # Truncate for storage
            confidence=confidence,
            alternatives_count=len(alternatives)
        )
    
    def log_state_change(self, session_id: str, turn: int,
                         variable: str, old_value: Any, new_value: Any):
        logger.info("state_change",
            session_id=session_id,
            turn=turn,
            variable=variable,
            old_value=str(old_value)[:100],
            new_value=str(new_value)[:100],
            changed=old_value != new_value
        )

Log Storage and Querying

Store logs in a system that supports:

  • JSON parsing (for structured querying)
  • Time-range queries (find all logs for a session in last hour)
  • Full-text search (find all logs containing “timeout”)
  • Aggregation (count tool calls by type)

Popular choices: Elasticsearch, Loki, Datadog, Splunk, or even SQLite for small scale.

Query examples:

-- Find all LLM calls for a session
SELECT * FROM logs 
WHERE session_id = 'session:abc123' 
  AND type = 'llm_call'
  AND timestamp > '2026-09-08T10:00:00Z'
  ORDER BY turn;

-- Calculate average latency per tool
SELECT tool, AVG(latency_ms) as avg_latency
FROM logs
WHERE type = 'tool_call' AND success = true
GROUP BY tool
ORDER BY avg_latency DESC;

-- Find sessions with high error rates
SELECT session_id, 
       COUNT(*) as total_calls,
       SUM(CASE WHEN success = false THEN 1 ELSE 0 END) as failed_calls
FROM logs
WHERE type = 'tool_call'
  AND timestamp > '2026-09-08T00:00:00Z'
GROUP BY session_id
HAVING failed_calls * 1.0 / total_calls > 0.1
ORDER BY failed_calls DESC;

2. Distributed Tracing: Seeing the Full Picture

Agents often make multiple tool calls and may call other agents. Tracing shows the full journey.

Trace Structure for an Agent Turn

Trace: agent_turn_123
├── Span: llm_call (gpt-4o-mini)
│   └── Attributes: model, tokens, cost
├── Span: tool_call (web_search)
│   ├── Attributes: tool, args, success
│   └── Event: result_received (15 results)
├── Span: tool_call (web_extract) × 3
│   ├── Attributes: urls, success
│   └── Event: content_extracted (avg 2.3k chars)
├── Span: llm_call (synthesis)
│   └── Attributes: prompt_tokens, completion_tokens
└── Span: agent_decision
    └── Attributes: decision, confidence

Implementation with OpenTelemetry

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Setup
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)
otlp_exporter = OTLPSpanExporter(endpoint="http://localhost:4317")
span_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)

# In your agent
class TracedAgent:
    async def run_turn(self, session_id: str, turn: int, input_data: dict):
        with tracer.start_as_current_span(f"agent_turn_{turn}") as turn_span:
            turn_span.set_attribute("agent.session_id", session_id)
            turn_span.set_attribute("agent.turn", turn)
            turn_span.set_attribute("agent.input_size", len(str(input_data)))
            
            # LLM call span
            with tracer.start_as_current_span("llm_call") as llm_span:
                llm_span.set_attribute("llm.model", "gpt-4o-mini")
                # ... make call ...
                llm_span.set_attribute("llm.prompt_tokens", 1500)
                llm_span.set_attribute("llm.completion_tokens", 400)
            
            # Tool call spans
            for tool_call in planned_tool_calls:
                with tracer.start_as_current_span(f"tool_{tool_call.name}") as tool_span:
                    tool_span.set_attribute("tool.name", tool_call.name)
                    tool_span.set_attribute("tool.args", str(tool_call.args))
                    # ... execute tool ...
                    tool_span.set_attribute("tool.success", True)
                    tool_span.set_attribute("tool.latency_ms", 1200)
            
            # Decision span
            with tracer.start_as_current_span("agent_decision") as decision_span:
                decision_span.set_attribute("agent.decision", chosen_action)
                decision_span.set_attribute("agent.confidence", 0.85)
                
            turn_span.set_attribute("agent.output_size", len(str(output)))
            return output

What Traces Reveal

  1. Latency bottlenecks - Which steps take the longest?
  2. Error propagation - Where did failures start?
  3. Tool usage patterns - Which tools are called most/least?
  4. Agent-to-agent communication - See the full chain of delegation
  5. Retry effectiveness - Did retries actually help?

Visualization: Use Jaeger, Tempo, or Datadog APM to see trace waterfalls.

3. Metrics: The Vital Signs

Key Metrics to Track

Workflow Metrics

  • agent_tasks_total (counter) - Total tasks processed
  • agent_tasks_successful (counter) - Successfully completed tasks
  • agent_tasks_failed (counter) - Failed tasks
  • agent_task_duration (histogram) - Time to complete task
  • agent_turns_per_task (histogram) - Average turns needed

LLM Metrics

  • llm_calls_total (counter) - Total LLM API calls
  • llm_input_tokens (counter) - Total input tokens consumed
  • llm_output_tokens (counter) - Total output tokens generated
  • llm_cost_usd (counter) - Total cost in USD
  • llm_latency (histogram) - LLM response time

Tool Metrics

  • tool_calls_total (counter) - Total tool invocations
  • tool_calls_successful (counter) - Successful tool calls
  • tool_calls_failed (counter) - Failed tool calls
  • tool_latency (histogram) - Tool response time (by tool type)
  • tool_rate_limited (counter) - Calls rate-limited by external APIs

Resource Metrics

  • agent_concurrent_tasks (gauge) - Currently running tasks
  • agent_memory_usage (gauge) - Memory consumption
  • agent_context_length (gauge) - Average context length used
  • agent_cache_hits (counter) - Semantic/exact match cache hits
  • agent_cache_misses (counter) - Cache misses

Business Metrics

  • agent_cost_per_task (gauge) - Average cost to complete a task
  • agent_faithfulness_score (gauge) - % of outputs supported by sources
  • agent_hallucination_rate (gauge) - % of outputs with hallucinations
  • agent_user_satisfaction (gauge) - From feedback/thumbs up/down

Implementation Example (Prometheus-style)

from prometheus_client import Counter, Histogram, Gauge

# Define metrics
TASKS_TOTAL = Counter('agent_tasks_total', 'Total tasks processed', ['site_id', 'status'])
TASK_DURATION = Histogram('agent_task_duration_seconds', 'Task duration', ['site_id'])
LLM_TOKENS = Counter('agent_llm_tokens_total', 'LLM tokens consumed', ['type', 'model'])
TOOL_CALLS = Counter('agent_tool_calls_total', 'Tool calls', ['tool', 'outcome'])
CONTEXT_LENGTH = Gauge('agent_context_length_tokens', 'Context length used', ['session_id'])
COST_PER_TASK = Gauge('agent_cost_per_task_usd', 'Average cost per task', ['site_id'])

# In your agent workflow
async def process_task(self, task_id: str, site_id: str):
    start_time = time.time()
    try:
        # ... process task ...
        TASKS_TOTAL.labels(site_id=site_id, status="success").inc()
    except Exception as e:
        TASKS_TOTAL.labels(site_id=site_id, status="failed").inc()
        raise
    finally:
        duration = time.time() - start_time
        TASK_DURATION.labels(site_id=site_id).observe(duration)
        
        # Update cost per task (would be calculated from actual usage)
        # COST_PER_TASK.labels(site_id=site_id).set(calculated_cost)

4. Alerting: Knowing When Things Go Wrong

Set up alerts for anomalous behavior.

Critical Alerts

groups:
- name: agent-alerts
  rules:
  - alert: AgentHighFailureRate
    expr: rate(agent_tasks_total{status="failed"}[5m]) / rate(agent_tasks_total[5m]) > 0.1
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "High agent failure rate ({{ $value | printf \"%.1%\" }})"
      description: "More than 10% of agent tasks are failing over the last 5 minutes."

  - alert: AgentLLMCostSpike
    expr: increase(agent_llm_cost_usd[1h]) > 100  # $100/hour
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "LLM cost spike detected"
      description: "Agent LLM usage costs are increasing rapidly."

  - alert: AgentHallucinationRateHigh
    expr: agent_hallucination_rate > 0.05
    for: 10m
    labels:
      severity: critical
    annotations:
      summary: "High hallucination rate ({{ $value | printf \"%.1%\" }})"
      description: "Agent is producing unsupported facts at an unacceptable rate."

  - alert: AgentContextLengthGrowing
    expr: deriv(agent_context_length_tokens[1h]) > 1000  # Growing by 1k tokens/hour
    for: 15m
    labels:
      severity: warning
    annotations:
      summary: "Agent context length growing uncontrollably"
      description: "Context length is increasing, possibly indicating a memory leak or runaway conversation."

  - alert: AgentToolFailureBurst
    expr: rate(agent_tool_calls_total{outcome="failed"}[5m]) > 10
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Burst of tool failures"
      description: "Multiple tool calls are failing rapidly - check external API health."

5. Agent-Specific Observability Patterns

1. Turn-Based Logging

Structure logs around agent turns (input → thinking → tool use → output → decision).

{
  "timestamp": "2026-09-08T14:30:00Z",
  "session_id": "session:abc123",
  "turn": 5,
  "phase": "tool_call_start",
  "tool": "web_extract",
  "args": {"urls": ["https://example.com/article"]},
  "duration_ms": null
}
{
  "timestamp": "2026-09-08T14:30:02Z",
  "session_id": "session:abc123",
  "turn": 5,
  "phase": "tool_call_end",
  "tool": "web_extract",
  "success": true,
  "result_size": 4520,
  "duration_ms": 2100
}

2. Decision Traceability

Log not just what decision was made, but why.

{
  "timestamp": "2026-09-08T14:30:05Z",
  "session_id": "session:abc123",
  "turn": 5,
  "phase": "decision_made",
  "decision": "claim_topic",
  "reasoning": "Topic has high signal strength, matches researcher expertise, and commitment hash can be verified",
  "confidence": 0.85,
  "alternatives_considered": ["abandon_topic", "escalate_to_human"],
  "evidence": {
    "signal_strength": 0.78,
    "topic_age_hours": 2.3,
    "researcher_match": true
  }
}

3. Tool Contract Monitoring

Verify tools behave as expected.

{
  "timestamp": "2026-09-08T14:30:10Z",
  "session_id": "session:abc123",
  "turn": 5,
  "phase": "tool_contract_violation",
  "tool": "web_search",
  "expected": "Returns list of results with title, url, snippet",
  "actual": "Returned empty list",
  "severity": "medium",
  "likely_cause": "API rate limit or network issue"
}

4. Token Budget Tracking

Monitor token usage in real-time.

class TokenBudgetTracker:
    def __init__(self, max_tokens_per_turn: int = 8000):
        self.max_tokens = max_tokens_per_turn
        self.usage = defaultdict(int)  # session_id -> tokens used
    
    def add_usage(self, session_id: str, tokens: int):
        self.usage[session_id] += tokens
        if self.usage[session_id] > self.max_tokens:
            logger.warning("token_budget_exceeded",
                         session_id=session_id,
                         used=self.usage[session_id],
                         limit=self.max_tokens)
            return False
        return True

6. Production Observability Checklist

Logging ✅

  • All LLM calls logged (prompt, completion, tokens, cost, latency)
  • All tool calls logged (tool, args, result, latency, success)
  • All agent decisions logged (decision, reasoning, confidence, alternatives)
  • State changes logged when significant
  • Logs stored as structured JSON
  • Log retention policy defined (e.g., 30 days hot, 90 days warm)

Tracing ✅

  • Distributed tracing enabled (OpenTelemetry or equivalent)
  • Each agent turn is a trace
  • LLM calls, tool calls, and decisions are spans
  • Trace context propagated to child agents/tools
  • Traces exported to backend (Jaeger, Tempo, etc.)
  • Trace sampling configured (e.g., 10% for high volume)

Metrics ✅

  • Workflow metrics (tasks, success rate, duration)
  • LLM metrics (calls, tokens, cost, latency)
  • Tool metrics (calls, success rate, latency by tool)
  • Resource metrics (concurrent tasks, memory, context length)
  • Business metrics (cost/task, faithfulness, hallucination rate)
  • Metrics scraped by Prometheus or sent to monitoring system

Alerting ✅

  • Critical alerts configured (failure rate, cost spikes, hallucinations)
  • Warning alerts configured (context growth, tool failures)
  • Alerts routed to appropriate channels (PagerDuty, Slack, email)
  • Alert suppression during known maintenance windows
  • Alert fatigue prevention (rate limiting, deduplication)

Agent-Specific ✅

  • Turn-based logging implemented
  • Decision traceability with reasoning
  • Tool contract monitoring
  • Token budget tracking per session
  • Semantic cache hit/miss metrics
  • Commitment hash verification logging
  • Anonymous session tracking (session:xxx IDs)

7. Example: Debugging a Production Issue

Alert Fires: AgentHighFailureRate - 15% failure rate over 5 minutes

Investigation:

  1. Check logs for failed sessions:

    session_id=session:def456 turn=12 tool=web_extract success=false error="Timeout after 30s"
session_id=session:def456 turn=13 tool=web_extract success=false error="Timeout after 30s"
session_id=session:def456 turn=14 decision=abandon_topic reasoning="Repeated tool failures"

  2. Check traces for one failed session:

    • See that web_extract calls consistently take 25-30s (timeout threshold)
    • Notice that the same URLs are being retried
    • Observe that web_search preceding them returned those URLs

  3. Check metrics:

    • tool_latency{tool="web_extract"} shows 95th percentile at 28s (up from 2s baseline)
    • tool_calls_failed{tool="web_extract"} shows spike
    • agent_context_length shows normal values

  4. Conclusion: External web extraction service is degraded or blocking the agent’s IP.

Resolution:

  • Add circuit breaker for web_extract
  • Implement fallback to search snippets when extraction fails
  • Alert external team about service degradation
  • Post-mortem: Add timeout and retry logic with exponential backoff

The Takeaway

Observability is not optional for production agents.

You need to:

  1. Log everything - Structured, queryable, turn-based
  2. Trace everything - See the full journey from input to action
  3. Measure everything - Workflow, LLM, tool, resource, and business metrics
  4. Alert on anomalies - Catch issues before they impact users
  5. Agent-specific patterns - Turns, decisions, tool contracts, token budgets

Start with logging, add tracing, then metrics, then alerting.
Instrument your agents from day one - retrofitting observability is painful.

“In God we trust. All others must bring data.”
For agents, that data comes from observability. Build it in, use it constantly, and let it guide your improvements.