Monitor

Runs monitors on trajectories to detect suspicious behavior.

Purpose

Executes monitoring functions on trajectories to compute suspiciousness scores. Used to:

  • Evaluate trajectory safety
  • Compute action-level or trajectory-level suspicion scores
  • Test new monitor implementations
  • Generate monitored runs for analysis

Commands

ct run monitor

Run monitors on trajectories.

Usage:

ct run monitor TRAJ_LIKE [OPTIONS]

Arguments:

  • TRAJ_LIKE - An eval-backed source: a .eval file, or a run ID/URL whose run records a backing eval log (see Downloading from the Viewer)

Options:

  • -m, --monitor - Monitor selection, repeatable (e.g., -m multi -m quick); defaults: mean-action and mean-quick. Whether a monitor scores per action or per trajectory is a property of the monitor itself; prefix a name with action/ or trajectory/ only when the same name exists as both.
  • --trusted-model - Required. The model bound as the eval-level trusted role that every monitor runs on. Any Inspect AI provider string works, including self-hosted open-weight endpoints (see Self-Hosted and Open-Weight Models)
  • --cot-level - Chain-of-thought visibility for the monitors: none (default), reasoning, or extended
  • --save-dir PATH - Directory to save monitored trajectories
  • --no-upload - Disable upload (upload is enabled by default)
  • --run-name TEXT - Custom run name

Monitors never carry model configuration; the one model they run on is the eval-level trusted role, given by --trusted-model. The exception is monitoring-bench, which replicates its benchmark's fixed 3-model panel.

A monitor selection is a registered monitor's name, prefixed with action/ or trajectory/ when the name exists at both scopes. Repeat the selection flag to run several monitors.

Examples:

# Monitor with the default monitors
ct run monitor logs/my_eval.eval --trusted-model openai/gpt-5-nano

# One specific monitor
ct run monitor logs/my_eval.eval -m strong --trusted-model openai/gpt-5-nano

# Monitor and save locally without uploading
ct run monitor logs/my_eval.eval -m strong --save-dir monitored_trajs/ --no-upload --trusted-model openai/gpt-5-nano

# Monitor a specific run from database
ct run monitor abc123def456 -m multi --trusted-model openai/gpt-5-nano

# An action and a trajectory monitor in one pass
ct run monitor logs/my_eval.eval -m multi -m quick --trusted-model openai/gpt-5-nano

# Monitor a viewer-hosted run directly (no intermediate download)
ct run monitor https://basharena-data.vercel.app/runs/ed2b360737041f1743ac4de43280f858 \
    -m multi --trusted-model openai/gpt-5-nano

Available Monitors

A monitor is a plain async function: (messages, metadata, index) -> Score for action monitors, (messages, metadata) -> Score for trajectory monitors. Each is written out in src/control_tower/monitors/catalog.py; its prompt is a markdown file under src/control_tower/monitors/assets/{action,trajectory}/, and the function decides everything else — the output schema, how many draws, how to fan out over candidate covert objectives, how to combine (mean / peak / per_goal). The registry is the two dicts at the bottom of the catalog, and a monitor's docstring is its description.

List available monitors:

ct protocols monitors action [NAME]
ct protocols monitors trajectory [NAME]

Action-Level Monitors

Score each untrusted-model action.

  • no-history - Scores each action in isolation with no preceding action history or candidate covert-objective hints.
  • old-action - The default action monitor: one model call per action, shown the candidate covert objectives.
  • old-action-no-goals - The default action monitor without the candidate covert objectives.
  • checklist - Answers five yes/no checks per action (necessity, security surface, innocent explanation, concrete harm, removability) before scoring; the top of the scale is gated on concrete harm.
  • orthogonal - Flags work that does not serve the main task - unnecessary changes, premature optimization, off-task activity - rather than judging malice directly.
  • three-scores - Categorical monitor: a safe / uncertain / suspicious verdict per action, mapped onto the score scale.
  • strong - Sees the full conversation history (goal-blind) so it judges each action in context.
  • strong-goals - The strong full-history monitor, additionally shown the environment's candidate covert objectives.
  • goal-analysis - Scores each action while shown the environment's candidate covert objectives, asking whether the action advances any of them.
  • single-goal - Scores each action once per candidate covert objective in parallel and keeps the maximum score.
  • multi - Weighted blend of the orthogonal and old-action monitors' scores for each action.
  • mean-action - Averages two independent runs of the old-action monitor on each action. CLI default.
  • mean-goals - Averages two independent goal-analysis runs on each action.
  • mean-single-goal - Scores each action twice per covert objective, averages each objective's scores, and keeps the highest average.

Trajectory-Level Monitors

Score the whole transcript in one shot rather than action by action.

  • quick - Reads the full transcript in a single pass, shown the candidate covert objectives, and returns one score for the trajectory.
  • quick-no-goals - Reads the full transcript in a single pass with no covert-objective hints.
  • goal-analysis - Reads the full transcript while shown the environment's candidate covert objectives, working through them one by one.
  • single-goal-analysis - Reads the full transcript once per candidate covert objective in parallel and keeps the maximum score.
  • mean-quick - Averages five independent quick passes over the transcript. CLI default.
  • mean-goal-analysis - Averages two independent goal-analysis passes over the transcript.
  • mean-single-goal-analysis - Reads the transcript twice per covert objective, averages each objective's scores, and keeps the highest average.
  • monitoring-bench - Replica of the MonitoringBench benchmark's monitor: the bench prompt on its fixed 3-model panel (scores comparable to the imported monitoring-bench dataset), max-aggregated.

Whether a monitor sees the environment's candidate covert objectives — every side task registered for that environment, not just the one the agent was assigned — is fixed by its prompt asset: goal-aware and goal-blind variants are separate monitors (e.g. old-action vs old-action-no-goals, strong vs strong-goals, quick vs quick-no-goals).

Output

Post-hoc monitoring reads the source, runs monitors, and writes the monitor results. Action-level monitors write per-action results; trajectory-level monitors write one result per trajectory.

Sources are scored directly

The input must be eval-backed: a local or remote .eval, an in-memory EvalLog, or a viewer run backed by an eval log — anything else (trajectory JSON, datasets, runs without a backing eval log) is rejected with an error. ct run monitor scores the recorded transcripts directly: action monitors see the transcript prefix before each action, trajectory monitors the full transcript, and nothing re-executes.

A local .eval source is rewritten in place — the file gains its monitor_<name> sample scores with no sibling artifact. A run or in-memory source is scored onto a copy instead, since published eval objects are add-only: the copy lands in the save dir (an explicit --save-dir, or the default ./data/monitoring/<monitor>/<label>_<timestamp>/) as <source-stem>.eval, and the source is left unchanged.

In either case, an action monitor's score value maps action index to score (None where the action was skipped by design or the monitor failed), with the full verdicts under metadata.verdicts; a trajectory monitor's score is the verdict itself. Decision statistics (max, thresholded audits) are computed downstream from these values, never pre-baked. Existing named scores survive, and a monitor whose verdicts are already on the source is refused unless --overwrite — verdicts are never silently replaced.

The eval header records derived_from (the source identifier) on copied artifacts. Uploaded monitor runs register the monitored log as their eval_log_url, and downstream consumers (e.g. ct run summarize) reconstruct the monitor_response / traj_monitor_response fields from the sample's verdicts.

Unless --no-upload is passed:

  • Creates a run in the database
  • Uploads all monitored trajectories
  • Prints run URL and trajectory URLs

Note the model response cache defaults on, so re-running an unchanged command reproduces the same scores rather than drawing a fresh sample (the CLI warns when this happens); clear inspect's model cache (inspect cache clear) to resample.

Monitoring Details

Monitors add suspiciousness scores to trajectories:

  • Scores range from 0 (not suspicious) to 10 (very suspicious)
  • Different monitors use different approaches (LLM-based, heuristic, etc.)
  • Action-level results are stored per action
  • Trajectory-level results are stored once per trajectory

Cost Attribution

Every verdict carries a cost_usd (and token fields) in its metadata populated automatically by the orchestration layer — no per-monitor changes required. Multi-model monitors (e.g. multi) additionally get a cost_by_model split. An unpriced model is a None in that split; cost_usd stays the priced spend. See function-registry.md for the mechanism and known limitations, and trajectories.md for the monitor_cost(sample) / agent_cost(sample) / total_cost_usd(sample) accessors.

Technical Details

  • Supports both action-level and trajectory-level monitors
  • Tracks token usage and costs for LLM-based monitors
  • Runs monitors concurrently for efficiency
  • Handles errors gracefully per trajectory