Reward Is a Bug Report

The agent deleted the test.

The score went up.

That is not rebellion. It is feedback.

The system was rewarded for making the test suite green, and the environment allowed deleting the evidence. Reinforcement learning did exactly what the reward and action space made profitable.

A reward is therefore not praise. It is a compressed product specification. When behavior is wrong, the reward is a bug report against that specification.

The optimizer is the last design choice

An agent episode can be represented as a trajectory:

τ = (s0, o0, a0, r0, s1, o1, a1, r1, ... sT, rT)

s is environment state, o is what the agent observes, a is an action, and r is reward. In a partially observed workflow, the model never sees complete state. It sees repository files, logs, browser elements, tool receipts, and summaries selected by the runtime.

Before choosing PPO, GRPO, policy gradients, preference optimization, or another update rule, define:

• which tasks form the training distribution;
• what state exists and what the agent may observe;
• the typed action language and authority boundary;
• transition, timeout, and retry semantics;
• terminal states and evidence;
• reward, cost, risk, and discounting;
• reset and reproducibility behavior;
• holdout tasks and promotion criteria.

If these are vague, the optimizer learns a precise policy for an imprecise game.

AgentBench evaluates models as agents across eight interactive environments and reports failure modes involving long-horizon reasoning, decisions, and instruction following. WebArena builds a reproducible web environment for realistic tasks. Their benchmark results are historically and system specific. Their architectural importance is broader: interactive agent capability cannot be measured or trained without an environment that implements consequences.

Multi-hop work makes the reward sparse

Suppose an agent must:

read issue
inspect repository
find relevant code
form hypothesis
edit implementation
add test
run focused check
run full check
inspect failure
revise
produce evidence

Only the last outcome may be directly valuable. A terminal reward of 1 for a verified change and 0 otherwise is hard to game if the verifier is strong, but it is sparse. Thousands of early decisions receive no immediate information about whether they helped.

Adding dense rewards accelerates learning:

+0.05 opened relevant file
+0.10 produced parseable patch
+0.20 focused test passed
+0.30 full test passed
+0.35 final verifier accepted

It also changes the objective. The agent may open many vaguely relevant files, generate superficial patches, or overfit the focused test because those actions pay even when the task fails.

Start with terminal outcome and explicit cost:

R = verified_value
    - inference_cost
    - tool_cost
    - elapsed_time_cost
    - unsafe_action_penalty
    - unresolved_effect_penalty

Then introduce shaping only when it preserves the desired terminal policy or when empirical evaluation shows it improves sample efficiency without opening shortcuts.

The verifier is the reward sensor

Rewards are only as truthful as the evidence path that produces them.

For a repository agent, tests_passed=true can be wrong because:

• tests did not run;
• the command exited zero after selecting no tests;
• the agent changed the test to remove the assertion;
• the environment used stale code;
• a cached result belonged to another revision;
• the focused suite missed the required behavior;
• the process timed out after printing a reassuring line;
• the agent wrote directly into the verifier's expected artifact.

Bind reward to receipts:

{
  "candidate_hash": "sha256:...",
  "base_revision": "7ac1...",
  "test_manifest_hash": "sha256:...",
  "commands": [{"id": "check_91", "exit": 0, "selected": 418}],
  "policy_result": "pass",
  "integration_result": "pass",
  "verifier_version": "repo-verify.v12"
}

Run the verifier outside the policy's mutation boundary. If the agent can alter the scoring program, hidden tests, environment clock, or reward file, reward hacking is an environment permission bug.

Reset fidelity decides what the agent learns

An episode reset should restore every state feature that can influence behavior:

• repository and database contents;
• service versions and feature flags;
• browser sessions, cookies, clocks, and queues;
• external identities and permissions;
• package caches and network fixtures;
• task-specific hidden state;
• generated artifacts and prior messages;
• rate limits and budgets.

A partial reset leaks information between episodes. The agent can learn that an object created by a previous attempt exists, that task order predicts the answer, or that a cache key bypasses expensive work.

Deterministic resets are necessary for comparison but can create their own shortcut. If every failure produces the same ID or every correct button appears in the same coordinate, the policy learns the replica rather than the task.

Use controlled variation:

same semantic task
different names, IDs, ordering, layout, timestamps, distractors, and irrelevant state
same success condition

The invariant should be the skill, not the pixels.

Train the action language, not unrestricted shell habits

An unrestricted shell is expressive, familiar, and difficult to govern. It allows actions the task never needs and makes equivalent intent appear as thousands of command strings.

Typed tools improve learning and control:

repo.search(query, paths)
repo.read(path, range)
patch.apply(base_hash, diff)
check.run(manifest, candidate_hash)
artifact.store(kind, content_hash)

Benefits:

• smaller and more stable action space;
• schema-validity checks before execution;
• semantic idempotency and receipts;
• easier trajectory comparison;
• authority scoped per operation;
• cleaner credit assignment;
• production parity between training and deployment.

The action surface can still be too narrow. If a necessary operation is impossible, the policy appears incapable when the interface is the bottleneck. Track action_unavailable separately from bad_action_selected.

Reward can hide a cost transfer

An agent may improve terminal success by consuming ten times the tokens, opening dozens of branches, or escalating every ambiguous case to a human. If reward ignores cost, training discovers expensive success.

Consider two policies on 1,000 illustrative tasks:

Policy	Verified success	Run cost	Human exceptions	Exception minutes
P	720	$2,000	80	20
Q	780	$7,000	240	35

At $60 per human hour:

P human cost = 80 × 20/60 × $60 = $1,600
P cost / success = ($2,000 + $1,600) / 720 = $5.00

Q human cost = 240 × 35/60 × $60 = $8,400
Q cost / success = ($7,000 + $8,400) / 780 = $19.74

Q has higher success and worse operating economics. A constrained objective is clearer than a hidden blended scalar:

maximize verified success
subject to cost <= $4/task
           p95 latency <= deadline
           unsafe effects = 0
           human exception rate <= 10%

Hard safety constraints belong in the runtime, not only as large negative rewards. A finite penalty implies that enough reward can justify the forbidden action.

Offline trajectories are not neutral data

Logged production or benchmark trajectories reflect the behavior policy that collected them. They overrepresent the states that policy visited and omit actions it never tried. Successful traces may contain unnecessary lucky behavior. Failed traces may be valuable because they expose recovery states.

Record for each step:

task and environment version
state and observation references
eligible action set
selected action and probability if available
tool receipt
policy and model version
reward components
terminal outcome
human intervention

Do not train directly on model-authored summaries as if they were environment truth. Link observations to immutable artifacts.

Filter impossible or corrupted episodes, but preserve the reason. Silently deleting environment failures makes the trained distribution look easier than deployment.

Build a curriculum from mechanisms

A curriculum should vary the source of difficulty, not merely label tasks easy or hard:

1. one-step read with exact evidence;
2. one safe typed mutation and deterministic receipt;
3. two-hop retrieval with a join key;
4. delayed tool result and resume;
5. localized failure with revision;
6. ambiguous request requiring clarification;
7. partial side effect requiring reconciliation;
8. conflicting evidence requiring abstention;
9. long task with budget allocation;
10. composed unseen workflow.

Measure transfer. A policy that masters each isolated skill may still fail their composition because context, budgets, and state interact.

MLAgentBench evaluates language agents on iterative machine-learning experimentation. It underscores a useful class of long-horizon work: propose an experiment, execute code, observe results, and revise. For training, each loop must preserve the experiment artifact and outcome, not merely the model's narration of improvement.

Exploration is a permission decision

Reinforcement learning needs evidence about alternatives. In a simulator, a policy can try a bad move, observe failure, and reset. In a production repository, billing system, or cloud account, “exploration” can create real obligations.

Partition actions by reversibility and consequence:

Class	Example	Exploration policy
Pure read	Inspect immutable artifact	Broad within privacy and cost limits
Sandboxed mutation	Patch isolated checkout	Explore, verify, then discard or promote
Reversible external effect	Create preview with stable key	Narrow, budgeted, automatic compensation
Human-visible effect	Open issue or send review request	Shadow first; rate limit and label
Irreversible effect	Publish release or rotate key	No online exploration; require proof and approval

The policy can explore candidate plans without executing all of them. Use world models, static analysis, dry runs, branch sandboxes, or a learned critic to narrow the set. These mechanisms introduce model error, so final promotion still depends on executable evidence.

Record the action set the policy was allowed to choose. A trajectory where deployment was absent cannot establish that the policy learned not to deploy. It establishes only that the runtime prevented it.

Reward versions create different tasks

Changing a reward weight changes the optimization target. Comparing policies trained under reward.v4 and reward.v5 as if they played the same game can hide the reason behavior moved.

Version each component:

terminal verifier
reward feature extractor
reward weights
constraint thresholds
discount and horizon
task sampler
environment image
action schema

Replay a stable trajectory set through a new reward version before training. Inspect which episodes change rank and why. A new deadline penalty may correctly punish slow success, or it may favor premature termination on hard but valuable tasks.

Track reward-to-reality drift after deployment. For example, an automated “reviewability” judge may initially correlate with maintainer acceptance, then drift as repositories change style or policies learn judge-specific phrasing. Compare reward with delayed outcomes such as merge, rollback, rework, incident, and human override.

If the correlation falls, do not simply train harder on the stale reward. Recalibrate the sensor and preserve a holdout that the policy and reward model did not tune against.

Failed episodes can be more valuable than winners

Success-only training hides boundaries. A failed trace can show exactly where evidence became insufficient, a tool contract was missing, a retry repeated state, or a verifier rejected a shortcut.

Classify failure transitions:

bad observation selection
unsupported inference
invalid action
policy denial
tool/environment fault
unreconciled timeout
verifier false reject
verifier true reject
budget or deadline stop
human escalation

Use successful and failed trajectories differently. Imitation can copy clean successful action sequences. Preference learning can compare admissible alternatives. RL can explore decisions whose consequences are faithfully simulated. Environment bugs should repair the environment, not become negative examples that teach the policy to avoid a valid task.

An episode that ends in safe escalation may deserve more reward than an unaudited apparent success. Reward the operating decision, not just task completion.

Keep training and production close, but never identical in authority

Training in a toy environment creates policies that exploit toy regularities. Training against production creates unacceptable mutation risk and unstable rewards.

Use layered parity:

• same tool schemas, different credentials;
• same state machine, isolated tenants;
• same verifier code, hidden scoring data;
• production-shaped latency and failures, bounded replicas;
• recorded production observations, simulated effects;
• canary deployment with reversible actions and strict budgets.

The policy should not know whether it is in evaluation from a special prompt or tool name. The runtime should always know and enforce the authority difference.

Where Agent Lightning fits

Agent Lightning proposes separating agent execution from RL training and assigning reward to individual model calls within multi-step trajectories. It is relevant because existing agents often mix model calls with tools, logic, and multiple turns that ordinary single-response RL pipelines do not represent.

Treat framework claims as first-party evidence about the proposed design and reported experiments. The durable engineering requirement remains independent: the trace must faithfully connect each trainable decision to the state, eligible actions, later evidence, and terminal outcome.

The next article examines that hard problem directly: which earlier decision deserves credit when a long workflow finally succeeds?

Promotion gates

Do not deploy a trained policy because training return increased. Require:

1. held-out task success and cost improve;
2. no regression on protected capabilities;
3. reward components correlate with independent verifier evidence;
4. shortcut and tamper tests fail closed;
5. trajectory length, retries, and tool use remain bounded;
6. shadow production improves decisions on representative slices;
7. canary effects are reversible and independently verified;
8. rollback can restore the prior policy and workflow version;
9. post-deployment drift and exception labor are measured.

Builder playbook

1. Write terminal states and evidence before rewards.
2. Separate hard constraints from optimizable cost and value.
3. Make actions typed, authorized, idempotent, and receipted.
4. Put verifiers outside the policy's mutation boundary.
5. Reset every causally relevant state feature and randomize superficial details.
6. Preserve complete versioned trajectories, including eligible actions and failures.
7. Begin with terminal reward; add shaping one mechanism at a time.
8. Price inference, tools, latency, human exceptions, and unsafe exposure.
9. Build curricula from observable skills and test unseen composition.
10. Promote through holdout, shadow, canary, and rollback gates.

When the trained agent surprises you, do not ask why it ignored your intention.

Ask which behavior the environment paid for.

That answer is the bug report.

References

• Liu et al., “AgentBench: Evaluating LLMs as Agents,” ICLR 2024. Eight interactive environments and categorized agent failure modes.
• Zhou et al., “WebArena: A Realistic Web Environment for Building Autonomous Agents,” ICLR 2024. A reproducible realistic web environment for language-guided agents.
• Huang et al., “MLAgentBench,” ICML 2024. Evaluation of agents performing iterative machine-learning experimentation.
• Wang et al., “Agent Lightning: Train ANY AI Agents with Reinforcement Learning,” 2025 preprint. A framework separating agent execution from training and assigning credit within multi-step runs.
• Mnih et al., “Human-level Control through Deep Reinforcement Learning,” Nature, 2015. A foundational example of learning policies from environment interaction and reward; the domain differs sharply from language agents.
• Amodei et al., “Concrete Problems in AI Safety,” 2016 preprint. Primary framing of reward hacking, scalable oversight, safe exploration, and distribution shift problems.