Rahul Marchand

A Steerable Decision Threshold in a Planning Agent

2026-08-28

Summary

Trained AI systems increasingly behave as though pursuing goals, but our definitions and understanding of goal-directedness remain mostly behavioural. We want to know what inside a trained agent makes this goal-model predictive, and to build concepts of goal-directedness that refer to mechanisms and implementations rather than behaviour: we want to differentiate between a look-up table indexing goal-directed behaviour and a mesa-optimiser-style utility maximiser. This post is an exploratory first step towards this.

We train agents to solve mazes with two goals of different value and a step cost. The optimal policy weighs each goal’s value against the distance to reach it, and the maximum number of extra steps an agent will accept to reach the higher-valued goal is its decision threshold.

We find this threshold is a single direction in the weights. Adding that direction to the base agent’s weights by hand moves the threshold as precisely as fine-tuning on a different reward function, extrapolates beyond the range it was fitted on, and leaves the agent’s route planning intact. We find the same in a DRC (recurrent), ResNet and ViT agent trained with RL, and in a transformer trained using imitation learning.

weight edit α = 0.00⏸

Motivation

AI systems increasingly display goal-directed behaviour ranging from the simple “win the game” objective of a chess engine to the more complicated objectives of an LLM (Everitt et al., 2025).

Work on understanding goal-directed behaviour in agents has so far been largely behavioural (Dennett, 1981; Orseau et al., 2018; MacDermott et al., 2024). However, there are proven limits on what behavioural data alone can tell us about how an agent will act in new, out-of-distribution situations (Bellot et al., 2025).

Many AI safety relevant questions are of this out-of-distribution sort. Why do some models pursue goals more persistently than others? And what proxy goals form during training at all: some theory predicts that optimal policies tend to seek power (Turner et al., 2021) but whether trained agents actually internalise such instrumental goals is unknown.

Mechanistic claims about goals can only be validated in environments with a known ground-truth objective. So our starting point is to build such an environment and attempt to fully reverse-engineer the goal representation our agents learn.

Background

What does it mean for a system to be goal-directed? The classic behavioural definition (Dennett, 1981, later formalised by Orseau et al., 2018 and MacDermott et al., 2024) is that a system is goal-directed to the extent that modelling it as rationally pursuing a goal predicts its behaviour better than any comparably simple goal-free description. This is a matter of degree rather than a binary property and it is a claim about our description of the system as opposed to its internals. But when our goal-model is effective, something inside the agent is making this true. These internal mechanisms are what we aim to understand.

What work has been done in this area so far? In a maze agent known to exhibit goal misgeneralisation, Mini et al., 2023 found the goal representation in the activations and could steer the agent by intervening on it. What they found was not a single clean variable: the representation was spread across many circuits, mixing target-tracking with a simple location bias. And because the training environment left the goal underspecified, there was no quantitative ground truth to validate a goal-reading against.

Planning is the machinery by which an agent might pursue a goal; the goal is the criterion that decides which plan is worth executing. This planning side of the picture is now mapped in some detail. Bush et al., 2025 gave the first mechanistic evidence of emergent planning in a DRC agent (Guez et al., 2019) playing Sokoban, finding internal plans that causally steer its behaviour, and Taufeeque et al., 2024, 2026 traced how these plans are stored: in dedicated channels of the hidden state, readable up to ~50 steps ahead. Similar learned look-ahead has been found in a chess policy network (Jenner et al., 2024).

Finally, our method works in weight space: we fine-tune copies of an agent and study the differences this leaves in the weights, \(\theta_{\text{ft}} - \theta_{\text{base}}\). This is what the model-editing literature calls a task vector (Ilharco et al., 2023). These differences have turned out to carry surprising structure: task vectors can be added and subtracted to compose behaviours (Ilharco et al.), interpolating the weights of policies fine-tuned on different reward functions trades off smoothly between them (Ramé et al., 2023), and the difference of two contrastive fine-tunes yields a direction that steers behaviours like sycophancy further out-of-distribution than activation steering (Fierro & Roger, 2026). We return to how our results relate to this line of work in the discussion.

The Task

We build on Bush et al., 2025, keeping the DRC agent but swapping Sokoban for maze navigation with two rewarded objectives. Planning in single-goal mazes was already demonstrated in an appendix of the blog post accompanying their paper, where a DRC’s decoded route plans generalise to mazes far larger than any seen in training; our environment adds the second objective, and with this the valuation step.

A single-objective task mostly exercises planning as every candidate plan leads to the same target, and ranking them is simply picking the shortest path. With two objectives and a step cost, plans to different targets must be compared and that comparison is exactly the valuation step we set out to isolate. The underlying maze-navigation skill is unaffected.

Let \(R_A\) and \(R_B\) be the two objectives’ rewards, \(d_A\) and \(d_B\) the path lengths to each, and \(c\) the per-step cost. The optimal policy takes objective A whenever

\[R_{A} - R_{B} > c \, (d_{A} - d_{B})\]

Each maze presents the agent with a distance gap \(\Delta d = d_A - d_B\), and the optimal rule is to accept up to \(\tau^* = (R_A - R_B)/c\) extra steps to reach the higher-reward objective. \(\tau^*\) is the environment’s ground-truth decision threshold. Crucially, the agent never observes the rewards: its input contains only the maze geometry. Whatever threshold \(\hat{\tau}\) it actually uses must be internalised during training.

The mazes are \(11\times11\) perfect mazes so a unique path connects any two cells, and \(d_A\) and \(d_B\) are well-defined. The agent observes a \(4\times11\times11\) array: one channel each for the walls, its own position, and the two objectives’ positions (Figure 2a). Walls, start position and objective positions are randomised across levels. Further environment details are in Appendix A.

We train four agents on three seeds (ViT one seed): a DRC, a ResNet and a ViT with reinforcement learning, plus a prefix transformer (conditioned on the episode’s observation–action history) trained with imitation learning, specifically behaviour cloning (BC), supervised learning on an optimal solver’s state–action pairs. Hyperparameters and training details are in Appendix B.

We reproduced Bush et al.’s planning results on our new environment. We can read the agent’s plan from its activations (Figure 2b) and writing plans in also works, though less reliably than in their Sokoban setting. Nothing downstream depends on plan writing.

Finding the Decision Threshold in the Weights

From each base agent we fine-tune a grid of arms: copies of the agent trained for a further 400k steps with one goal’s value shifted. The grid covers offsets of \(0\) to \(\pm0.4\) in steps of \(0.1\) on each of the two objectives giving 2 sets of 9 arms per base agent (details on base agents in Appendix B). Note we also fine-tuned arms outside this grid but never used these to fit the axis (these are indicated by square markers in Figure 3).

Each arm leaves a weight change \(\Delta\theta_i = \theta_i - \theta_{\text{base}}\). We model these jointly as

\[\Delta\theta_i = \mathbf{d} + \alpha_i\,\mathbf{v} + \mathbf{r}_i\]

where \(\mathbf{d}\) is a drift shared by every arm (the common effect of further fine-tuning, whatever the values), \(\mathbf{v}\) is a single shared direction, the value axis, its scale calibrated so that \(\alpha_i\) is simply the arm’s value offset and \(\mathbf{r}_i\) is the residual this model we are fitting leaves unexplained. We fit \(\mathbf{d}\) and \(\mathbf{v}\) by least squares over the grid arms, minimising \(\sum_i \lVert \mathbf{r}_i \rVert^2\). When relevant the value we are trying to write to is held out from this fit.

To produce the breakdown in the table below we fit \(\mathbf{d}\) and \(\mathbf{v}\) over all 18 arms for an agent then decompose the weight change at \(\alpha = 0.3\) into its three components. The components are roughly orthogonal in weight space so their squared norms sum to approximately \(||\Delta \theta_{0.3}||^2\) and each entry in the table is reported as a percentage of that total.

We find that the \(\mathbf{v}_0\) fitted on arms trained by varying one objective and the \(\mathbf{v}_1\) fitted on arms trained by varying the other objective are approximately the same \(\mathbf{v}\); see Appendix C.

For a short study of how the proportions of \(\mathbf{d}\), \(\mathbf{v}\) and \(\mathbf{r}_i\) change as we fine-tune for longer see Appendix D.

For an analysis of where in the weights \(\mathbf{v}\) concentrates see Appendix E.

The residual error that the model we are fitting leaves unexplained is large. We suggest that both the drift and residual components have almost no effect on the model’s decision threshold. To test this we write using our \(\mathbf{v}\) in the next section.

Steering the Decision Threshold

When writing a value offset of \(\alpha\) into the weights we hold out that value from the grid of fine-tunes used to fit \(\mathbf{d}\) and \(\mathbf{v}\). We then add the scaled axis to the frozen base weights: \(\theta_{\text{base}} + \alpha\,\mathbf{v}\) (no drift component included).

We read an agent’s learnt decision threshold, \(\hat{\tau}\), from its behaviour: we roll it out on held-out mazes, record for each episode how many steps farther the \(1.0\) goal was and which goal it chose, and fit a logistic curve to the fraction choosing the \(1.0\) goal against that gap. \(\hat{\tau}\) is the gap at which the agent is indifferent between the goals, the curve’s 0.5 crossing. Details are in Appendix F.

Figure 3 compares these estimated values of \(\hat{\tau}\) across the different fine-tunes (filled circles) and the axis written agents (hollow circles). The optimal \(\tau^*\) is designated by the grey dashed line.

We also test some fine-tunes and axis writes further out of distribution. These are not included in the fitting of \(\mathbf{v}\) and these fine-tunes are shown as filled in squares. They span from \(-0.7\leq\alpha \leq-0.5\). We do not test \(\alpha > 0.4\) as the size of the maze makes \(\hat{\tau}\) estimates impossible.

The write reproduces the held-out arm’s threshold, as seen in Figure 3: to 0.4 [0.3, 0.5] steps in the DRC, 0.5 [0.2, 0.8] in the ResNet, 0.2 in the ViT and 0.9 [0.7, 1.1] in the BC transformer. A minority component of the weight change carries the majority of the behavioural effect.

Writing the axis into the weights also doesn’t hurt the agent’s maze navigation capability. Across the written range the DRC still reaches a goal in 100% of held-out mazes at every dose, within 0.03 steps of the shortest path on average.

Outside the fitted window the axis extrapolates well until attempting to write \(\tau < 0\). Fine-tuning is mostly still able to retarget the agent to this reward function but fitting an axis to these new fine-tunes fails even in the same \(\tau < 0\) regime and the axis fit on fine-tunes with \(\tau > 0\) fails to write in this region. Appendix G shows the full picture.

Notably the agent’s learnt exchange rates are not perfect. The task-optimal slope is \(\pm 20\) steps per unit of value; the DRC moves \(14\), the ResNet \(16\), the ViT \(10\), and the BC transformer \(22\). The RL agents are trained undiscounted (\(\gamma = 1\)) so would receive maximum reward for the task-optimal trade-off.

Discussion

Why is the threshold a single direction in the weights?

Linear directions in weight space that control a behaviour are not new. Task vectors add and subtract (Ilharco et al., 2023), the difference of two contrastive fine-tunes steers an LLM (Fierro & Roger, 2026), and interpolating between policies fine-tuned on different rewards trades off smoothly between them, even for a control policy trained from scratch (Ramé et al., 2023). There are a couple of existing candidate explanations for this.

The first describes fine-tuning as a small perturbation of the base weights. Fine-tunes starting from the same trained weights stay in one basin, close in parameter space (Neyshabur et al., 2020), and fine-tuning is often well described by linearising the network around its starting weights (Malladi et al., 2023). Our base training is the shared initialisation, the fine-tunes are short, and the change we ask for is one-dimensional, so a line between value-shifted fine-tunes is expected on these grounds. The return is linear in the reward, so the fine-tuning gradient is affine in the value shift, and to first order the weight change is linear in that gradient (Koh & Liang, 2017; Daheim et al., 2024). That is exactly the form we fit, a drift shared by every arm plus one direction scaled by the shift.

But this explanation is limited: it predicts a line near the base, and says nothing about how far the line holds, how it fails, or which part of the weight change matters. We have three observations that this leaves unexplained:

• The line holds across the whole measurable range of \(\tau\), from the largest the mazes allow down to \(\tau\) near zero (Figure 3). A local expansion has no reason to hold this far.
• The write collapses at \(\tau<0\), and no axis can be fitted past that point (Appendix G). The failure is sharp and sits at a meaningful point, parity, while a first-order account would predict gradual degradation with distance from the base.
• \(\mathbf{v}\) is 14–30% of each arm’s weight change (Table 1), yet written alone, with drift and residual left out, it moves \(\tau\) across this whole range and leaves navigation untouched.

Second, in language models, one hypothesis is that the steered property varies across the pretraining corpus, so the model learns a latent variable for it. The weights fix a prior over this variable and each prompt is evidence the model conditions on to infer its value (Xie et al., 2021), with fine-tuning, in this view, then shifting the prior. Persona-vector work is loosely in this spirit, with fine-tuning shifting a model along the activation direction that prompting elicits (Chen et al., 2025). Our goal values never vary and are never observed, so there is no such variable to fix a prior over, and this hypothesis cannot explain our results.

We suggest that \(\mathbf{v}\) exists because it was efficient to encode this part of the agent’s computation in this legible way. Solving the task requires three things: a planned route to each goal, a score for each route, and a comparison of the two scores. Scoring a route requires a value for the goal it reaches. The agent never observes the goal values, so it must store them, and \(\mathbf{v}\) is where this is stored. Though note that we cannot tell whether the constant is a value difference, a step cost or the entire threshold itself. This explanation could account for the three observations above. The line holds across the whole range because we are directly varying a parameter in the model’s computation. It fails at parity because the input’s goal channels mean “the richer goal” and “the poorer goal” throughout training and a setting that reverses them contradicts what the circuit was built to read. And navigation survives the write because what changes is how the goals are valued, not how routes to them are found. What is notable is not that this constant exists at all, since any circuit that scores by value has one somewhere, but that it sits in the weights as a single legible direction despite never varying during training.

Evidence for modular structure

So far we understand two properties of the weights. We reproduced the evidence Bush et al., 2025 and the follow-up work (Taufeeque et al., 2026) found showing that these agents form detailed plans (Figure 2b). We have also shown that a single direction in its weights changes which objective the agent chooses without affecting the agent’s planning capabilities (Figure 3). Whatever computes the route and whatever sets the choice can be moved independently: the computation has at least two separable parts.

We have the planning circuits which produce a route to each goal and the valuation step that scores each route. Moving along \(\mathbf{v}\) sets the constant that this trade-off turns on.

Note that the present evidence is consistent with this picture rather than conclusively proving it. Some parts of the internal mechanism are not yet understood. For example separability alone does not fix the order of the stages. In fact it seems likely that in some of the architectures (DRC, ResNet) the axis is acting during the planning stage and that even though valuation can be intervened on separately from planning these both occur in the same intertwined stage.

One further detail that constrains the space of possible mechanisms that would be consistent with our observations is that the mechanism breaks past \(\tau < 0\), so the comparison is built around which goal is the richer one rather than being symmetric.

On a separate note the BC transformer was trained to predict the next action of an optimal solver, and it developed the same threshold parameter as the RL agents. In fact it was the cleanest of the four to intervene on.

We note an interesting parallel to Dennett’s definition of goal-directedness introduced in Section 2. We said a system is goal-directed to the extent that a goal-model is a compact predictor of its behaviour. In a parallel way behaviour cloning essentially directly optimises for and builds a compact predictor of behaviour. Predictors internalising the latent variables of whatever generated their data has precedent, a board state from move sequences (Li et al., 2023), other agents’ preferences and beliefs from their trajectories (Rabinowitz et al., 2018), and, it has been argued, the intentions of authors from their text (Andreas, 2022). Humans appear to infer goals the same way, by inverting a planner (Baker et al., 2009). We state the general claim as a hypothesis: the goal-and-planner decomposition is the short description of goal-directed behaviour, optimisation finds short descriptions, so any system trained to produce or predict goal-directed behaviour should tend to build a form of goal circuit, and the better a goal-model compresses the behaviour, the cleaner the circuit.

Conclusion

We set out to find the goal representation of an agent. One component of it, the decision threshold, turns out to be a single direction in the weights: fitted from a grid of fine-tunes, it moves the threshold as precisely as retraining, extrapolates past the fitted range, leaves navigation untouched, and appears in three RL architectures and in a transformer trained to imitate an optimal solver. The parameter is separable from the planning it steers.

We aim to understand how far trained agents decompose into the recognisable components of planning, valuation and choice, with goals as parameters of the valuation step, in the way hand-built agents once did by design (Russell & Norvig, 2020, §2.4.5).

Potential next steps

Build a clearer picture of the mechanisms implemented in the agent’s weights. See if we can observe each stage of the hypothesised planning-goal-action hierarchy directly.

Make the valuation step more complicated. More goals, nonlinear values, goals nested inside plans. How does the representation of goals evolve?

Formalise and quantify what we observe. A theoretical account of why we would expect to observe this parameter in the weights would be a natural complement to the empirical work here. Additionally, we are interested in quantifying the extent to which we can retarget an agent towards new goals.

Acknowledgements

Matthew Farrugia-Roberts supervised this work throughout and reviewed a draft of this post. The research was funded by the Witt Lab. Thanks also to Tom Bush for feedback along the way. This is work in progress, and any mistakes are my own.

Bibliography

• Andreas, J. (2022). Language Models as Agent Models. Findings of EMNLP 2022. arXiv:2212.01681
• Baker, C. L., Saxe, R., & Tenenbaum, J. B. (2009). Action Understanding as Inverse Planning. Cognition, 113(3), 329–349. doi:10.1016/j.cognition.2009.07.005
• Bellot, A., Richens, J., & Everitt, T. (2025). The Limits of Predicting Agents from Behaviour. ICML 2025. arXiv:2506.02923
• Bush, T., Chung, S., Anwar, U., Garriga-Alonso, A., & Krueger, D. (2025). Interpreting Emergent Planning in Model-Free Reinforcement Learning. ICLR 2025 (oral). arXiv:2504.01871
• Chen, R., Arditi, A., Sleight, H., Evans, O., & Lindsey, J. (2025). Persona Vectors: Monitoring and Controlling Character Traits in Language Models. arXiv:2507.21509
• Daheim, N., Möllenhoff, T., Ponti, E. M., Gurevych, I., & Khan, M. E. (2024). Model Merging by Uncertainty-Based Gradient Matching. ICLR 2024. arXiv:2310.12808
• Dennett, D. C. (1981). True Believers: The Intentional Strategy and Why It Works. In A. F. Heath (Ed.), Scientific Explanation. Oxford University Press. Reprinted in The Intentional Stance (MIT Press, 1987). PDF
• Everitt, T., Garbacea, C., Bellot, A., Richens, J., Papadatos, H., Campos, S., & Shah, R. (2025). Evaluating the Goal-Directedness of Large Language Models. arXiv:2504.11844
• Fierro, C., & Roger, F. (2026). Steering Language Models with Weight Arithmetic. ICLR 2026. arXiv:2511.05408
• Guez, A., Mirza, M., Gregor, K., et al. (2019). An Investigation of Model-Free Planning. ICML 2019. arXiv:1901.03559
• Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., & Farhadi, A. (2023). Editing Models with Task Arithmetic. ICLR 2023. arXiv:2212.04089
• Jenner, E., Kapur, S., Georgiev, V., Allen, C., Emmons, S., & Russell, S. (2024). Evidence of Learned Look-Ahead in a Chess-Playing Neural Network. NeurIPS 2024. arXiv:2406.00877
• Koh, P. W., & Liang, P. (2017). Understanding Black-box Predictions via Influence Functions. ICML 2017. arXiv:1703.04730
• Langosco, L., Koch, J., Sharkey, L., Pfau, J., Orseau, L., & Krueger, D. (2022). Goal Misgeneralization in Deep Reinforcement Learning. ICML 2022. arXiv:2105.14111
• Li, K., Hopkins, A. K., Bau, D., Viégas, F., Pfister, H., & Wattenberg, M. (2023). Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task. ICLR 2023. arXiv:2210.13382
• MacDermott, M., Fox, J., Belardinelli, F., & Everitt, T. (2024). Measuring Goal-Directedness. NeurIPS 2024 (spotlight). arXiv:2412.04758
• Malladi, S., Wettig, A., Yu, D., Chen, D., & Arora, S. (2023). A Kernel-Based View of Language Model Fine-Tuning. ICML 2023. arXiv:2210.05643
• Mini, U., Grietzer, P., Sharma, M., Meek, A., MacDiarmid, M., & Turner, A. M. (2023). Understanding and Controlling a Maze-Solving Policy Network. arXiv:2310.08043
• Neyshabur, B., Sedghi, H., & Zhang, C. (2020). What is Being Transferred in Transfer Learning? NeurIPS 2020. arXiv:2008.11687
• Orseau, L., McGill, S., & Legg, S. (2018). Agents and Devices: A Relative Definition of Agency. arXiv:1805.12387
• Rabinowitz, N. C., Perbet, F., Song, H. F., Zhang, C., Eslami, S. M. A., & Botvinick, M. (2018). Machine Theory of Mind. ICML 2018. arXiv:1802.07740
• Ramé, A., Couairon, G., Shukor, M., Dancette, C., Gaya, J.-B., Soulier, L., & Cord, M. (2023). Rewarded Soups: Towards Pareto-Optimal Alignment by Interpolating Weights Fine-Tuned on Diverse Rewards. NeurIPS 2023. arXiv:2306.04488
• Russell, S., & Norvig, P. (2020). Artificial Intelligence: A Modern Approach (4th ed.), §2.4.5 Utility-based agents. Pearson. aima.cs.berkeley.edu
• Taufeeque, M., Quirke, P., Li, M., Cundy, C., Tucker, A. D., Gleave, A., & Garriga-Alonso, A. (2024). Planning in a Recurrent Neural Network that Plays Sokoban. ICML 2024 Mechanistic Interpretability Workshop. arXiv:2407.15421
• Taufeeque, M., Tucker, A. D., Gleave, A., & Garriga-Alonso, A. (2026). Path Channels and Plan Extension Kernels: A Mechanistic Description of Planning in a Sokoban RNN. ICLR 2026. arXiv:2506.10138
• Turner, A. M., Smith, L., Shah, R., Critch, A., & Tadepalli, P. (2021). Optimal Policies Tend to Seek Power. NeurIPS 2021 (spotlight). arXiv:1912.01683
• Xie, S. M., Raghunathan, A., Liang, P., & Ma, T. (2021). An Explanation of In-context Learning as Implicit Bayesian Inference. ICLR 2022. arXiv:2111.02080

Appendices

Appendix A: Environment details

Actions

Four moves (up, down, left, right). A move into a wall leaves the agent in place and still costs a step.

Reward

Every step costs 0.05, including the step that reaches a goal. Reaching either goal ends the episode and adds that goal’s value. Episodes that reach neither goal are cut off after 120 steps.

Generation

Mazes are generated by recursive backtracking, which produces perfect mazes: fully connected, with a unique path between any two cells. Levels are rejection sampled so that each goal is reachable without passing through the other (roughly half of draws are rejected), so every level poses a genuine choice between the two.

Levels and splits

A single pre-generated pool of one million \(11\times11\) mazes. The train/evaluation split is by wall layout, so no layout seen during training appears in any measurement; every threshold in the post is decoded on held-out layouts.

Appendix B: Agent configurations and hyperparameters

Threshold is the measured indifference point in extra steps, one value per independent base model, on held-out levels at the training values. The task optimum is 10.

Agent	Architecture	Params	Budget	Training	Threshold
DRC	ConvLSTM(3,3), 3 layers × 3 ticks	1,449,605	150M	RL	6.9 / 8.0 / 5.4
ResNet	9 conv stages × 32 ch, rescaled input	1,234,405	150M	RL	7.6 / 7.8 / 8.6
ViT	transformer, 4 layers, d 64, 4 heads	2,192,133	250M	RL	5.2
BC transformer	prefix-LM, 4 layers, d 128, 4 heads	806,533	30k updates	imitation	10.8 / 10.9 / 10.6

Reinforcement learning (DRC, ResNet, ViT)

Algorithm

IMPALA with V-trace (\(\lambda = 0.97\)); 256 parallel environments, 20-step rollouts, 8 minibatches per update. \(\gamma = 1\), so the trained objective is exactly the value-minus-step-cost quantity every measurement uses.

Optimiser

Adam, learning rate annealed \(4\times10^{-4} \rightarrow 4\times10^{-6}\); gradient-norm clip 0.015; entropy weight 0.01, value-loss weight 0.25. The ViT instead uses a peak rate of \(1\times10^{-3}\) with floor \(1\times10^{-4}\), chosen by a learning-rate sweep.

Input encoding

The DRC and ResNet consume the \(4\times11\times11\) observation directly as image planes. The ViT embeds each cell’s four channels as one token (121 tokens), adds a learned two-dimensional position embedding, and applies pre-LayerNorm attention blocks. All three flatten their final spatial hidden state through the same dense readout to the actor and critic heads, so the architectures differ only in the trunk.

Behaviour cloning (BC transformer)

Demonstrations

One per training level: the optimal route under the task’s utility, computed by the exact undiscounted solver, roughly 900k routes over the train split.

Tokenisation

The input sequence is the 121 maze cells in raster order, a separator, then the action sequence. Each cell token is a linear map of that cell’s observation channels plus learned row and column embeddings; actions carry learned position embeddings. Attention is bidirectional over the cells and causal over the actions (a prefix LM), and the loss is taken on the action positions only.

Optimiser

30k updates at batch 256; AdamW at \(3\times10^{-4}\) with 500 warmup steps and cosine decay to zero; weight decay 0.01; gradient clip 1.0.

Seed selection

A seed counts as converged when its greedy policy finishes held-out episodes reliably: 95% for the DRC and ResNet, 90% for the ViT, whose plateau sits lower. The DRC kept 3 of 5 seeds (two still improving at 150M), the ResNet 3 of 6 (three diverged early, with value loss above \(10^3\)), the ViT 1 of 3 (two were competent when sampling but looped under the greedy action selection used for measurement, and were retrained at the tuned rate), and the BC transformer 3 of 3. Two ResNets converged and then collapsed late in training; for those the base is the last checkpoint before the collapse (120M and 90M of 150M). Every other base is its final checkpoint.

Fine-tuning arms

Per arm

400k steps at a constant \(10^{-4}\) (annealing would put each arm at a different point on its own schedule), starting from the base weights with a fresh optimiser, on a 150k-level dataset generated at that arm’s values.

Past parity

Arms that take the swept goal’s value beyond the other’s use levels whose colours are pinned to objectives. Under the usual scheme the generator repaints whichever goal is richer, so such an arm would learn a relabelled preference instead of a reversed one.

Measurement

Every arm and every written model is decoded at the base values on held-out layouts, so all thresholds are comparable regardless of what an arm trained on.

Compute

Everything was trained on single RTX 4090s: a base agent is 4–5.5 hours, and a full 18-arm value sweep with its measurements about 1.5 hours per base.

Appendix C: Do the two sweeps find the same \(\mathbf{v}\)?

These cosines compare \(\mathbf{v}_0\), fitted only on the arms sweeping one objective’s value, against \(\mathbf{v}_1\), fitted only on the other’s.

There are two issues with using a raw cosine.

First, it is attenuated by each fit’s own fine-tuning noise, so a noisy pair of fits hides a shared axis. We correct for this with each sweep’s reliability \(\rho\): splitting the sweep’s mirrored arm pairs in half and fitting an axis on each half gives the correlation between two independent estimates of the same direction, corrected to full length by Spearman-Brown since each half fits on half the arms. If the two sweeps measure one shared axis, the expected magnitude of their cosine is then \(\sqrt{\rho_0 \rho_1}\), the attenuation the two fits’ noise alone would produce, so the cosine is judged against this benchmark rather than against 1.

Second, fine-tuning concentrates its weight changes in the same few directions whatever the values trained toward. To control for this we fit axes on arms trained with shuffled value labels. The Null column in the table below is the 5th percentile of the cosine under this shuffled-label null.

Whether the two sweeps are adjusting the weights along the same axis. Ranges are min to max over independent base agents. Each sweep’s reliability \(\rho\) is its split-half correlation, Spearman-Brown corrected; \(\sqrt{\rho_0 \rho_1}\) is the expected magnitude of the cosine if both sweeps measure one shared axis; the ratio uses the cosine’s magnitude. The null 5th percentile is the 5% tail of the cosine’s distribution when the arms’ value labels are shuffled within both sweeps; against that null the observed cosine has \(p < 0.001\) for every base (2,000 resamples).

	\(\cos(\mathbf{v}_0, \mathbf{v}_1)\)	\(\sqrt{\rho_0 \rho_1}\)	\(\lvert\cos\rvert / \sqrt{\rho_0 \rho_1}\)	Null 5th percentile
DRC ×3	−0.75 to −0.61	0.60 to 0.74	1.01 to 1.02	−0.63 to −0.48
ResNet ×3	−0.38 to −0.12	0.12 to 0.38	0.95 to 0.98	−0.27 to −0.08
ViT ×1	−0.23	0.27	0.85	−0.16
BC ×3	−0.98	0.97	1.01	−0.87 to −0.85

A third check uses one sweep’s axis as a measuring instrument on the other’s fine-tunes. Each arm of a sweep is a weight change trained at a known value offset, and projecting it onto the other sweep’s axis gives a one-number reading of how far the fine-tune moved along that axis. Across all ten bases these readings match the trained offsets with correlation −0.97 to −1.00: an axis fitted from sweeping one objective reads off, almost exactly, how much the other objective’s value was changed. The readings are scaled down by a factor equal to the measured cosine, which is precisely the damping the two fits’ noise predicts.

Appendix D: How does length of fine-tune affect results?

Every arm in the post is fine-tuned for 400k steps. Here we extend the DRC’s fine-tunes to 3M steps, seven times that budget, and read them at checkpoints along the way, refitting the decomposition of Table 1 and the axis at each one. Figure 4 reads three quantities down this ladder.

Panel (a) tracks where the weight change goes. The drift grows steadily, from roughly a quarter of the change at the standard budget to two fifths at 3M steps: it is the accumulated effect of any further training, whatever the values, so it can only grow, and the write and residual shares shrink in proportion.

Panel (b) tracks the exchange rate the arms learn. It climbs from about 9 steps per unit of value at 100k to 14 at the standard budget and plateaus near 16 from 1M steps on, still short of the task’s optimal 20. Some of the undershoot in Figure 3 is unfinished learning; some of it may be a stable gap.

Panel (c) tracks two arms trained past parity, both asked for \(\tau^* = -4\). They still sit near \(\hat{\tau} = +1.5\) at the standard budget, cross zero only after about 750k steps, and reach \(-3\) by 3M. The written points ask whether a better axis would follow. At each length the axis is refitted from grid arms fine-tuned equally long, then written into the frozen base at the dose the reversal arms trained on, so a longer budget gets every advantage it could confer on the fit. The written threshold creeps from \(+3\) toward \(+1\) and never crosses zero. A reversal is learnable given time; no fitting budget yields an axis that can express it (Appendix G).

Appendix E: Locating \(\mathbf{v}\) in the weights

A fitted axis is mostly fine-tuning noise. We therefore use both sweeps. The axis fitted from sweeping one objective’s value and the axis fitted from sweeping the other’s come from disjoint fine-tunes, so their noise is independent, while both are assumed to represent the same direction in the weights with opposite sign; Appendix C tests that assumption. Because the two fits’ noise is independent, it is attenuated in their inner product while the shared axis accumulates. Splitting that inner product tensor by tensor therefore measures how much of the shared axis each tensor holds; each tensor’s fraction of the total is its share of the axis (Figure 5).

For every tensor we also write only its slice of the axis into the base agent and decode the threshold on held-out levels, at the same \(\alpha = \pm 0.3\) as the full write. The shift it produces, as a fraction of the full write’s shift, is the red bar. The fractions sum to between 85% and 102% across the four architectures, so the effect of the full write is close to the sum of its tensors’ effects, and writing everything except a tensor loses what writing it alone gives.

The two shares disagree in consistent ways. In every architecture the flatten layer before the heads holds a large slice of the axis and none of its effect. In the DRC it is 68% of the parameters, 12% of the axis and 0% of the effect; in the ViT, 59% of the axis and 3% of the effect.

Where the effect does live differs by architecture. In the DRC it sits at the input. Half of the axis’s effect sits in 7% of the weights, and in every base the densest tensor is the 1,152-weight input convolution, 0.08% of the network holding 28 to 45% of the axis and 41% of the effect. Its slice alone moves the threshold by 1.5 of the full write’s 3.8 steps; the rest comes from conv 1 and the cells’ input convolutions, with the recurrent convolutions carrying almost nothing. Inside the input convolution the axis lies entirely on the two colour planes, amplifying one colour’s kernel and attenuating the other’s, so at its purest the axis appears to be a gain on how “loudly” each colour is seen. The ResNet repeats the pattern one layer deeper, in the input convolution and stages 0 to 3, with nothing from stage 4 on. The ViT’s full write is small, 1.8 steps, and its effect sits in block 0, well away from the flatten layer where its length is. The BC transformer is the diffuse case and its axis shares track the parameter shares, and the effect is carried by the MLPs of blocks 0 to 2.

A tensor’s slice is linearly controllable. Writing it at half, one, two, four and eight times its own size moves the threshold in proportion, in every tensor that moves it at all (Figure 6).

Appendix F: Fitting \(\hat{\tau}\)

Every threshold in the post is read from behaviour the same way. We roll out the agent on held-out levels and record, per episode, the distance gap (how many steps farther the 1.0 goal is than the 0.5 goal) and which goal it reached. Binning episodes by gap gives the fraction that took the 1.0 goal; an unbinned maximum-likelihood logistic fit, \(P(\text{took 1.0}) = \sigma\!\left((\hat{\tau} - \text{gap})/s\right)\), then yields the threshold \(\hat{\tau}\), the gap at which the agent is indifferent between the goals (the curve’s 0.5-crossing), and the scale \(s\), the width of the indifference region: the agent’s decision noise.

Figure 7 shows these curves for each base agent and for the same frozen agent with the axis written at \(\alpha = \pm 0.2\). The write slides the whole curve sideways rather than flattening it: \(\hat{\tau}\) moves \(7.1 \rightarrow 9.9\,/\,5.0\) steps in the DRC, \(7.6 \rightarrow 10.2\,/\,5.5\) in the ResNet, \(5.2 \rightarrow 6.5\,/\,4.2\) in the ViT and \(11.1 \rightarrow 16.6\,/\,7.1\) in the BC transformer. The scale \(s\) stretches roughly in proportion to the threshold as the write moves it, in all four architectures.

Appendix G: \(\mathbf{v}\) below \(\tau = 0\)

The fitted grid stops at offsets of \(\pm0.4\). Here we carry both sweeps far beyond it, to values that push the optimal threshold \(\tau^*\) through zero: past parity the swept goal is worth more than the other, and the optimal policy reverses its preference. None of these arms enter any axis fit.

Fine-tuning works past \(\tau = 0\) in the ResNet, the ViT and the BC transformer. The DRC at the standard 400k steps is the exception. Its arms stall just above zero, near \(\hat{\tau} = +2\), and it reverses only under longer fine-tuning, crossing after about 750k steps (Figure 4c).

The write does not work past \(\tau = 0\). Approaching it, the written threshold flattens away from the fine-tuned line, and pushed further the written agent stops finishing episodes at all.

The failure is not a matter of distance from the base weights. Fitting a fresh axis to the far arms themselves fails even within the \(\tau < 0\) regime that those arms occupy.