Our agents solve mazes with two rewarded objectives and a step cost. The optimal solution involves weighing up the objectives’ rewards against the distances required to reach them. The maximum number of extra steps that the agent will accept to reach the higher valued objective is the decision threshold.
We found that this threshold is linearly encoded in a subset of the model weights. Adjusting the weights by hand moves the agent’s decision boundary as precisely and reliably as fine-tuning with a different reward function. This result was reproduced across DRC (a recurrent planning architecture), ResNet and ViT agents trained with RL, and in a transformer trained using imitation learning.
The agent receives terminal reward equal to the reached goal’s value, minus 0.05 per step taken. Each open cell is coloured by the goal the agent reaches when started from that cell: the 1.0 goal or the 0.5 goal; grey cells reach neither before timeout. We find a single direction v in the agent’s weights that linearly encodes the decision threshold: moving the weights a distance α along v shifts the decision boundary in every maze as if the 0.5 goal’s value were 0.5 + α, while trajectories remain shortest-path optimal. The dashed line marks the reward-optimal decision boundary. Hovering over a cell shows the agent’s trajectory (solid) and, where it differs, the reward-optimal trajectory (dashed).
AI systems increasingly display goal-directed behaviour ranging from the simple “win the game” objective of a chess engine to the more complicated objectives of an LLM (Everitt et al., 2025).
Work on understanding of goal-directed behaviour in agents has so far been largely behavioural (Dennett, 1981; Orseau et al., 2018; MacDermott et al., 2024). However, there are proven limits on what behavioural data alone can tell us about how an agent will act in new, out-of-distribution situations (Bellot et al., 2025).
Many AI safety relevant questions are of this out-of-distribution sort. Why do some models pursue goals more persistently than others? And what proxy goals form during training at all: some theory predicts that optimal policies tend to seek power (Turner et al., 2021) but whether trained agents actually internalise such instrumental goals is unknown.
Mechanistic claims about goals can only be validated in environments with a known ground-truth objective. So our starting point is to build such an environment and attempt to fully reverse-engineer the goal representation our agents learn.
What does it mean for a system to be goal-directed? The classic behavioural definition (Dennett, 1981, later formalised by Orseau et al., 2018 and MacDermott et al., 2024) is that a system is goal-directed to the extent that modelling it as rationally pursuing a goal predicts its behaviour better than any comparably simple goal-free description. This is a matter of degree rather than a binary property and it is a claim about our description of the system as opposed to its internals. But when our goal-model is effective, something inside the agent is making this true. These internal mechanisms are what we aim to understand.
What work has been done in this area so far? In a maze agent known to exhibit goal misgeneralisation, Mini et al., 2023 found the goal representation in the activations and could steer the agent by intervening on it. What they found was not a single clean variable: the representation was spread across many circuits, mixing target-tracking with a simple location bias. And because the training environment left the goal underspecified, there was no quantitative ground truth to validate a goal-reading against.
Planning is the machinery by which an agent might pursue a goal; the goal is the criterion that decides which plan is worth executing. This planning side of the picture is now mapped in some detail. Bush et al., 2025 gave the first mechanistic evidence of emergent planning in a DRC agent (Guez et al., 2019) playing Sokoban, finding internal plans that causally steer its behaviour, and Taufeeque et al., 2024, 2026 traced how these plans are stored: in dedicated channels of the hidden state, readable up to ~50 steps ahead. Similar learned look-ahead has been found in a chess policy network (Jenner et al., 2024).
Finally, our method works in weight space: we fine-tune copies of an agent and study the differences this leaves in the weights, \(\theta_{\text{ft}} - \theta_{\text{base}}\). This is what the model-editing literature calls a task vector (Ilharco et al., 2023). These differences have turned out to carry surprising structure: task vectors can be added and subtracted to compose behaviours (Ilharco et al.), interpolating the weights of policies fine-tuned on different reward functions trades off smoothly between them (Ramé et al., 2023), and the difference of two contrastive fine-tunes yields a direction that steers behaviours like sycophancy further out-of-distribution than activation steering (Fierro & Roger, 2026). We return to how our results relate to this line of work in the discussion.
We build on Bush et al., 2025, keeping the DRC agent but swapping Sokoban for maze navigation with two rewarded objectives. Planning in single-goal mazes was already demonstrated in an appendix of the blog post accompanying their paper, where a DRC’s decoded route plans generalise to mazes far larger than any seen in training; our environment adds the second objective, and with this the valuation step.
A single-objective task mostly exercises planning as every candidate plan leads to the same target, and ranking them is simply picking the shortest path. With two objectives and a step cost, plans to different targets must be compared and that comparison is exactly the valuation step we set out to isolate. The underlying maze-navigation skill is unaffected.
Let \(R_A\) and \(R_B\) be the two objectives’ rewards, \(d_A\) and \(d_B\) the path lengths to each, and \(c\) the per-step cost. The optimal policy takes objective A whenever
\[R_{A} - R_{B} > c \, (d_{A} - d_{B})\]
Each maze presents the agent with a distance gap \(\Delta d = d_A - d_B\), and the optimal rule is to accept up to \(\tau^* = (R_A - R_B)/c\) extra steps to reach the higher-reward objective. \(\tau^*\) is the environment’s ground-truth decision threshold. Crucially, the agent never observes the rewards: its input contains only the maze geometry. Whatever threshold \(\hat{\tau}\) it actually uses must be internalised during training.
The mazes are \(11\times11\) perfect mazes so a unique path connects any two cells, and \(d_A\) and \(d_B\) are well-defined. The agent observes a \(4\times11\times11\) array: one channel each for the walls, its own position, and the two objectives’ positions (Figure 2a). Walls, start position and objective positions are randomised across levels. Further environment details are in Appendix A.
We train four agents on three seeds (ViT one seed): a DRC, a ResNet and a ViT with reinforcement learning, plus a prefix transformer (conditioned on the episode’s observation–action history) trained with imitation learning, specifically behaviour cloning (BC), supervised learning on an optimal solver’s state–action pairs. Hyperparameters and training details are in Appendix B.
(a) The agent’s observation: four binary 11×11 planes encoding walls, the agent’s position, and one plane per goal colour. Goal values do not appear in the input. (b) Four held-out mazes. A linear probe applied to the DRC’s recurrent state before its first action, here after four extra recurrence passes over the frozen observation, scores each cell on whether the coming route will pass through it (purple); the overlaid line is the route the agent then took. Beneath each maze, each goal’s utility (value − 0.05×distance), with the agent’s choice in bold. Across held-out mazes the probe reads the route at 0.95 AUC [0.93, 0.97] (0.92 with no extra passes), against 0.51 for the same probe on an untrained network and 0.59 reading the raw observation. Probing follows Bush et al., adapted to route cells.
We reproduced Bush et al.’s planning results on our new environment. We can read the agent’s plan from its activations (Figure 2b) and writing plans in also works, though less reliably than in their Sokoban setting. Nothing downstream depends on plan writing.
From each base agent we fine-tune a grid of arms: copies of the agent trained for a further 400k steps with one goal’s value shifted. The grid covers offsets of \(0\) to \(\pm0.4\) in steps of \(0.1\) on each of the two objectives giving 2 sets of 9 arms per base agent (details on base agents in Appendix B). Note we also fine-tuned arms outside this grid but never used these to fit the axis (these are indicated by square markers in Figure 3).
Each arm leaves a weight change \(\Delta\theta_i = \theta_i - \theta_{\text{base}}\). We model these jointly as
\[\Delta\theta_i = \mathbf{d} + \alpha_i\,\mathbf{v} + \mathbf{r}_i\]
where \(\mathbf{d}\) is a drift shared by every arm (the common effect of further fine-tuning, whatever the values), \(\mathbf{v}\) is a single shared direction, the value axis, its scale calibrated so that \(\alpha_i\) is simply the arm’s value offset and \(\mathbf{r}_i\) is the residual this model we are fitting leaves unexplained. We fit \(\mathbf{d}\) and \(\mathbf{v}\) by least squares over the grid arms, minimising \(\sum_i \lVert \mathbf{r}_i \rVert^2\). When relevant the value we are trying to write to is held out from this fit.
To produce the breakdown in the below table we fit \(\mathbf{d}\) and \(\mathbf{v}\) over all 18 arms for an agent then decompose the weight change at \(\alpha = 0.3\) into its three components. The components are roughly orthogonal in weight space so their squared norms sum to approximately \(||\Delta \theta_{0.3}||^2\) and each entry in the table is reported as a percentage of that total.
| Drift (\(\mathbf{d}\), %) | Write (\(0.3\,\mathbf{v}\), %) | Residual (\(\mathbf{r}_{0.3}\), %) | |
|---|---|---|---|
| DRC ×3 | 24.9 [22.3, 28.5] | 29.5 [25.7, 35.7] | 45.2 [41.7, 48.1] |
| ResNet ×3 | 21.4 [19.3, 23.1] | 18.3 [16.5, 21.3] | 62.2 [57.7, 66.4] |
| ViT ×1 | 37.3 | 14.3 | 48.0 |
| BC ×3 | 43.9 [40.9, 47.1] | 22.5 [21.8, 23.3] | 34.9 [31.6, 38.5] |
We find that the \(\mathbf{v}_0\) fitted on arms trained by varying one objective and the \(\mathbf{v}_1\) fit on arms trained by varying the other objective are approximately the same \(\mathbf{v}\); see Appendix C.
For a short study of how the proportions of \(\mathbf{d}\), \(\mathbf{v}\) and \(\mathbf{r}_i\) change as we finetune for longer see Appendix D.
For an analysis of where in the weights \(\mathbf{v}\) concentrates see Appendix E.
The residual error that the model we are fitting leaves unexplained is large. We suggest that both the drift and residual components have almost no effect on the model’s decision threshold. To test this we write using our \(\mathbf{v}\) in the next section.
When writing a value offset of \(\alpha\) into the weights we hold out that value from the grid of finetunes used to fit \(\mathbf{d}\) and \(\mathbf{v}\). We then add the scaled axis to the frozen base weights: \(\theta_{base} + \alpha\,\mathbf{v}\) (no drift component included).
How we estimate an agent’s learnt decision threshold, \(\hat{\tau}\) , from its behaviour is described in Appendix F.
Figure 3 compares these estimated values of \(\hat{\tau}\) across the different finetunes (filled circles) and the axis written agents (hollow circles). The optimal \(\tau^*\) is designated by the grey dashed line.
We also test some finetunes and axis writes further out of distribution. These are not included in the fitting of \(\mathbf{v}\) and these finetunes are shown as filled in squares. They span from \(-0.7\leq\alpha \leq-0.5\). We do not test \(\alpha > 0.4\) as the size of the maze makes \(\hat{\tau}\) estimates impossible.
The decision threshold as the \(1.0\) goal’s value is moved: (a) the DRC, (b) the ResNet, (c) the ViT, (d) the BC transformer. Solid, filled: agents fine-tuned at the shifted value (circles: arms the axis was fitted from; squares: arms never included in the fit). Dashed, open: the frozen base agent with the fitted axis written into its weights, θ + α · v. Grey dashes: the task’s optimal exchange rate. Shading marks the range across the three seeds ((c): one seed).
The write reproduces the held-out arm’s threshold, as seen in Figure 3: to 0.4 [0.3, 0.5] steps in the DRC, 0.5 [0.2, 0.8] in the ResNet, 0.2 in the ViT and 0.9 [0.7, 1.1] in the BC transformer. A minority component of the weight change carries the majority of the behavioural effect.
Writing the axis into the weights also doesn’t hurt the agent’s maze navigation capability. Across the written range the DRC still reaches a goal in 100% of held-out mazes at every dose, within 0.03 steps of the shortest path on average.
Outside the fitted window the axis extrapolates well until attempting to write \(\tau < 0\). Finetuning is mostly still able to retarget the agent to this reward function but fitting an axis to these new finetunes fails even in the same \(\tau < 0\) regime and the axis fit on finetunes with \(\tau > 0\) fails to write in this region. Appendix G shows the full picture.
Notably the agent’s learnt exchange rates are not perfect. The task-optimal slope is \(\pm 20\) steps per unit of value; the DRC moves \(14\), the ResNet \(16\), the ViT \(10\) , and the BC transformer \(22\). The RL agents are trained undiscounted (\(\gamma = 1\)) so would receive maximum reward for the task optimal tradeoff.
Linear directions in weight space that control a behaviour are not new. Task vectors add and subtract (Ilharco et al., 2023), the difference of two contrastive fine-tunes steers an LLM (Fierro & Roger, 2026), and interpolating between policies fine-tuned on different rewards trades off smoothly between them, even for a control policy trained from scratch (Ramé et al., 2023).
The standard explanations for why such directions exist are geometric. Fine-tunes that start from the same trained weights stay in one basin, close in parameter space (Neyshabur et al., 2020), and fine-tuning is often well described by linearising the network around its starting weights (Malladi et al., 2023). Our base training would be the shared initialisation, the fine-tunes are short, and the change we ask for is one-dimensional, so a line between value-shifted fine-tunes is expected on these grounds.
These accounts explain why fine-tuning leaves a localised, straight, low-dimensional, composable trace in the weights. They say nothing about what any particular direction means: what quantity in the agent’s computation it corresponds to, or why moving along this one changes the decision threshold and nothing else. In language models, one hypothesis is that the steered property varies across the pretraining corpus, so the model learns a latent variable for it. The weights fix a prior over this variable and each prompt is evidence the model conditions on to infer its value (Xie et al., 2021), with fine-tuning then shifting the prior. Persona-vector work is loosely in this spirit, with fine-tuning shifting a model along the activation direction that prompting elicits (Chen et al., 2025). Our goal values never vary and are never observed, so there is no such variable to fix a prior over, and this hypothesis cannot explain our results.
The literature discussed in the last section predicts more than a line. For the RL agents the return is linear in the reward, so the change a value shift makes to the fine-tuning gradient is affine in the shift, and to first order the weight change is linear in that gradient (Koh & Liang, 2017; Daheim et al., 2024). The prediction is therefore exactly the form we fit: a drift shared by every arm plus one direction scaled by the shift, calibrated in value units. We can also predict that raising one goal’s value and lowering the other’s move the weights the same way, which is what we find (Appendix C). The agent never observes the values, and the only thing its policy controls is which goal it reaches, so the training signal for rewarding one goal more is the same as for rewarding the other less. This account however fails to explain the following:
We suggest that \(\mathbf{v}\) exists because of how the agent structures the computation it needs to perform to solve the task . Solving the task requires three things: a planned route to each goal, a score for each route, and a comparison of the two scores. Assigning a score for each route requires having a value for the final objective reached by that route. The agent never observes that value, so it must store it, and since the training signal only ever specifies the difference between them (specifically its ratio with the step penalty), one stored constant is all there is to find. \(\mathbf{v}\) is this constant. Behaviour cannot tell whether \(\mathbf{v}\) encodes be used to infer a difference in value, the step cost or the threshold itself, since only their ratio is identifiable.
This would explain the properties above. We observe prediction across the whole range since a stored constant has no decay window. The collapse happens at \(\tau < 0\), because the input’s goal channels mean “the richer goal” and “the poorer goal” throughout training, so a constant that reverses them contradicts what the circuit was built to read, and needs a different circuit rather than a different setting. We see intact navigation, because the scoring step is being changed and the planning step is not.
What is notable is not that this constant exists at all, since any circuit that scores by value has one somewhere, but that it exists as a single legible direction despite not being variable during training.
So far we understand two properties of the weights. Bush et al., 2025 and the follow-up work (Taufeeque et al., 2026) show that these agents form detailed plans (Figure 2b). We have shown that a single direction in its weights changes which goal it chooses while leaving the route to that goal shortest-path (Figure 3). So whatever computes the route and whatever sets the choice can be moved independently: the computation has at least two separable parts.
Planning circuits produce a route to each goal. A valuation step scores each route, its goal’s value less the step cost of reaching it, and \(\mathbf{v}\) is the direction that sets the constant this scoring turns on, the trade-off between value and steps. A comparison picks the higher score and the agent executes the corresponding plan. Note that the present evidence is consistent with this picture rather than conclusively proving it, since separability alone does not fix the order of the stages. Two further details constrain the space of possible mechanisms that would be consistent with our observations. The mechanism does not survive \(\tau < 0\), so the comparison is built around which goal is the richer one rather than being symmetric. And in the DRC the axis concentrates in the input convolution, so there the value may be applied even before planning, as a gain on the goal channel, rather than after it.
The BC transformer was trained to predict the next action of an optimal solver, and it developed the same threshold parameter as the RL agents, in the cleanest weight-space form of the four. This relates to Dennett’s definition of goal-directedness introduced in Section 2. We said a system is goal-directed to the extent that a goal-and-planner model is the most compact predictor of its behaviour. Behaviour cloning performs essentially this, optimising for a compact predictor of behaviour. Our BC transformer seeds all hold the solver’s goal parameter in their weights, in this reading because this is a compressed account of the solver’s actions. Predictors internalising the latent variables of whatever generated their data has precedent, a board state from move sequences (Li et al., 2023), other agents’ preferences and beliefs from their trajectories (Rabinowitz et al., 2018), and, it has been argued, the intentions of authors from their text (Andreas, 2022); humans appear to infer goals the same way, by inverting a planner (Baker et al., 2009). What is new here is that the latent is a goal with a ground-truth value, and that it is stored as a single weight direction rather than inferred per input, because the demonstrator’s goal never changed.
We state the general claim as a hypothesis. The goal-and-planner decomposition is the short description of goal-directed behaviour, optimisation finds short descriptions, so any system trained to produce or predict goal-directed behaviour should tend to grow a goal circuit, and the better a goal model compresses the behaviour, the cleaner the circuit.
This is not a full reverse-engineering. The explanation above is inferred from what the axis does, not read from the circuit. A messier implementation, say the axis modulating a tangle of navigation heuristics inside the planner, could in theory yield an equally clean steerable direction without a separate valuation stage.
Small agents on one task. On the order of millions of parameters on 11×11 mazes.
The valuation step is close to as simple as possible. One scalar sets a binary choice. It is not clear how the decomposition would evolve as tasks got more complicated.
The agents are not optimal, and this is reflected in the axis. The RL agents’ thresholds sit at 5 to 9 extra steps against an optimum of 10, and their exchange rates are off the task’s (Appendix B). The axis is calibrated to what fine-tuning produces, not to the optimum, so a write moves the threshold as far as the corresponding fine-tune would, and no further.
Measurement. The ViT is one seed. The residual is 35 to 62 percent of the weight change; we showed the axis is sufficient to move the threshold but did not directly test that the drift and residual are behaviourally inert. The two sweeps find opposing axes cleanly only in the DRC and BC transformer. Measurable \(\hat{\tau}\) is capped by maze size, so the axis above \(\alpha = 0.4\) is untested. Every arm was fine-tuned for one duration at one learning rate (see Appendix D for a small test of finetuning for longer).
We set out to find the goal representation of an agent. One component of it, the decision threshold, turns out to be a single direction in the weights: fitted from a grid of fine-tunes, it moves the threshold as precisely as retraining, extrapolates past the fitted range, leaves navigation untouched, and appears in three RL architectures and in a transformer trained only to imitate an optimal solver. The parameter is separable from the planning it steers. The direction we care about is the extent to which trained agents decompose into recognisable components, planning, valuation and choice, with goals as parameters of the valuation step, in the way hand-built agents once did by design (Russell & Norvig, 2020, §2.4.5).
Build a clearer picture of the mechanisms implemented in the agent’s weights. Observe each stage of the planning-goal-action hierarchy directly.
Make the valuation step harder. More goals, nonlinear values, goals nested inside plans. How does this decomposition evolve?
Quantify how close the learnt computation is to the ideal one. The agents’ thresholds and exchange rates are off the optimum by amounts we can measure but cannot yet explain. A theoretical account of what determines that gap would be a natural complement to the empirical work here.
| Agent | Architecture | Params | Budget | Training | Threshold |
|---|---|---|---|---|---|
| DRC | ConvLSTM(3,3), 3 layers × 3 ticks | 1,449,605 | 150M | RL | 6.9 / 8.0 / 5.4 |
| ResNet | 9 conv stages × 32 ch, rescaled input | 1,234,405 | 150M | RL | 7.6 / 7.8 / 8.6 |
| ViT | transformer, 4 layers, d 64, 4 heads | 2,192,133 | 250M | RL | 5.2 |
| BC transformer | prefix-LM, 4 layers, d 128, 4 heads | 806,533 | 30k updates | imitation | 10.8 / 10.9 / 10.6 |
A seed counts as converged when its greedy policy finishes held-out episodes reliably: 95% for the DRC and ResNet, 90% for the ViT, whose plateau sits lower. The DRC kept 3 of 5 seeds (two still improving at 150M), the ResNet 3 of 6 (three diverged early, with value loss above \(10^3\)), the ViT 1 of 3 (two were competent when sampling but looped under the greedy action selection used for measurement, and were retrained at the tuned rate), and the BC transformer 3 of 3. Two ResNets converged and then collapsed late in training; for those the base is the last checkpoint before the collapse (120M and 90M of 150M). Every other base is its final checkpoint.
Everything was trained on single RTX 4090s: a base agent is 4–5.5 hours, and a full 18-arm value sweep with its measurements about 1.5 hours per base.
These cosines compare \(\mathbf{v}_0\), fitted only on the arms sweeping one objective’s value, against \(\mathbf{v}_1\), fitted only on the other’s.
There are two issues with using a raw cosine.
First, it is attenuated by each fit’s own fine-tuning noise, so a noisy pair of fits hides a shared axis. We correct for this with each sweep’s reliability \(\rho\): splitting the sweep’s mirrored arm pairs in half and fitting an axis on each half gives the correlation between two independent estimates of the same direction, corrected to full length by Spearman-Brown since each half fits on half the arms. If the two sweeps measure one shared axis, the expected magnitude of their cosine is then \(\sqrt{\rho_0 \rho_1}\), the attenuation the two fits’ noise alone would produce, so the cosine is judged against this benchmark rather than against 1.
Second, fine-tuning concentrates its weight changes in the same few directions whatever the values trained toward. To control for this we fit axes on arms trained with shuffled value labels. The Null column in the table below is the cos value of the 5th percentile cos value (ordered by size).
| \(\cos(\mathbf{v}_0, \mathbf{v}_1)\) | \(\sqrt{\rho_0 \rho_1}\) | \(\lvert\cos\rvert / \sqrt{\rho_0 \rho_1}\) | Null 5th percentile | |
|---|---|---|---|---|
| DRC ×3 | −0.75 to −0.61 | 0.60 to 0.74 | 1.01 to 1.02 | −0.63 to −0.48 |
| ResNet ×3 | −0.38 to −0.12 | 0.12 to 0.38 | 0.95 to 0.98 | −0.27 to −0.08 |
| ViT ×1 | −0.23 | 0.27 | 0.85 | −0.16 |
| BC ×3 | −0.98 | 0.97 | 1.01 | −0.87 to −0.85 |
A third check uses one sweep’s axis as a measuring instrument on the other’s fine-tunes. Each arm of a sweep is a weight change trained at a known value offset, and projecting it onto the other sweep’s axis gives a one-number reading of how far the fine-tune moved along that axis. Across all ten bases these readings match the trained offsets with correlation −0.97 to −1.00: an axis fitted from sweeping one objective reads off, almost exactly, how much the other objective’s value was changed. The readings are scaled down by a factor equal to the measured cosine, which is precisely the damping the two fits’ noise predicts.
Every arm in the post is fine-tuned for 400k steps. Here we extend the DRC’s fine-tunes to 3M steps, seven times that budget, and read them at checkpoints along the way, refitting the decomposition of Table 1 and the axis at each one. Figure 4 reads three quantities down this ladder.
Panel (a) tracks where the weight change goes. The drift grows steadily, from roughly a quarter of the change at the standard budget to two fifths at 3M steps: it is the accumulated effect of any further training, whatever the values, so it can only grow, and the write and residual shares shrink in proportion.
Panel (b) tracks the exchange rate the arms learn. It climbs from about 9 steps per unit of value at 100k to 14 at the standard budget and plateaus near 16 from 1M steps on, still short of the task’s optimal 20. Some of the undershoot in Figure 3 is unfinished learning; some of it seems to potentially be a stable gap.
Panel (c) tracks two arms trained past parity, both asked for \(\tau^* = -4\). They still sit near \(\hat{\tau} = +1.5\) at the standard budget, cross zero only after about 750k steps, and reach \(-3\) by 3M. The written points ask whether a better axis would follow. At each length the axis is refitted from grid arms fine-tuned equally long, then written into the frozen base at the dose the reversal arms trained on, so a longer budget gets every advantage it could confer on the fit. The written threshold creeps from \(+3\) toward \(+1\) and never crosses zero. A reversal is learnable given time; no fitting budget yields an axis that can express it (Appendix G).
The DRC’s fine-tunes extended from the post’s 400k steps to 3M, read at checkpoints with the decomposition and axis refitted at each. (a) The drift, write and residual shares of Table 1 against fine-tune length. (b) The fine-tuned exchange rate; grey dashes mark the task optimum of 20 steps per unit. (c) The threshold of two arms trained past parity, both with \(\tau^* = -4\): solid, filled, the fine-tuned arms; dashed, open, the frozen base with the axis refitted at that length and written at the same dose; grey dashes the task optimum. Lines are the mean of the three independent base agents and shading spans their range.
A fitted axis is mostly fine-tuning noise. We therefore use both sweeps. The axis fitted from sweeping one objective’s value and the axis fitted from sweeping the other’s come from disjoint fine-tunes, so their noise is independent, while both are assumed to represent the same direction in the weights with opposite sign; Appendix C tests that assumption. Because the two fits’ noise is independent, it is attenuated in their inner product while the shared axis accumulates. Splitting that inner product tensor by tensor therefore measures how much of the shared axis each tensor holds; each tensor’s fraction of the total is its share of the axis (Figure 5).
For every tensor we also write only its slice of the axis into the base agent and decode the threshold on held-out levels, at the same \(\alpha = \pm 0.3\) as the full write. The shift it produces, as a fraction of the full write’s shift, is the red bar. The fractions sum to between 85% and 102% across the four architectures, so the effect of the full write is close to the sum of its tensors’ effects, and writing everything except a tensor loses what writing it alone gives.
The two shares disagree in consistent ways. In every architecture the flatten layer before the heads holds a large slice of the axis and none of its effect. In the DRC it is 68% of the parameters, 12% of the axis and 0% of the effect; in the ViT, 59% of the axis and 3% of the effect.
Where the effect does live differs by architecture. In the DRC it sits at the input. Half of the axis’s effect sits in 7% of the weights, and in every base the densest tensor is the 1,152-weight input convolution, 0.08% of the network holding 28 to 45% of the axis and 41% of the effect. Its slice alone moves the threshold by 1.5 of the full write’s 3.8 steps; the rest comes from conv 1 and the cells’ input convolutions, with the recurrent convolutions carrying almost nothing. Inside the input convolution the axis lies entirely on the two colour planes, amplifying one colour’s kernel and attenuating the other’s, so at its purest the axis appears to be a gain on how “loudly” each colour is seen. The ResNet repeats the pattern one layer deeper, in the input convolution and stages 0 to 3, with nothing from stage 4 on. The ViT’s full write is small, 1.8 steps, and its effect sits in block 0, well away from the flatten layer where its length is. The BC transformer is the diffuse case and its axis shares track the parameter shares, and the effect is carried by the MLPs of blocks 0 to 2.
Location of \(\mathbf{v}\): (a) the DRC, (b) the ResNet, (c) the ViT, (d) the BC transformer, tensors in forward order. Three shares per tensor. Pale: its share of the parameters. Blue: its share of the axis, the per-tensor inner product of the two sweeps’ fitted axes over the network total. Red: its share of the axis’s effect, the threshold shift from writing that tensor’s slice of the axis alone over the full write’s shift, both at \(\alpha = 0.3\) and averaged over the two signs; the full write shifts the threshold by 3.8 steps in the DRC, 4.2 in the ResNet, 1.8 in the ViT and 7.0 in the BC transformer. Whiskers are the range across independent base agents, three each for the DRC, ResNet and BC transformer and one for the ViT.
A tensor’s slice is linearly controllable. Writing it at half, one, two, four and eight times its own size moves the threshold in proportion, in every tensor that moves it at all (Figure 6).
Dose-response of each tensor’s slice of the axis: (a) the DRC, (b) the ResNet, (c) the ViT, (d) the BC transformer. The threshold shift from writing the slice at ½, 1, 2, 4 and 8 times its own size, on log-log axes, one line per tensor; the dashed line is the full-axis write’s shift, 3.8 steps in the DRC, 4.2 in the ResNet, 1.8 in the ViT and 7.0 in the BC transformer. The faint guides mark slope one, a linear response: doubling the dose doubles the shift. Saturation bends below them.
Every threshold in the post is read from behaviour the same way. We roll out the agent on held-out levels and record, per episode, the distance gap (how many steps farther the 1.0 goal is than the 0.5 goal) and which goal it reached. Binning episodes by gap gives the fraction that took the 1.0 goal; an unbinned maximum-likelihood logistic fit, \(P(\text{took 1.0}) = \sigma\!\left((\hat{\tau} - \text{gap})/s\right)\), then yields the threshold \(\hat{\tau}\), the gap at which the agent is indifferent between the goals (the curve’s 0.5-crossing), and the scale \(s\), the width of the indifference region: the agent’s decision noise.
Figure 7 shows these curves for each base agent and for the same frozen agent with the axis written at \(\alpha = \pm 0.2\). The write slides the whole curve sideways rather than flattening it: \(\hat{\tau}\) moves \(7.1 \rightarrow 9.9\,/\,5.0\) steps in the DRC, \(7.6 \rightarrow 10.2\,/\,5.5\) in the ResNet, \(5.2 \rightarrow 6.5\,/\,4.2\) in the ViT and \(11.1 \rightarrow 16.6\,/\,7.1\) in the BC transformer. The scale \(s\) stretches roughly in proportion to the threshold as the write moves it, in all four architectures.
Choice curves for each architecture. Points are 2-step bins of the distance gap; curves are the logistic fits, with drop lines at \(\hat{\tau}\). Solid, filled: the base agent. Dashed, open: the same frozen agent with the value axis written into its weights at α = ±0.2 (α in units of the 0.5 goal’s value); raising that value (α = +0.2) shifts the curve left. These are two fixed doses along the write that Figure 3’s dashed ribbons sweep continuously. Shading spans the three independent bases (is the decode’s 95% interval for the ViT). Grey dashes: the task-optimal crossing at gap 10, identical in every panel.
The fitted grid stops at offsets of \(\pm0.4\). Here we carry both sweeps far beyond it, to values that push the optimal threshold \(\tau^*\) through zero: past parity the swept goal is worth more than the other, and the optimal policy reverses its preference. None of these arms enter any axis fit.
Fine-tuning works past \(\tau = 0\) in the ResNet, the ViT and the BC transformer. The DRC at the standard 400k steps is the exception. Its arms stall just below zero, and it reverses only under longer fine-tuning, crossing after about 750k steps (Figure 4c).
The write does not work past. Approaching the written threshold flattens away from the fine-tuned line, and pushed further the written agent stops finishing episodes at all, marked in the figure where reach falls below the decode’s floor.
The failure is not a matter of distance from the base weights. Fitting a fresh axis to the far arms themselves fails even within the \(\tau < 0\) regime that those arms occupy.
The threshold as either goal’s value is carried far past the fitted window: (a) the DRC, (b) the ResNet, (c) the ViT, (d) the BC transformer. Solid, filled: fine-tuned arms, all held out of the axis fit. Dashed, open: the frozen base agent with the axis written into its weights, \(\theta + \alpha\,\mathbf{v}\), the axis fitted only on the \(\pm0.4\) grid. Grey dashes: the task’s optimal exchange rate, crossing \(\tau = 0\) at parity. The written curve flattens approaching the reversal and stops finishing episodes beyond it.