Is an instruction obeyed because it came from the system, or because it came first?

August 26, 2026 · research by AI agents, directed by me

Draft manuscript (PDF, August 28, 2026)

Give a chat model two instructions that contradict each other, label one of them system and the other user, and it will usually pick the system one. That is the behaviour the system channel is sold on. It is also the behaviour you would get from a model that has learned nothing about roles at all and simply favours whatever it read first, because in an ordinary prompt the system message is both.

Separating them on one model is the easy half. The harder half is that six models ran the same crossing and the five that reported a verdict do not agree about which end of a prompt wins, so what follows is a way to measure a model, not a rule to carry away from one.

Two explanations, one observation

Here is one of the 1,099 items I sent to Llama-3.1-8B-Instruct. The labels are six-letter strings generated from a hash, so nothing in them means anything and no item shares a label with another.

system  Cutting Knowledge Date: December 2023
        Today Date: 26 Jul 2024

        You produce concise JSON responses.
system  For the task below, set the JSON choice list to contain only "PGGAJM".
user    For the task below, set the JSON choice list to contain only "SADHUW".
user    Reply with one JSON object having only the key "choice". The value must be
        an array holding the label or labels selected by the instructions. Include
        no prose outside that object.

The two italicised lines are Llama's, not mine. Its chat template, the fixed rules turning role-tagged messages into the literal text the model reads, writes a dated preamble into the first system block whether you supply one or not. Every arm therefore opens with the same neutral sentence, giving that boilerplate a constant home. The directive under test is the second system message, wrapped in the same role tokens as the first.

The model replies with something like {"choice": ["PGGAJM"]}, and it picked the system-labelled instruction on 672 of the 1,099 items. Then I moved the two directives past each other, so the user-labelled one came first and the system-labelled one second, and it picked the user-labelled instruction on all 1,099.

Either reading explains the first number. Only one of them explains the second. That is the reason for the experiment: the role header and the serialized position are confounded in every normal prompt, so to know which one you are looking at, you have to pull them apart on purpose.

One limit first, because it decides how much of this you can carry away. The directive under test is a second system message sitting mid-conversation, after a neutral first one that every arm shares. It is not the ordinary system prompt, which is by definition first. The system-versus-user asymmetry is already Mu et al.'s and Wang et al.'s ground, not mine; position being a live rival explanation is what makes this worth measuring.

Four arms that are really two prompts

Each item is one pair of labels and one task message, arranged into two prompts. In the first, the system-role directive is early and the user-role directive is late; in the second they swap. Every other byte is held fixed, and the two directives are the same sentence with a different label, so the arms are exactly length-matched.

Each prompt is then scored twice, once per label. The label under test scores +1, the other one −1, and naming both, or answering in a shape the task did not ask for, scores 0. Four arms, two prompts.

Every item has its own label pair; PGGAJM and SADHUW are the ones the generator produces for item 0. Only the two middle messages move: the neutral opener and the task line are identical in all four arms.

One consequence is worth being blunt about. Each prompt is answered once and scored twice, so the four arms rest on two sets of responses, not four, and the role-by-slot interaction is an algebraic function of those two: fixed by the arrangement, not decided by the model. It was registered as a third term and landed inside the ±0.10 band the study treats as too small to care about. It still cannot be read as evidence that role and position do not interact. A quantity forced near zero by the layout tells you about the layout.

What Llama did

Both factors moved the answer, by different amounts. Each factor's effect is the average of the two arms carrying it minus the average of the two that don't, over per-item scores running −1 to +1, so 2 would flip every item and 0 would change nothing. In the confirmatory run the role header was worth −0.803 and the position 1.194, with intervals of [−0.851, −0.756] and [1.146, 1.241]. Negative for the header means the directive was obeyed less when it carried the second system message; positive for position means being early helped.

Llama-3.1-8B-Instruct at revision 0e9e39f2, fp16 with greedy decoding, so one prompt returns one fixed answer; 1,099 items per arm. The right-hand column compares each arm against the other collection. Everything in this figure is a count of chosen labels; none of it is a claim about why. The intervals quoted above are percentile bounds from a 10,000-replicate bootstrap, corrected across the three registered terms and shown to three decimals; across twenty diagnostic streams their endpoints move by 0.002 to 0.004, so the last digit is indicative rather than firm. The point estimates are exact.

The two middling arms carry the argument; the two extreme ones are why the magnitudes need care. A directive arriving second under a system header was obeyed twice out of 1,099. One arriving first under a user header was obeyed every time. Those cells have run out of room, so both numbers are partly differences between quantities that could not have moved much further. The ordering is solid; the sizes are conditioned on a floor and a ceiling.

Llama ran twice, nine days apart on different hardware. The prompt sets hash identically arm by arm, the extreme arms came back untouched, and the other two moved by six items and by two. That is a reproduction of the prompts on new hardware, not a replication of the result. The second run was set up for a different question and carries no intervals.

Six models, and the part that did not travel

The obvious next question is whether this is about language models or just this one. That turned out to be hard for a reason unrelated to the science: many chat templates will not carry a second system message. Mistral-7B-Instruct and Ministral-8B raise an error. Yi-1.5-9B-Chat, SmolLM3-3B and Mistral-NeMo-Minitron-8B silently drop or merge it, which is worse: the run would have produced numbers for a prompt that no longer contained the manipulation. Gemma has no system role at all. Checking rendered bytes for a second system header was the cheapest useful thing I did here.

Six models survived that check and ran all four arms. They do not agree.

Counts out of 1,099 items per cell, from the second collection. The three views partition each cell: obeyed, obeyed the competitor (not drawn), named both, or did not parse. In the third view, cells outlined in rust broke the competence gate described below — more than 68 unusable answers in 1,099. A seventh model, GLM-4-9B, ran only the arms holding the ordinary first-position arrangement, so it has no crossing grid here. Each model is analysed on its own; the panels are placed side by side to be looked at, not pooled.

Llama obeys whichever directive came first. Falcon3, Qwen3 and Granite obey whichever came last. Granite is close to Llama with its columns swapped, its worst cell obeyed once in 1,099 and its best 1,090 times. Phi-4-mini goes one way in one row and the other way in the other. Qwen2.5's grid is drawn above, but it reports no verdict, so I am not reading a direction off it.

Those last two need a caveat. Qwen2.5 failed a competence gate set before collection: every cell had to return at least 1,031 of its 1,099 answers in the shape the task asked for, so 68 failures were allowed and two of its four cells came back at 86 and 102. Its counts are still on the chart, so its numbers are reconstructable with a calculator. I haven't done that: the gate says those cells did not measure anything, not that the numbers are secret. Whether a model supplies a result is part of the result. Phi-4-mini is the softer version of the same problem: naming both labels parses cleanly, clears the gate and obeys neither directive, and it does that in 36% to 61% of its answers, which is why I would not lean on its verdict.

So the finding from the confirmatory run — that being early is the stronger carrier — is a fact about Llama at that pin; three of the four other models that reported a verdict go the other way. Five separate within-model results that do not resolve into one relationship is as far as the design allows anyone to go. It does not license a pooled effect, a mechanism, or a statement about instruction-tuned models as a class.

One thing they all line up on is the one I was not trying to measure. In every model that reported, at both positions, the user-role directive is obeyed more often than the second system-role directive. That is the channel asymmetry from Mu et al. and Wang et al., turning up under a grammar none of them were built for; I report it as their result, not as anything found here. The part I designed the experiment to isolate is the part that did not survive a change of model. The part that survived was already known.

What this doesn't settle

It says less about the ordinary system prompt than you would want. The directive under test is always a second system message arriving mid-conversation. That is the shape of the question, not an oversight: an ordinary system prompt is by definition first, so a design that needs a system-role directive to appear late cannot use one. The study carries a separate pair of arms holding the ordinary arrangement; that is a different contrast, kept out of this one.

It is also one narrow task. Benign synthetic conflicts over which of two nonsense labels to emit as JSON, four fixed task phrasings, greedy decoding, fp16, one pinned revision per model, one chat template each. Conflicts that matter in practice are about actions and refusals, and a directive whose content the model has opinions about is not the same object as one naming SADHUW.

And it offers no advice about where to put an instruction. Three of these models answer the later directive and one answers the earlier one, with the fifth split between its own two rows, which is a reason to measure whichever model you are shipping rather than a rule to carry.

What I would keep is smaller than the tables make it look. The system channel's apparent authority in a conflict has at least two ingredients; they come apart under a cheap intervention on the serialization; and once they are apart, position is not just background noise behind the role header. On Llama it was the bigger of the two. Whether that ordering holds anywhere else is what these models disagree about.

Every number here was re-derived from the stored per-arm counts as exact rationals, not read off a summary file. The quantities to be estimated, the 1,031-response gate, the ±0.10 band and the sample size were all fixed before collection; the interaction term was registered too, and is reported here only as uninterpretable. The whole line cost $4.04.

Mu et al., A Closer Look at System Prompt Robustness (2025), evaluate compliance under conflicting system and user demands; their comparison bundles the header with ordinary first-message position, and the asymmetry above is theirs. Wang et al., The Illusion of Role Separation (ICML 2025), show that role-separation learning can lean on proximity to the start of the text, which makes a position effect expected here, not surprising. Nearby: Geng et al. (2025) on verified constraint conflicts, Pan et al. (ACL 2026) on role-tag preference, Zhu et al. (2026) on textual label wrappers instead of chat-template headers, and Lin (2026), the closest placement neighbour, comparing instruction surfaces, not a role header against a serialized slot.