Evaluating language models on human behavior
Senior thesis · Yale Computer Science · advised by Dr. David van Dijk · Spring 2025
Can a language model predict how a specific person will answer a survey question, or does it just learn the shape of the questions? My senior thesis built the evaluation to tell those two apart. The models I tested leaned on the shape. I think that says more about their scale and unfinished pre-training than about the ceiling for the approach, but the evaluation is exactly what you’d use to find out.
Fine-tuned language models beat a strong statistical baseline on questions they’d seen during training, but fell below it on questions they hadn’t: they were exploiting surface patterns, not learning transferable behavior. A follow-up ablation showed that much of the apparent benefit of behavioral pre-training came from the model learning the answer format rather than the behavior itself.
The question
BehavioralGPT (BPT) is a research foundation model that represents a person not as a block of text but as a point in a small “simplex” of learned behavioral archetypes: a compressed, interpretable profile that the model conditions on before it answers. The hypothesis is that decoupling who someone is from the words they use should help a model generalize human behavior better than an off-the-shelf chat model does.
My thesis tested that hypothesis. I designed the downstream evaluation: a battery of tasks that ask whether BPT and a standard Qwen2.5 chat model can predict how real people answer real survey questions, and whether either one generalizes beyond what it memorized.
The architecture being tested
BPT (my collaborator Olin Geimer’s design) is a variational autoencoder wrapped around a language model. The encoder reads the answers a person is already known to have given and compresses them into a point on a low-dimensional simplex: a set of mixture weights over learned behavioral archetypes, non-negative and summing to one. The decoder is the language model itself, which conditions on that profile while predicting the person’s answer to a new question. Training pulls on both ends in the usual VAE way. A reconstruction term rewards the decoder for recovering the person’s held-out answers, and a KL term keeps the encoder’s posterior close to a prior, so the latent space stays smooth instead of degenerating into one memorized point per person.
The simplex is the interesting choice. An unconstrained latent vector would compress just as well, but mixture weights over archetypes are legible: each vertex should come to stand for a recognizable cluster of people, and any individual is a blend you can read off directly. If pre-training works, similar respondents land near each other and traits like political party or gender fall out as directions in the space. That legibility gave my evaluation something concrete to inspect when the results came back negative.
What I built
I drew on two large, nationally representative surveys, Pew Research’s American Trends Panel (Wave 70) and the OECD’s Survey on Drivers of Trust in Public Institutions, covering roughly 68,000 respondents and 337 behavioral variables. Each respondent became a block of question–answer pairs, which I split three ways:
- Profile questions, shown to the model as known context about the person.
- Holdout questions, used to fine-tune it to predict that person’s answers.
- Test questions, held out entirely and used only to score it.
The design choice that made the evaluation work was a two-by-two split: every test case was labeled by whether the participant and the question had been seen during fine-tuning. That separates four very different things a model could be doing: memorizing a person, memorizing a question, or actually generalizing to a new person or a new question. Both models were fine-tuned with LoRA (trainable adapter weights on a frozen Qwen2.5 backbone), so the comparison turned on the behavioral representation rather than raw model capacity.
How I measured it
Raw accuracy is misleading when questions have different numbers of options, so I normalized it: categorical questions are scored so that random guessing earns 0 and a perfect answer earns 1; ordinal questions (like age brackets) get partial credit for being close to the right answer. Then I held every model to two baselines: a uniform baseline (blind guessing) and, the one that matters, a distributional baseline that guesses in proportion to the true answer distribution. A model only counts for anything if it beats that distributional guess. I also ran a variance study across question splits, because the particular split turned out to move scores more than the models did.
What I found
The numbers below show a clear gap between seen and unseen questions, and almost none between seen and unseen people:
| Evaluation case | Fine-tuned LLM | Distributional baseline |
|---|---|---|
| Seen participant · seen question | 0.46 | 0.34 |
| Unseen participant · seen question | 0.50 | 0.35 |
| Seen participant · unseen question | 0.17 | 0.31 |
| Unseen participant · unseen question | 0.25 | 0.35 |
Normalized categorical accuracy on the OECD Trust Survey (0 = random, 1 = perfect). On unseen questions, the model scores below a simple distributional guess.
On questions it had seen, the fine-tuned model comfortably beat the baseline. On questions it hadn’t, it dropped below it. The model wasn’t carrying any generalizable understanding of human behavior across to new questions; it was leaning on surface patterns from the questions it had already been trained on. The simplex model, meanwhile, showed no significant advantage over the plain chat model. Its pre-training was unfinished, and its latent space didn’t yet organize people by traits like political party or gender the way the architecture intends.
Separating format from behavior
There was one confound I kept coming back to. When a “behaviorally pre-trained” model fine-tunes faster, is it learning behavior, or just the shape of the task (the special tokens, the way an answer is supposed to look)? To isolate that, I built a trivial dataset: every question had exactly one answer choice (“…the largest planet in our solar system? Options: Jupiter.”). No reasoning is possible; the only thing left to learn is the format.
A behaviorally pre-trained model mastered that dataset in under an epoch; a base model was still struggling after two. So a large share of the apparent “pre-training advantage” was the model learning the answer format, not anything about people. That one control changes how you read the other numbers in the study.
Honest scope
These are exploratory results, and I’d locate the problem in scale rather than theory: the models were small, and BPT’s pre-training was not finished when I submitted. So this is a negative result about one under-trained run, not a verdict on the architecture, which may well prove out with more compute. The BPT model itself (its architecture, training code, and pre-training) is the work of my collaborator Olin Geimer; the data pipeline began from a survey parser by Mati Hassan that I adapted and extended. My contribution is the evaluation: the task design, the normalized metrics and baselines, the seen/unseen split, the variance study, and the single-option ablation.
What would settle it
That run hasn’t happened yet. Once BPT finishes pre-training, or the same design gets tried at a serious scale, the column to watch is the unseen questions. A model can score well on questions it trained on without having learned anything general about people, so the real test is whether it beats the distributional baseline on questions it never saw. That’s the number I’d check first.