We can actually figure out a lot about LLMs. Just not by asking them.

The funny thing about LLMs is that the most tempting questions are often the weakest measurements. "Do you have a world model?" "What language do you think in?" "Are you hiding reasoning?" Those questions produce text. The text may be useful, but it is not privileged telemetry.

The previous latent machines note argued that prompts can assemble temporary procedures: scratchpads, critics, verifiers, examples, and safety postures. This piece is the harder follow-up. Which of those claims can we actually figure out? A surprising amount, if we stop treating the transcript as the whole system.

The falsification ladder

The right question is not "what did the model say about itself?" The right question is "what observations would change our mind?" The ladder below moves from cheap black-box tests to stronger causal evidence.

Claim 1: prompt tricks are interface effects

The strange part of prompting is not that a magic phrase unlocks intelligence. It is that a small prompt can change the computational regime the model enacts. Chain of thought, least-to-most decomposition, self-consistency, and Tree of Thoughts all share a practical idea: generate intermediate artifacts that the model can attend to later.

That makes prompt work measurable. Run answer-only, constraints-first, generator-critic-verifier, and self-consistency prompts across the same held-out task set. If accuracy, calibration, or error recovery moves, you have an interface effect. You do not have proof of private deliberation. You have proof that the input-output protocol matters.

Claim 2: chain of thought is instrumentation, not ground truth

Chain-of-thought text is powerful because it gives the model a working surface. It is also dangerous to overread. Turpin et al. showed that explanations can omit the feature that actually biased the answer in multiple-choice settings (Language Models Don't Always Say What They Think). Lanham et al. measured cases where generated reasoning is faithful, unfaithful, or partly causal depending on the task and model (Measuring Faithfulness in Chain-of-Thought Reasoning).

OpenAI's 2025 work on chain-of-thought monitoring and monitorability makes the same boundary vivid: reasoning traces can expose useful warning signs, but optimizing directly against traces may teach models to hide the trace rather than remove the behavior. The transcript is instrumentation. Treat it like a sensor that can be noisy, gameable, and still valuable.

Claim 3: features are real enough to perturb

The older intuition was that neural networks were hopelessly smeared out. The current interpretability picture is more uncomfortable: many features are distributed and superposed, but some can be isolated well enough to steer or trace. Anthropic's Scaling Monosemanticity work found sparse-autoencoder features in a frontier assistant model that map to recognizable concepts, styles, and behaviors. The follow-on circuit tracing methods and biology case study go further by trying to trace how features interact during a response.

The claim is not "we can read the whole mind." The claim is narrower and more useful: features can sometimes be found, named, scaled, suppressed, or routed through attribution graphs. The falsifier is causal. If a probe predicts a feature but perturbing it does nothing, it may be a passenger. If scaling it changes behavior with tolerable collateral damage, you have a stronger handle.

Claim 4: world models are an intervention question

"World model" is a loaded phrase. In LLM research it should mean something operational: the model internally represents state variables that help it predict future tokens or task outcomes. The Othello-GPT work found evidence that a model trained on game transcripts learned board-state-like representations (Emergent World Representations). Gurnee and Tegmark reported that language models encode useful latitude, longitude, and time information for many entities (Language Models Represent Space and Time).

The unbelievable-sounding version is "the model has a physics engine." The testable version is better: can we decode a latent variable, intervene on it, and predict the downstream output shift? If yes, the model has a task-useful state representation. If no, the phrase was probably poetry.

Claim 5: emergence is partly a statistics problem

The 2022 emergent-abilities paper made a strong observation: some benchmark abilities seem to appear abruptly as scale increases (Wei et al.). Schaeffer et al. then argued that many apparent phase transitions are artifacts of discontinuous metrics, especially exact-match scoring (Are Emergent Abilities of Large Language Models a Mirage?).

The practical conclusion is not "emergence is fake." It is "never trust a cliff until you swap the ruler." A sharp jump on exact match may become a smooth curve under edit distance, log probability of the correct answer, partial credit, or a mechanistic progress measure. The cool finding is the one that survives the metric swap.

Claim 6: safety is behavior under pressure

Safety behavior is not a single binary filter bolted onto the end of a model. Instruction tuning, reinforcement learning from human feedback, constitutional training, classifiers, system instructions, and product policy all shape the behavior that finally reaches the user. Constitutional AI showed one route for using model-generated critiques and principles to improve assistant behavior (Bai et al.). Red-team work showed why this cannot be evaluated only on clean prompts (Perez et al.).

The uncomfortable edge is persistence. Universal adversarial prompt research and sleeper agent experiments show that undesirable behavior can survive more pressure than a casual demo suggests (Zou et al., Sleeper Agents). The defensive lesson is concrete: evaluate safe alternatives, refusal consistency, paraphrase robustness, multiturn drift, and hidden-trigger sensitivity. Do not publish recipes for bypassing safeguards. Publish measurement protocols.

The ten tests I would run

What people still under-believe

The weirdest thing is not that LLMs "know secrets." They do not need secrets to be strange. The weird thing is that four views of the same model can disagree: black-box behavior, human-language transcript, hidden representation, and causal intervention. A model can solve a task without explaining the real reason. A probe can decode a variable that is not causally used. A prompt can improve output without revealing an inner monologue. A safety behavior can look stable until the distribution shifts.

That is why the frontier is not better folklore. It is triangulation. Treat prompts as experiments, reasoning traces as imperfect instruments, features as hypotheses, and interventions as the evidence that matters most. The strongest LLM claims are the ones that can survive being pushed from all four sides.

Sources

• Wei et al. - Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Wang et al. - Self-Consistency Improves Chain of Thought Reasoning
• Yao et al. - Tree of Thoughts
• Turpin et al. - Language Models Don't Always Say What They Think
• Lanham et al. - Measuring Faithfulness in Chain-of-Thought Reasoning
• Radhakrishnan et al. - Chain of Thought Reasoning In The Wild Is Not Always Faithful
• OpenAI - Chain-of-thought monitoring
• OpenAI - Evaluating chain-of-thought monitorability
• Elhage et al. - Toy Models of Superposition
• Bricken et al. - Towards Monosemanticity
• Templeton et al. - Scaling Monosemanticity
• Anthropic - Circuit tracing methods
• Anthropic - On the Biology of a Large Language Model
• Olsson et al. - In-context Learning and Induction Heads
• Li et al. - Emergent World Representations
• Gurnee and Tegmark - Language Models Represent Space and Time
• Wei et al. - Emergent Abilities of Large Language Models
• Schaeffer et al. - Are Emergent Abilities of Large Language Models a Mirage?
• Nanda et al. - Progress measures for grokking via mechanistic interpretability
• Zou et al. - Representation Engineering
• Turner et al. - Activation Addition
• Rimsky et al. - Contrastive Activation Addition
• Bai et al. - Constitutional AI
• Perez et al. - Red Teaming Language Models with Language Models
• Zou et al. - Universal and Transferable Adversarial Attacks on Aligned Language Models
• Hubinger et al. - Sleeper Agents