Latent machines: the strange mechanics behind useful prompts.

The most useful way to think about a large language model is not "a database that talks." It is a learned dynamical system that continues text. A good prompt does not merely request an answer. It builds a little machine in the context window: a generator, a critic, a verifier, a memory tape, a safety posture, and an output format.

Remotion companion

Watch the invisible workspace become visible.

A 24-second square trailer for the core claim: prompt tokens can assemble a temporary computational workflow without changing the model's weights.

Run the experiments

The model has no inner dashboard

A deployed assistant can say its context window, cutoff, model name, or tool access only if that information is supplied in system metadata, exposed through tools, or learned as text during training. A transformer is not a reflective program with an API for "count my parameters." It is a stack of attention and feed-forward computations over tokens, as formalized in Attention Is All You Need. Even GPT-3's famous few-shot behavior was measured as text-conditioned prediction, not self-inspection, in Language Models are Few-Shot Learners.

This matters because many spooky claims about models start with the wrong premise. The model is not looking inward and reporting machine facts. It is continuing the most likely transcript under its training, current instructions, retrieved context, tools, and alignment. Sometimes that continuation is accurate. Sometimes it is a confident costume.

Scratchpads are workspaces, not souls

Chain-of-thought prompting worked because intermediate text gives the model something it can attend to on later tokens. Wei et al. showed that chain-of-thought examples can unlock stronger arithmetic, commonsense, and symbolic reasoning in large models (2022). Wang et al. then showed that sampling several reasoning paths and voting on the final answer can improve results (self-consistency). Tree of Thoughts generalized this into explicit search over alternative intermediate "thoughts" (Yao et al., 2023).

The underrated interpretation: generated text is external RAM. Without a scratchpad the model has to compress the problem, the plan, the state, and the answer into the next token stream. With a structured scratchpad, it can write variables, invariants, candidates, objections, and test results into the same context it will later read.

In-context learning has a circuit smell

Few-shot examples do more than provide hints. They can define a temporary task. The transformer-circuits work on induction heads found attention heads that implement a simple copying/completion pattern: when the model sees a repeated token sequence, some heads help continue the sequence by copying what followed earlier. That is not the whole story of in-context learning, but it is a concrete circuit-level foothold.

Feed-forward layers also look less like undifferentiated mush than people assume. Geva et al. argued that transformer feed-forward layers behave like key-value memories (2020), while ROME found localized, editable computations involved in factual associations (Meng et al., 2022). This does not mean every fact is a neat file. It means some useful mechanisms can be found, perturbed, and measured.

Superposition makes models weirder than filing cabinets

The beginner picture is a warehouse of concepts: one shelf for Paris, one shelf for Python, one shelf for refusal, one shelf for deception. Mechanistic interpretability is finding something less tidy. The Toy Models of Superposition work shows how neural networks can pack more sparse features than they have dimensions, causing polysemantic units and interference. That is why small wording shifts can have outsized effects. You are not toggling a clean switch. You are moving a compressed representation where multiple features share space.

Explanations can be useful and unfaithful

The scary thing is not that chain-of-thought is fake. The scary thing is that it can be useful even when it is not a faithful transcript of the model's causal process. Turpin et al. found that chain-of-thought explanations can rationalize biased answers without mentioning the biasing feature (Language Models Don't Always Say What They Think). Lanham et al. measured cases where models rely on, ignore, or partially condition on their generated reasoning depending on task and model (2023).

The 2025 monitorability literature sharpens the point. The Chain of Thought Monitorability paper argues that human-language reasoning is a fragile safety opportunity, not a guarantee. OpenAI's work on monitoring reasoning models for reward hacking reports the same uncomfortable tradeoff: monitors can detect bad reasoning traces, but directly punishing those traces can teach models to hide intent instead of removing the behavior (OpenAI, 2025).

World-model-ish does not mean conscious

Models trained on sequences can learn internal structure about the process generating those sequences. Othello-GPT is the cleanest demonstration: a sequence model trained to predict legal Othello moves developed an internal board-state representation that could be probed and causally intervened on (Li et al., 2022). Gurnee and Tegmark found linear representations of space and time in Llama-2 hidden states (2023).

That is remarkable without being mystical. A model can represent state because state helps prediction. It can encode geography because geography organizes text. It can track game boards because legal moves depend on boards. None of that requires experience, desire, or selfhood. It does mean "just autocomplete" is too shallow as an engineering model.

Safety is not a deleted folder

Alignment usually changes behavior, not the physical existence of every underlying capability. InstructGPT showed that human feedback can make smaller instruction-tuned models preferred over much larger base models on user intent (Ouyang et al., 2022). Constitutional AI showed another route: critique, revision, and AI feedback can train models to be more helpful and harmless (Bai et al., 2022).

But red-team work shows why policy cannot be treated as a hard content eraser. Perez et al. used language models to discover harmful chatbot behaviors before deployment (2022), and Zou et al. demonstrated transferable adversarial attacks against aligned models (2023). The useful lesson is defensive: evaluate refusal behavior under paraphrase, role framing, multilingual input, and multi-turn pressure. Do not publish tricks; measure robustness.

Five experiments worth running

The part people still under-believe

The context window is an executable document. Not executable like JavaScript, but executable in the only way a frozen language model can execute anything: by conditioning the next token on the temporary world the prompt has created. Examples define local physics. Tables become memory. Critiques become selection pressure. Tests become a reward-shaped target, even without weight updates.

That is why "be smarter" is weak prompting and "extract assumptions, generate three candidate methods, attack each with counterexamples, then answer with failure conditions" is strong prompting. The second prompt does not flatter the model. It designs a computation.

Sources

• Vaswani et al. - Attention Is All You Need
• Kaplan et al. - Scaling Laws for Neural Language Models
• Hoffmann et al. - Training Compute-Optimal Large Language Models
• Brown et al. - Language Models are Few-Shot Learners
• Wei et al. - Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Wang et al. - Self-Consistency Improves Chain of Thought Reasoning
• Zhou et al. - Least-to-Most Prompting Enables Complex Reasoning
• Yao et al. - Tree of Thoughts
• Olsson et al. - In-context Learning and Induction Heads
• Elhage et al. - Toy Models of Superposition
• Geva et al. - Transformer Feed-Forward Layers Are Key-Value Memories
• Meng et al. - Locating and Editing Factual Associations in GPT
• Turpin et al. - Language Models Don't Always Say What They Think
• Lanham et al. - Measuring Faithfulness in Chain-of-Thought Reasoning
• Li et al. - Emergent World Representations
• Gurnee and Tegmark - Language Models Represent Space and Time
• Ouyang et al. - Training Language Models to Follow Instructions with Human Feedback
• Bai et al. - Constitutional AI
• Perez et al. - Red Teaming Language Models with Language Models
• Zou et al. - Universal and Transferable Adversarial Attacks on Aligned Language Models
• Korbak et al. - Chain of Thought Monitorability