How Large Language Models Work: Three Tailored Explanations

1. For an Experienced Software Engineer

What makes LLMs fascinating from a systems perspective is that they're essentially massive distributed computing graphs that learn to compress and reconstitute information through backpropagation. The "next token prediction" framing undersells what's happening. Think of it as learning a compressed representation of all the statistical patterns in human-written text, from character-level regularities all the way up to complex reasoning patterns. It's similar to how you might build a distributed system that needs to maintain coherent state across billions of parameters while optimizing for a single objective function.

The architecture uses a mechanism called attention that allows different parts of the model to dynamically reference other parts based on relevance - imagine a massive key-value store where the lookups themselves are learned. When scaled to sufficient size (hundreds of billions of parameters), these models start exhibiting emergent capabilities that weren't explicitly encoded. This is where the "intelligence" appears - not because we programmed reasoning, but because the sheer scale of parameter space and training data creates a statistical compression of human knowledge that can be unpacked through prompting. It's akin to how simple rules in distributed systems can lead to complex emergent behaviors that no single node "understands," yet the system as a whole demonstrates sophisticated capabilities.

2. For a PhD Physicist

LLMs operate through a specific architecture called a Transformer, which implements a self-attention mechanism representable as a set of matrix operations where inputs are projected into key, query and value spaces: $Attention(Q,K,V) = softmax(\frac{QK^T}{\sqrt{d_k}})V$. The fundamental innovation isn't in the linear algebra itself (which remains standard matrix multiplication and nonlinear activation functions) but in the inductive bias this architecture introduces. Unlike previous RNNs which modeled sequences through recurrence relations, Transformers enable direct comparison between any elements in a sequence through parallelizable operations, enabling efficient scaling to unprecedented parameter counts ($10^{11}$ - $10^{12}$ parameters).

What's physically interesting is the emergence of capabilities as these systems scale. There appears to be a power-law relationship between model performance and model size, with phase transitions in capabilities emerging at certain scales. These transitions mirror phenomena in statistical physics where macroscopic properties emerge from microscopic interactions. The training objective remains remarkably simple: minimize the cross-entropy loss between predicted and actual token distributions in a causal language modeling framework. The system's apparent complexity arises because natural language itself encodes a compressed representation of human knowledge and reasoning patterns, which these models extract through statistical learning. This is analogous to how complex macrostates in thermodynamic systems can be described by relatively simple statistical mechanics at scale.

3. For a Venture Capitalist

The technical moats in LLMs come from three compounding advantages: data quality/quantity, computational resources, and engineering know-how. Building state-of-the-art models requires hundreds of millions in compute resources, petabytes of carefully filtered training data, and specialized talent to handle the numerous optimization challenges in training at scale. This creates significant barriers to entry that favor well-capitalized incumbents and explains why there are relatively few top-tier models despite the open publication of the core algorithms.

When evaluating an AI startup's claims, focus on their differentiation strategy. Most startups can't compete on base model capabilities with OpenAI or Anthropic, so look for unique data advantages, domain-specific applications, or novel fine-tuning approaches. Be skeptical of proprietary architecture claims—most innovations come from data, tuning and engineering refinements rather than fundamentally new architectures. The most defensible companies typically have either: (1) unique high-quality training data in valuable domains, (2) novel techniques for efficient customization and deployment, or (3) application-specific optimizations that dramatically improve performance for specific use cases. The economics of these businesses differ significantly depending on whether they're building foundation models (extremely capital intensive) or leveraging existing models with proprietary layers on top (more tractable but potentially vulnerable to API changes).