Explain Like I'm a Specific Expert

For the Experienced Software Engineer

Think of training an LLM as building a compression algorithm for human knowledge, except instead of minimizing file size, you're minimizing prediction error across trillions of tokens. The architecture is a transformer—essentially a massive function that takes a sequence of tokens and outputs a probability distribution over what comes next. During training, you backpropagate gradients to adjust billions of parameters so the model's predictions match actual text. What's counterintuitive is that "predict the next word" is a deceptively hard objective. To predict well, the model has to learn syntax, facts, reasoning patterns, even theory of mind—because all of those improve prediction accuracy. You're not explicitly programming these capabilities; they emerge because they're useful for the objective.

The "just predicting the next word" skepticism is warranted but misses something crucial: the model isn't doing lookup or interpolation. It's building internal representations that generalize. When you've built distributed systems, you know that simple local rules can produce complex emergent behavior—consensus protocols, eventual consistency, self-organizing networks. This is similar. The training process forces the model to develop what you might call "compressed world models" because that's the most parameter-efficient way to predict diverse text. At inference time, you're essentially doing a forward pass through a learned function, autoregressively sampling from the output distribution. The "intelligence" isn't magic—it's the result of gradient descent finding representations that capture statistical regularities in human-generated text, including the regularities we'd call reasoning.

What makes this different from a glorified Markov chain or n-gram model is the depth and the attention mechanism. Attention lets the model dynamically route information across the entire context window—it's learning which tokens are relevant to which predictions, and those relevance patterns can encode arbitrarily complex relationships. The stacked layers build hierarchical representations: early layers might capture syntax, later layers capture semantics and pragmatics. You can actually probe these representations and find interpretable structure. It's not a black box in the sense that we have no idea what's happening—it's more like a codebase so large that no one fully understands it, but you can inspect modules and trace behavior. The surprising part isn't that it works; it's how far you can push this one simple objective.

---

For the PhD Physicist

Let's be precise about what's actually happening mathematically. A transformer is a parameterized function $f_\theta: \mathbb{R}^{n \times d} \to \mathbb{R}^{|V|}$ mapping a sequence of $n$ token embeddings in $\mathbb{R}^d$ to a probability distribution over a vocabulary $V$. The core mechanism is self-attention: $\text{Attention}(Q,K,V) = \text{softmax}(QK^T/\sqrt{d_k})V$, where $Q$, $K$, $V$ are learned linear projections of the input. Training minimizes cross-entropy loss over a corpus, which is equivalent to maximum likelihood estimation under the autoregressive factorization $P(x_1,...,x_n) = \prod_i P(x_i|x_{<i})$. You're right that this is "just" linear algebra and nonlinear activations—there's no exotic physics here. The architecture is fully differentiable, trained with standard SGD variants.

What's genuinely novel, and not just marketing, is the empirical discovery of scaling laws: test loss follows a power law in compute, parameters, and data across many orders of magnitude—$L(C) \propto C^{-\alpha}$ with $\alpha \approx 0.05$ for compute. This is a robust empirical regularity that held predictively as models scaled from millions to trillions of parameters. The theoretical explanation is incomplete, but the phenomenon is real and was not obvious a priori. There's also the emergence of capabilities at scale—models below a certain size fail completely at certain tasks, then succeed above a threshold. This isn't just quantitative improvement; it's qualitative phase transitions in capability. Whether you find this impressive depends on your priors, but it's not something you'd have predicted from the architecture alone.

The honest answer to "what's actually going on" is that we don't fully know. The function class is expressive enough to represent complex computations—transformers are Turing complete given unbounded depth—but we don't have tight characterizations of what representations emerge or why. There's active work on mechanistic interpretability, which is essentially reverse-engineering the learned circuits. Some findings: models learn interpretable features (syntax trees, semantic categories, even some causal reasoning patterns), but the representations are superposed and distributed. The field is pre-paradigmatic in the Kuhnian sense—we have powerful empirical tools without a satisfying theoretical framework. If you're looking for the physics-style "here's the Hamiltonian, here are the symmetries," it doesn't exist yet. But the engineering results are not hype—these systems do things that would have been dismissed as impossible ten years ago, and the scaling trends suggest we haven't hit fundamental limits.

---

For the Venture Capitalist

The core technology is actually straightforward to explain: these models learn to predict text by training on massive datasets, and the "intelligence" emerges from doing that prediction really well at enormous scale. What matters for your diligence is understanding where the moats are—and where they aren't. The transformer architecture is public, the training algorithms are well-known, and the basic approach is commoditized. If a startup tells you their secret sauce is "a better architecture" or "proprietary training techniques," be skeptical unless they can point to benchmark results that can't be explained by just spending more on compute. The real defensibility in this space comes from three places: proprietary data, distribution/product lock-in, and specialized fine-tuning for domains where incumbents can't easily follow.

Data is the most underrated moat. Frontier models are trained on most of the public internet, so everyone has access to roughly the same pretraining corpus. But fine-tuning on proprietary data—customer interactions, domain-specific documents, feedback loops from production usage—creates compounding advantages. Ask founders: where does your training data come from, and why can't OpenAI or Anthropic replicate it? The second moat is product integration. Once an AI system is embedded in a workflow and accumulating user feedback, switching costs increase. The model gets better from usage data, users build habits around it, and you've got a flywheel. The third is domain specialization—a medical AI startup with FDA clearance and clinical validation data has real barriers that a foundation model lab won't easily cross.

Red flags to watch for: founders who can't clearly explain what they're building on top of vs. what they're building themselves, claims of "AGI" or "breakthrough" capabilities without reproducible benchmarks, and business models that assume foundation model APIs will stay expensive forever. Inference costs are dropping fast—what's expensive today may be cheap in 18 months. Also pressure-test the team: do they have people who've actually trained large models, or are they just API wrappers with nice UX? The latter can be a real business, but it's a different risk profile. The technology is real and transformative, but the value capture question is genuinely open. The best opportunities are probably in vertical applications where domain expertise and data create barriers, not in trying to out-scale the foundation model labs.