From the First Token

In early 2023, I gave GPT a serious try for the first time. I had heard about it, but the descriptions had not made much of an impression on me. Then I asked it to write some PyTorch code. I remember being surprised by what appeared on the screen. I did not even run that first snippet. Seeing it write familiar code from a short conversation was enough to make me stop and pay attention.

By then, I had been researching deep learning algorithms for four or five years, often working with PyTorch and Transformer-based methods. I continued to follow new models and papers, and later built agent workflows in my day-to-day work. Those experiences kept bringing me back to one question: how did all the pieces inside a language model actually work together? Using a model, reading its architecture and tracing an implementation each reveal different parts of the answer. I wanted to connect them.

That led me back through the whole pipeline. I took each step apart, worked through a small example, wrote and ran the code, and checked the intermediate results. Then I put the pieces together and trained a small model from scratch. Each run gave me something concrete to examine: a shape I had misunderstood, a loss that needed explaining, or a result that did not support the conclusion I had expected. Writing those explanations down became this book.

For me, learning by doing means giving an idea a calculation and an implementation that I can inspect. The small examples matter because they make it possible to follow every value. Connecting those examples matters because a working system creates questions that isolated formulas leave open. I hope the route in these pages gives you room to try things, make mistakes, and work out why a correction helps.

You may be building your first language model, or revisiting methods you already use. Either way, take your time with the examples. Change a value, predict what should happen, and then run the code. I hope you enjoy the process, and that the next model or paper you encounter becomes a little easier to reason about.

2023年年初，我第一次认真试用了 GPT。此前听说过它，但那些介绍并没有让我特别在意。后来，我让它写了一段 PyTorch 代码。看到屏幕上出现的内容时，我确实有些惊讶。我当时甚至没有把那段代码运行起来。仅仅是看到它通过几句对话，就能写出自己熟悉的代码，已经足以让我停下来，认真关注它。

那时，我已经从事深度学习（deep learning）算法研究四五年，研究中也常用 PyTorch 和 Transformer 相关方法。此后，我一路关注模型和论文的进展，也在工作中搭建智能体工作流（agent workflows）。这些经历让我越来越想把一个问题弄清楚：语言模型内部的各个部分，究竟是怎样一起工作的？使用模型、阅读架构和追踪具体实现，各自能让人看到一部分答案。我希望把它们贯通起来。

于是，我重新梳理了整条流程。每一步都先拆开，走完一个小算例，再写代码、运行代码、检查中间结果；然后把这些部分接起来，从头训练一个小模型。每次运行都会留下可以具体追查的问题：某个形状是不是理解错了，损失为什么这样变化，某个结果是否真的支持原先的判断。把这些解释逐步整理下来，就成了这本书。

对我来说，边做边学（learning by doing）意味着给一个想法配上能够检查的计算和实现。小算例的价值，在于可以追清每一个数值；把算例接成系统，则会带出单看公式时不容易遇到的问题。我希望书中的路线能给你留出动手尝试、犯错，以及弄清一次修正为什么有效的空间。

你可能正在搭建自己的第一个语言模型，也可能是在重新梳理已经使用过的方法。无论从哪里开始，都可以在算例上多花一点时间：改一个数，先想想结果应该怎样变化，再运行代码。我希望你能享受这个过程，也希望你下次遇到一个新模型或一篇新论文时，能够更容易地想清楚它在做什么。