Skip to main content
Selected work

2026Author

Glassbox

Most people who use transformers have never watched one learn, because autograd hides the one part that actually teaches you how the model works: the gradients.

The result

A working transformer in NumPy where every forward and backward pass is hand-derived and gradient-checked to 1e-8, trained and generating text on a plain CPU.

Glassbox is a transformer built from scratch in Python with NumPy and nothing else. It supports both GPT-2 and Llama-style architectures, so RMSNorm, RoPE, SwiGLU, and a KV-cache are all written out by hand. There is no autograd anywhere: every derivative in the backward pass was worked out on paper and then verified numerically.

Gradient check
1e-8
Frameworks
None (NumPy)
Architectures
GPT-2 · Llama
Runs on
CPU

Architecture

Input tokens

Embedding

token + RoPE positions

Transformer blocks

RMSNorm, attention, SwiGLU, KV-cache

Output logits

Cross-entropy loss

Hand-derived backward pass

Gradient check

matches finite differences to 1e-8

Weight update

The weight update feeds back into the blocks each training step, so the same code both trains and generates.

The forward half is the standard transformer flow: tokens become embeddings with RoPE positions, pass through stacked blocks (RMSNorm, attention, SwiGLU, and a KV-cache for generation), and come out as logits scored against a cross-entropy loss. The interesting half is the return trip. Instead of calling a framework's backward method, I derived the gradient of every operation by hand and implemented each one, then checked each against a finite-difference estimate. Only once the analytic and numeric gradients agreed to 1e-8 did a layer count as done.

Key decisions and trade-offs

Hand-derive backprop instead of leaning on autograd

Autograd is the right tool in production, but it also means you can ship a model without ever understanding why a gradient vanishes or explodes. Deriving every backward pass myself, then gradient-checking it to 1e-8, turned the chain rule from something I trusted into something I can reproduce from memory. That is the whole point of the project: the box is glass.

One codebase, two architectures

Supporting both GPT-2 and Llama-style stacks (RMSNorm and RoPE and SwiGLU next to the older LayerNorm and learned-position design) forced the components to be genuinely modular rather than a single hardcoded model. It also made the differences between the two families concrete instead of abstract.

NumPy only, CPU only

No deep-learning framework, no GPU, no CUDA to hide behind. The constraint keeps the code readable and portable, and it proves the ideas do not need an accelerator to be correct. The trade-off is speed, which is fine for a project whose goal is understanding, not throughput.

Results

  • Every forward and backward pass gradient-checked to 1e-8 against finite differences.
  • Supports GPT-2 and Llama-style architectures (RMSNorm, RoPE, SwiGLU, KV-cache).
  • Trains and generates text end to end on a CPU.

Inside the repo

The source is public, so here is a map before you open it: what each core file does and how they fit together. Everything below is taken straight from the modules themselves. Read the source.

glassbox/

  • tensor_ops.pyThe primitives every layer is built from - matmul, softmax, LayerNorm, RMSNorm, GELU, SwiGLU, cross-entropy - each with a forward pass and a hand-written backward pass beside it.
  • attention.pySelf-attention itself, the mechanism that makes a transformer a transformer, forward and backward by hand.
  • rope.pyRotary position embeddings, the position encoding used by Llama, Mistral, and essentially every current open model.
  • model.pyAssembles the verified parts into a full GPT-style decoder: tokens to embeddings to blocks to logits.
  • optimizer.pyAdam written out by hand, so the update rule is visible rather than imported.
  • gradcheck.pyThe honesty capstone: every hand-derived gradient is checked against a numerical gradient, so a wrong derivation fails loudly instead of quietly training worse.
  • interpret.pyMakes the internals visible - renders what the model actually attends to.
  • data.pyA character-level dataset plus the training loop.
  • demo.pyThe whole story in one command: proves the gradients, trains, then generates text.
  • web.pyA local dashboard for poking at the model in a browser.

How it fits together

Read it bottom-up, the same order it was built: tensor_ops.py holds the primitives, attention.py and rope.py use those primitives to build the parts unique to a transformer, and model.py stacks them into a working decoder. optimizer.py trains it and data.py feeds it. gradcheck.py is the file to read first if you are checking whether any of it is real - it is what proves the hand-derived backward passes match numerical gradients. demo.py runs the whole chain end to end.

Tests
39 test functions in tests/test_glassbox.py
Stack
Python and NumPy, no other dependencies

Repository