2026Author

Forge

Efficient fine-tuning methods like LoRA and DoRA get used as one-line library calls, which means almost nobody who runs them can tell you what they actually do to the weights.

The result

LoRA, DoRA, and QLoRA-style int8 quantization all rebuilt from the math in NumPy, with hand-written and gradient-checked backward passes, adapter hot-swap, and a base model that is provably left untouched.

Forge is a from-scratch implementation of modern parameter-efficient fine-tuning, built in Python with NumPy. It covers LoRA, DoRA (the 2024 weight-decomposed variant), and QLoRA-style int8 quantization, each reconstructed from the paper math rather than wrapped around an existing library. Adapters hot-swap on and off a shared base model, perplexity evaluation measures the effect, and the base weights are verified to stay exactly as they were.

Methods: LoRA · DoRA · QLoRA
Base model: Provably frozen
Adapters: Hot-swap
Frameworks: None (NumPy)

Architecture

Frozen base model

provably untouched

LoRA adapter

low-rank update

DoRA adapter

weight-decomposed, 2024

QLoRA int8

quantized base + adapter

Adapters hot-swap on and off the shared base; perplexity evaluation compares the methods on equal footing.

The frozen base model sits at the center and never changes. Each fine-tuning method is an adapter that attaches to it: LoRA as a low-rank update, DoRA as the weight-decomposed 2024 variant, and QLoRA as an int8-quantized base with an adapter on top. Because the adapters are separable, they hot-swap: you can load one, evaluate, unload it, and load another against the same untouched base. Perplexity is the common yardstick that lets the methods be compared on equal footing.

Key decisions and trade-offs

Rebuild LoRA, DoRA, and QLoRA from the math up

Calling a fine-tuning library teaches you the API, not the method. Deriving each one from its paper (the low-rank decomposition, DoRA's split into magnitude and direction, the int8 quantize-dequantize path) and hand-writing plus gradient-checking the backward passes meant I had to understand exactly where each method spends its parameter budget and why. That is the difference between using these techniques and being able to reason about when they break.

Prove the frozen base is untouched, do not just assume it

The core promise of adapter tuning is that the base model stays intact and only the adapter carries the change. Forge verifies that promise instead of trusting it, so a bug that silently leaks gradients into the base is caught rather than shipped. It is a small check that guards the entire premise.

Adapter hot-swap as a first-class feature

Making adapters load and unload against a shared base keeps the base in memory once and turns method comparison into a swap rather than a reload. It mirrors how adapters are served in practice and makes the perplexity comparison across LoRA, DoRA, and QLoRA cheap to run.

Results