Torchlet Implementation Notes
Torchlet preserves one narrow LLM inference path as a sequence of named implementation Versions. These notes make that sequence explicit: each page states what changed, why the change exists, which principle it demonstrates, and which files are worth comparing.
Implementation Versions
Each Version keeps the preceding implementation visible, so design pressures and structural changes can be examined directly.
v00_full_recompute
Single-request Qwen2.5 baseline
implementedv01_0_ragged_batch
Batch variable-length requests with req_indptr
implementedv01_1_split_gqa
Make attention phases readable before adding cache
implementedv02_kv_cache
Separate prefill and decode to stop recomputing history
implementedv03_request_states
Track waiting, running, and finished requests
implementedv04_continuous_batching
Admit and retire requests while decoding continues
implementedv05_decode_slots
Give decode requests stable slot identities
implementedv06_static_buffers
Prepare fixed decode inputs for graph capture
implementedv07_cuda_graph
Capture and replay the static decode path
implementedv08_paged_gqa_py
Use a block pool and block table for KV cache
implementedv09_triton_basics
Introduce a small custom kernel before paged attention
implementedv10_triton_paged_gqa
Move paged attention to Triton and replay the fixed decode path
Reader Model
- The reader knows Python and basic PyTorch.
- The reader has seen Transformer attention, but may not know serving internals.
- The reader prefers comparing small code Versions to reading a monolithic final implementation.
Documentation Shape
- Start with the version map to understand the sequence.
- Read one version page at a time.
- Open the code compare page when a page names a file to inspect.
- Compare adjacent versions first, then compare early and late versions to see the accumulated design pressure.
Page Contract
Every Version page follows the same documentation contract:
- What it introduces.
- Why the previous version needs it.
- The core principle.
- The important files to compare.
- The main tradeoff intentionally left visible.
Build and Deploy
The site is generated by tools/build_docs.py using only the Python standard library. It writes static files to docs/_site, and the GitHub Pages workflow uploads that directory.