How to Implement Functional Components of Transformer and Mini-GPT Model from Scratch Using Tinygrad

Building Neural Networks from Scratch with Tinygrad

This tutorial details building neural networks from scratch using Tinygrad, focusing on tensors, autograd, attention mechanisms, and transformer architectures. The process culminates in a working mini-GPT model, demonstrating how Tinygrad’s simplicity aids understanding of model training, optimization, and kernel fusion.

We progressively build every component ourselves, from basic tensor operations to multi-head attention, transformer blocks, and, finally, a working mini-GPT model. Through each stage, we observe how Tinygrad’s simplicity helps us understand what happens under the hood when models train, optimize, and fuse kernels for performance.

Key Insights

• Lazy Evaluation in Tinygrad: Operations are only computed when .realize() is called, enabling kernel fusion for performance.
• Custom Operations: Tinygrad allows defining custom activation functions and automatically computes gradients.
• Mini-GPT Architecture: The implemented model achieves a functional mini-GPT with 18,816 parameters.

Working Example

import subprocess, sys, os
print("Installing dependencies...")
subprocess.check_call(["apt-get", "install", "-qq", "clang"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
subprocess.check_call([sys.executable, "-m", "pip", "install", "-q", "git+https://github.com/tinygrad/tinygrad.git"])
import numpy as np
from tinygrad import Tensor, nn, Device
from tinygrad.nn import optim
import time
print(f"🚀 Using device: {Device.DEFAULT}")
print("=" * 60)
print("\n📚 PART 1: Tensor Operations & Autograd")
print("-" * 60)
x = Tensor([[1.0, 2.0], [3.0, 4.0]], requires_grad=True)
y = Tensor([[2.0, 0.0], [1.0, 2.0]], requires_grad=True)
z = (x @ y).sum() + (x ** 2).mean()
z.backward()
print(f"x:\n{x.numpy()}")
print(f"y:\n{y.numpy()}")
print(f"z (scalar): {z.numpy()}")
print(f"∂z/∂x:\n{x.grad.numpy()}")
print(f"∂z/∂y:\n{y.grad.numpy()}")

Practical Applications

• Research: Experimenting with novel neural network architectures without relying on large frameworks.
• Pitfall: Ignoring the computational graph can lead to unexpected performance bottlenecks; understanding lazy evaluation is crucial.

References:

• https://www.marktechpost.com/2025/11/25/how-to-implement-functional-components-of-transformer-and-mini-gpt-model-from-scratch-using-tinygrad-to-understand-deep-learning-internals/