01 - Building Micrograd

so this is my journey of understanding what an autograd is, understadning mathematics behind back propagationa and implementing it from scratch. might have a few rants, so keep it up for me

Half an hour into reading karpathy's repo I had the though : "why build this when PyTorch already exists?" PyTorch is fast, works on tensors and a one-line .backward() that runs on CUDA and you're done. But that one line is also a black box. A micrograd rips that black box open gives a detailed undersdtanding of what's happening behind the mask.

Quick comparison

• PyTorch: Optimized C++/CUDA backend, tensor/array operations, automatic vectorized gradients, production-ready performance. You call .backward() and the library handles everything for you.
• Micrograd / tiny autograd: Plain Python implementation, node-by-node and number-by-number gradient propagation, extremely slow but crystal-clear about every step of backpropagation, ideal for learning the mechanics under the hood.

Backpropagation - in a nutshell

At training time we measure a loss L (how wrong the model is). Backpropagation computes the partial derivatives of that loss with respect to each parameter (weights and biases) using the chain rule. For a single neuron with z = w * x + b and output y = f(z), the gradient of the loss w.r.t the weight is dL/dw = dL/dy * dy/dz * dz/dw (and dz/dw = x).

Once we have the gradient, parameters are updated with a simple gradient-descent step: w_new = w - η * dL/dw and b_new = b - η * dL/db, where η is the learning rate, a small positive scalar that controls step size. Too large and training diverges; too small and learning is painfully slow.

ReLU (very short)

ReLU is the activation function ReLU(x) = max(0, x). Its derivative is 1 for positive inputs and 0 for negative inputs, which makes it simple and effective in practice.

So… this post never got finished

Fair. Life happened. Right after I started writing this I got an on-campus internship, went into a pretty chill mood, and then immediately landed a full-time role at a US-based startup. That energy just went elsewhere.

I did come back to micrograd eventually. Speedran it, tbh. Understood everything, implemented everything, the whole thing. Just got too lazy to write about it here. The learnings are there, they just live in my head and the repo now. Maybe I'll circle back when I feel like it, or maybe this post stays as the incomplete artifact it is. Either way, micrograd is done. Just the writing wasn't.