SET 1 - MAKEMORE (PART 4) 🔗
Overview
In this repository, we take the 2-layer MLP (with BatchNorm) from the previous video/lecture and backpropagate through it manually without using PyTorch autograd's loss.backward(). So we will be manually backpropagating through the cross entropy loss, 2nd linear layer, tanh, batchnorm, 1st linear layer, and the embedding table.
🗂️Repository Structure
├── .gitignore
├── starter-code.ipynb
├── exercise-1.ipynb
├── exercise-2.ipynb
├── exercise-3.ipynb
├── final-code.ipynb
├── README.md
└── names.txt
- Jupyter Notebooks: Step-by-step implementation and exploration of the concepts.
- README.md: Overview and guide for this repository.
- names.txt: Supplementary data file used in training the model.
📄Instructions
To get the best understanding:
The format and structure of this particular section of the project will be different from what I've implemented so far, as Andrej Karpathy himself had quoted- "I recommend you work through the exercise yourself but work with it in tandem and whenever you are stuck unpause the video and see me give away the answer. This video is not super intended to be simply watched."
So keeping this in mind, we will be focusing more on the notebook itself and only making notes whenever absolutely necessary.
You will find my notes/key points as comments in the code cells (Apart from the time stamps with necessary headers which will be in their normal format ofcourse)
⭐Documentation
For a better reading experience and detailed notes, visit my Road to GPT Documentation Site.
💡Pro Tip: This site provides an interactive and visually rich explanation of the notes and code. It is highly recommended you view this project from there.
✍🏻Acknowledgments
Notes and implementations inspired by the Makemore - Part 4 video by Andrej Karpathy.
For more of my projects, visit my Portfolio Site.