developing-gpt2-124M-from-scratch

📜 Inspired by Key Research Papers: Attention Is All You Need (Vaswani et al., 2017) Language Models are Unsupervised Multitask Learners (GPT-2 paper) (Radford et al., 2019) Language Models are Few-Shot Learners (GPT-3 paper) (Brown et al., 2020) Thanks to Andrej Karpathy’s clear and insightful explanations in his Neural Networks series. 📂 Dataset: FineWeb 10B Tokens For training our custom GPT-2 model, we utilized the FineWeb dataset with 10 billion tokens. This dataset underwent rigorous filtering to ensure high-quality data selection. Filtering steps included the removal of low-quality web content, duplicate text elimination, and language detection to maintain a clean and diverse corpus. As a result, the model learned from a balanced and representative sample of internet text, enhancing its performance on downstream tasks. 🏋 Training and Evaluaton Details GPU: A100 Training Time: 23 hours per epoch. Training Log : you will find it at src/log To evaluate model performance, we compared our results against GPT-2 and GPT-3 baselines across key metrics like: HellaSwag accuracy. Training Loss(Cross-Entropy Loss): Our model achieved a minimum training loss of 2.7826 after one epoch.