[Andrej Karpathy] Let’s build GPT: from scratch, in code, spelled out.
🎯 Загружено автоматически через бота:
🚫 Оригинал видео:
📺 Данное видео принадлежит каналу «Andrej Karpathy» (@AndrejKarpathy). Оно представлено в нашем сообществе исключительно в информационных, научных, образовательных или культурных целях. Наше сообщество не утверждает никаких прав на данное видео. Пожалуйста, поддержите автора, посетив его оригинальный канал.
✉️ Если у вас есть претензии к авторским правам на данное видео, пожалуйста, свяжитесь с нами по почте support@, и мы немедленно удалим его.
📃 Оригинальное описание:
We build a Generatively Pretrained Transformer (GPT), following the paper “Attention is All You Need“ and OpenAI’s GPT-2 / GPT-3. We talk about connections to ChatGPT, which has taken the world by storm. We watch GitHub Copilot, itself a GPT, help us write a GPT (meta :D!) . I recommend people watch the earlier makemore videos to get comfortable with the autoregressive language modeling framework and basics of tensors and PyTorch nn, which we take for granted in this video.
Links:
Google colab for the video:
GitHub repo for the video:
Playlist of the whole Zero to Hero series so far:
nanoGPT repo:
my website:
my twitter:
our Discord channel:
Supplementary links:
Attention is All You Need paper:
OpenAI GPT-3 paper:
OpenAI ChatGPT blog post:
The GPU I’m training the model on is from Lambda GPU Cloud, I think the best and easiest way to spin up an on-demand GPU instance in the cloud that you can ssh to: . If you prefer to work in notebooks, I think the easiest path today is Google Colab.
Suggested exercises:
EX1: The n-dimensional tensor mastery challenge: Combine the `Head` and `MultiHeadAttention` into one class that processes all the heads in parallel, treating the heads as another batch dimension (answer is in nanoGPT).
EX2: Train the GPT on your own dataset of choice! What other data could be fun to blabber on about? (A fun advanced suggestion if you like: train a GPT to do addition of two numbers, i.e. a b=c. You may find it helpful to predict the digits of c in reverse order, as the typical addition algorithm (that you’re hoping it learns) would proceed right to left too. You may want to modify the data loader to simply serve random problems and skip the generation of , . You may want to mask out the loss at the input positions of a b that just specify the problem using y=-1 in the targets (see CrossEntropyLoss ignore_index). Does your Transformer learn to add? Once you have this, swole doge project: build a calculator clone in GPT, for all of -*/. Not an easy problem. You may need Chain of Thought traces.)
EX3: Find a dataset that is very large, so large that you can’t see a gap between train and val loss. Pretrain the transformer on this data, then initialize with that model and finetune it on tiny shakespeare with a smaller number of steps and lower learning rate. Can you obtain a lower validation loss by the use of pretraining?
EX4: Read some transformer papers and implement one additional feature or change that people seem to use. Does it improve the performance of your GPT?
Chapters:
intro: ChatGPT, Transformers, nanoGPT, Shakespeare
baseline language modeling, code setup
reading and exploring the data
tokenization, train/val split
data loader: batches of chunks of data
simplest baseline: bigram language model, loss, generation
training the bigram model
port our code to a script
Building the “self-attention“
version 1: averaging past context with for loops, the weakest form of aggregation
the trick in self-attention: matrix multiply as weighted aggregation
version 2: using matrix multiply
version 3: adding softmax
minor code cleanup
positional encoding
THE CRUX OF THE VIDEO: version 4: self-attention
1 view
0
1
2 months ago 04:01:25 13
[Andrej Karpathy] Let’s reproduce GPT-2 (124M)
2 months ago 00:27:13 6
But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning
2 months ago 00:44:17 1
No Priors Ep. 80 | With Andrej Karpathy from OpenAI and Tesla
2 months ago 00:18:11 57
НОВОСТИ ИИ: Подписка на ChatGPT за 2000$
2 months ago 00:26:10 1
Attention in transformers, visually explained | Chapter 6, Deep Learning
3 months ago 00:06:53 1
Elon Musk says losers use LiDAR. [Explanation video]
4 months ago 00:47:27 1
Projeto Secreto da OpenAI: Descubra as Últimas Inovações da IA e Fique Super Atualizado no IA News#6
4 months ago 00:07:41 1
Educação 100% com IA, FBI invade celular, Hype das IA no fim, e muito mais
6 months ago 00:40:08 1
The Most Important Algorithm in Machine Learning
6 months ago 00:08:29 1
CATL’s sodium hybrid battery will be 30% cheaper & revolutionise the world
6 months ago 00:08:55 1
Tesla reveals timelline for massive electric Semi production at $ factory
8 months ago 00:26:53 1
Vedal & Neuro Build A Language Model From Scratch
8 months ago 00:03:19 2
How To Study Hard - Richard Feynman
9 months ago 00:16:39 1
Phi-1: A ’Textbook’ Model
9 months ago 00:20:13 1
GPT-5: Everything You Need to Know So Far
11 months ago 00:14:07 1
“Что в имени тебе моем?“ Учимся генерировать новые имена у звездного разработчика Tesla и OpenAI.
11 months ago 00:59:48 21
Введение в большие языковые модели от Andrej Karpathy
12 months ago 00:03:51 1
ČENDEŠ - Karpaty, Karpaty [OFFICIAL 2017 4K]
1 year ago 00:30:09 1
Andrej Karpathy - AI for Full-Self Driving at Tesla
1 year ago 01:03:42 3
[SafeCode Live] ML в AppSec
1 year ago 00:05:48 3
Advice for machine learning beginners | Andrej Karpathy and Lex Fridman
1 year ago 03:28:47 25
Andrej Karpathy Tesla AI, Self-Driving, Optimus, Aliens, and AGI Lex Fridman Podcast #333
1 year ago 03:08:46 1
George Hotz: Tiny Corp, Twitter, AI Safety, Self-Driving, GPT, AGI & God | Lex Fridman Podcast #387
2 years ago 05:14:51 1
John Carmack: Doom, Quake, VR, AGI, Programming, Video Games, and Rockets | Lex Fridman Podcast #309