Lecture 9: Creating Input-Target data pairs using Python DataLoader

[Music] hello everyone welcome to this lecture in the build large language models from scratch Series in the previous lecture we took a look at bite pair encoding and uh we saw that how bite pair encoding algorithm can be used for something which is called as subword tokenization so we saw the difference between word based subword based and character based tokenization and we looked in detail how GPT models such as GPT 2 3 and 4 use the bite pair encoding algorithm for tokenization if you have not seen the video for the previous lecture again I would

highly ENC encourage you to go through this so that you will follow along pretty well in this lecture if you are coming to this playlist for the first time welcome and uh we follow a very specific style in this playlist where we do a mix of writing on the White board plus showing you everything from scratch in the jupyter notebook code editor so that the theoretical understanding is also strong and the coding background is also strong up till now we have looked at tokenization which is needed for large language models so if you think of

the whole process we are currently at the data pre-processing stage before the data is given for the llm training in the pre-processing the first step is tokenization then we come to something called Vector embeddings we have not seen Vector embeddings yet uh and then after that we feed these Vector embeddings to the uh training or for the training process before we come to Vector embeddings there is one very important lecture which we need to cover and that is the topic of today's lecture creating input Target pairs essentially input output pairs if you look at other

machine learning tasks such as classification ation it's usually usually very clear right what is the input and what is the output if you want to distinguish between cats and dogs from images the images of cats and the images of dogs will be input and whether it's a cat or whether it's a dog will be the output if you consider a regression problem on the other hand let's say if you want to predict the price of a house B based on its area the area of the houses is the input and the price is the output

so creating the input output pairs or the input Target pairs is pretty easy for large language models we use a specific technique for creating these pairs and it's very important to devote a separate lecture for you to understand this so let's get started with today's lecture as I mentioned before now only one last step is remaining before we move to creating Vector embeddings which will then be fed to training the large language model and then last and then that last step is essentially create getting the input Target pairs so first when I say input Target

pairs what do I mean and what do input Target pairs looks like so let's say uh this is my uh sentence right which is the text sample llms learn to predict one word at a time so the blocks which are marked in blue will be the input to the llm and the block which are marked in red will be the Target or the output which the llms have to learn and why are there these different rows so these are different iterations let's look at the first iteration in the first iteration the input is llm and

the based on this input the out uh llm has to learn the output which is the learn so the next word is always the output whatever comes after the prediction is masked or it's not shown to the llm this is what happens in iteration number one now let's look at iteration number two so learn which was the output or the Target in the first iteration now is a part of the input so in the second iteration llms learn that is the input and two is the target that's the target pair that's the second iteration in

the third iteration two which was the output of the previous iteration now becomes the input so so llms learn to is the input in the third iteration and predict is the output I hope you have started understanding the pattern now in every iteration there is only the next word which is the output and whatever comes before that is the input these are the input Target pairs that's very important to remember so uh here also you'll see that llms learn to predict is the input and one is the output so at every stage of the iteration

process uh llms have input which is the part of the sentence up till the word which needs to be predicted and the word which needs to be predicted that is essentially the output this this figure which I'm saying is just for illustration purposes in today's lecture we'll learn something about context length which means how many words are given as the input the output length is always one one word will be predicted but we can essentially choose the input context length now uh in every iteration The Words which are after the target are essentially masked so

the llms cannot access The Words which are past the target so there are two things to remember here the first thing to remember is that within the sentence itself we break down the sentence into input and a Target which is the next word uh then in the second thing to remember is that in subsequent iterations whatever was the output in the previous iteration then becomes the input so this is a auto regressive model why Auto regressive because the output of the first iteration becomes an input of the next iteration like let's look at these two

iterations in in this iteration let me show it with a different color so that it becomes easy in this iteration one the result one was an output right but see in this iteration one is now a part of the input and then the next word is the output so it's called an auto regressive and it's also called a self-supervised learning or you can think of it as unsupervised learning itself because we are not labeling the input and the output the sentence structure itself uh is used to predict or is used to determine what is the

input and the output we do not have to do any special labeling so in cats and dogs we have to manually label this is a cat this is a dog right for the image classification but here to create the input Target pairs we don't have to say that look this label this as the input label this as the output we'll just write a simple code which utilizes the sentence structure itself and breaks down the sentence into input and the output so this is also an example of unsupervised learning and it's also called Auto regressive I

hope you have understood these two concepts so in pre-training we always do unsupervised learning because the sentence structure is exploited to create input output pairs or input Target pairs so I hope you have understood how the the input Target pairs look like and we are going to create this in today's lecture in Python uh if you understand up till this it's actually pretty easy to code it out in Python but I have I feel that students don't really understand this part intuitively and and hence they find the coding part of it a bit difficult okay

now I want to mention a few things uh which I've just which just serve as a summary of what all I explained up till now the first thing is what we are essentially doing here is that we are given a text sample and uh based on the text sample we are extracting input blocks that serve as the input to the llm correct and the llm prediction task during the training is to predict the next word that follows the input block so for example if you're looking at this input block the llm task is to predict

the output or the next word based on this input uh and that's what the llm is trained for and the last point to remember is that during the training process we will mask out all the words that are past the target so in every iteration the target is the target word right like in this iteration time or let me take an earlier iteration in this iteration two is the target so when we are doing this iteration the llm does not see anything which comes after two so this part is essentially masked and we'll see how

to implement all of these features in code Okay so until now I just wanted to explain what is the purpose and what is the aim of today's lecture and now we are going to code the input Target pairs in Python so I hope you are ready for this coding so let's get started with coding great so this coding section I've have titled creating input Target pairs as always I'll be sharing this Jupiter notebook code also with you along with the video so that you can run the code and check check whether you have understood the

concept or not yourself so in this section we are going to implement a data loader that fetches the input Target pairs using a sliding window approach so there are two parts of this sentence which might be confusing to you what is data loader that's part number one and what is the sliding window approach that's part number two don't worry I'll explain to you both of these in a lot of detail uh to get started what we will initi do is that we'll take the whole the verdict short story so remember our data set for this

entire coding Journey for this entire playlist is this short story The Verdict let me show you uh how it actually looks like so this is the short story called The Verdict this is the data set which we have been using I think this was published in uh so let me check the verdict edit won it was published in 1908 and we are using using this as the data set it's a toy data set but it's important because whatever we learn right now it scales exactly the same way for larger data sets as well so we

are going to use this data set and remember in the last lecture we looked at the bite pair encoding tokenizer we are going to encode this entire text using the bite pair encoding tokenizer it's a subword tokenizer so the tokens Can Be characters the token can be words the tokens can be subwords as well so if you if you are not familiar with the bite pair encoder please look at the previous lecture which we have covered great so we have uh defined the tokenizer already which is the bite pair encoder tokenizer and what we'll be

first doing is we will read the entire data set and store it in a variable called raw text and then we will encode the entire raw text remember what an encoder does is takes this text and converts it into token IDs and let us actually run this right now so I ran this right now and you will see that uh I've have printed out the length of the encoded text which means it's 5145 that means the vocabulary size which we have is 5145 what does a vocabulary mean well we covered this in the previous lecture

but let me show it to you again a vocabulary essentially looks something like this uh yeah so this is how let me go to the yeah this is how a vocabulary looks like like essentially we'll have different tokens and to every token a token ID will be uh attached so vocabulary is essentially dictionary which maps The Tokens into token IDs remember since we are using the bite pair encoder the tokens won't be words but they can be subwords or characters also so essentially what this size 5145 conveys is that our vocabulary for the for this

text which we have as the data set has the length of 514 5 which means we have 5145 tokens and corresponding token IDs great so uh I have just written this in blue executing the code above will return 5145 that is the total number of tokens in the training set after applying the bite PA encoding tokenizer great so what I'm going to demonstrate right now to you is just I'm going to look at the first uh so what I'm going to do as I'm going to remove the first 50 tokens for the from the data

set just so that the demonstration becomes a bit better uh after you remove the initial 50 tokens it results in a slightly more interesting text passage you can keep the entire tokens as well just to make the lecture more interesting what I'm going to do here is that the encoded tokens were in ENC or encoded unders scroll text so I'm going to Define one more variable called called encoded undor sample which just removes the first 50 tokens from the data set great now uh first I what I want you all to do is I want

you to pause here for a moment and think about this question yourself uh think about the question that let's say you are given this data set right now and you you'll I hope you understood this input output Target pairs which I mentioned what's the simplest thing which comes to your mind how can you convert this data set into such kind of uh input output Target pairs what will you need to make this conversion can you think about it a bit think about the simplest way don't think about complex algorithms anything like that what is the

simplest thing which comes to your mind you can pause the video here for some time because if you answer it it will really improve your understanding so let me reveal the answer now one of the easiest and most intuitive ways to create the input Ty Target pairs for the next word prediction task is to create two variables X and Y where X contains the input tokens and Y contains the targets which are essentially the input shifted by one so let me explain to you how this logic works okay so what do we exactly need here

let's say if uh let's say if my input is 1 2 3 and 4 let's say say if my input is this I want my output array let's say if this is my input array X I want my output array to be looking something like 2 3 4 and 5 so what I have done here exactly is that if one is the input two should be the output it's very similar here if llm is the input learn should be the output correct if one and two is the input then three should be the output which

means that if llm learn is the input then two should be the output if 1 2 3 are the input which means that if llms learn two is the input then the output should be four which means that predict should be the output and then finally if 1 2 3 4 if all of these four words are the input then five should be the output which means that if llm learn to predict is the input then the output should be equal to one this is what I actually want to create and how do I determine

the size of this uh why do why did I take the input size X to be four and the output size to be four so basically that is called as the context size the context size is how many words do you want to give as input for the model to be making its prediction so here the context size is equal to four right so if we give up to four four words the model will be able to predict the next word that is what the context size actually means so we want to create input output

arrays like this so let me show you how those can be created for this data set of the verdict which we have seen okay so first we have to determine the context size as I told you the context size determines how many tokens are included in the input so let me explain the context size a bit more here currently we are choosing context size of four you can choose anything which you want and you can play around with this code when I share it with you so the context size of four means that the model

is trained to look at a sequence of four words or tokens to predict the next word in the sequence so the input X is the first four tokens let's say 1 2 3 4 and the target Y is the next four tokens which is 2 3 4 5 so that is meant by context size so the input if the input is 1 2 3 4 the output is 2 3 4 5 what does it mean if the input is one the output will be two if the input is 1 two the output will be three

if the input is 1 2 3 the output will be four if the input is 1 2 3 4 the output will be five but the input cannot be one 2 3 4 5 because then the context size would be exceeded to think of it intuitively the context size is basically how many words the model should pay attention at one time to predict the next word so let's now take a simple thing so we have this encoded sample which contains the token IDs of the encoded data set what I will first do is that I

will first take the four elements which are the first four elements that is my X which is the input and then I'll just shift this x Matrix X array by one and then that will be my Y which is the output so let's print out X so the IDS which are associated with the first four encoded samples are 290 4920 2241 and 287 and then the IDS which are associated with Y which is the output is 4920 2241 287 and 257 what does this mean if the input ID is 290 then the output will be

4920 if the input is 290 and 4920 the output is 2241 if the input is 290 4920 and 2241 the output will be 287 and if the input is 290 4920 2241 and 287 the output will be 257 this is how the input output pairs are actually constructed so uh what we can do now is that processing the inputs along with the targets and remember the targets are just the input shifted by one position we can then create the next word prediction tasks as follows so what I just explained to you I've written this in

code so I have created two uh variables here called context and desired and I'm looping in so the context size is four so this Loop will go from 1 to five so in the when I is equal to 1 which is the first iteration the context will be just the first token ID which is 290 and the desired will be the next token ID which is 4920 awesome when I is equal to 2 then the context will be the first two tokens which is 290 and 4920 and the desired will be the next token which

is 2241 if I is equal to 3 uh then the context would be the first three token IDs which are 290 4920 and 2241 and the output will be or the desired will be the next token which is 287 and if I is equal to 4 then the context will be the first four tokens IDs and the desired will be the next which is 257 so everything on the left of the arrow here refers to the input the large language model would receive uh and the token ID on the right hand side of the arrow

represents the target token ID which the llm is supposed to predict so when we constructed these input output pairs this is what it actually means there are four prediction tasks here it's not one prediction task so when I created this input output pair of X and Y and even here here when I showed you the input output pair of X and Y it's not just one prediction task but there are four prediction tasks which are happening here and these are the four prediction tasks because the context size was four if the context size was eight

there would have been eight prediction tasks in each input output pair so when you look at input output pairs usually regression and classification problem one input output pair corresponds to one prediction task image of a dog needs to be classified as whether it's a cat or a dog but in the case of llms one input output pair corresponds to the number of prediction tasks as set by the context size that is very important now what I'm going to do is that I'm going to take this simple the same code but I'm going to decode it

into text so that you can get a feel of what is exactly happening here so here you can see I've taken the same code but I'm printing the decoded context and I'm printing the decoded desired value so if and is the input established is the output if and established is the input himself is the output if and established himself is the output the next word which is in is the output and if and established himself in is the input then the next word uh that is the output now this is exactly what we started the

lecture with right you remember we started the lecture with this and uh now what we have done is that through code we have just created very simple input output pairs X and Y and we have seen how these pairs can be used to create the input and the output awesome right so this is just the first step of what we need to be doing what we have done so far is we have created the input output pairs that we can turn into use for the llm training so later we are going to do llm training

and so we have created input output pairs now but we need to create them in a much more structured manner we need to create them for the entire data set not just that later we will parallel processing so if we have multiple CPUs and we need to do parallel Computing we need to do Computing in batches so we are going to do this in a very structured Manner and for that what we are going to be doing is we are going to use something called as uh data loader so now there is only one more

task which is remaining before we can look at the vector embeddings in the next lecture and that is implementing an efficient data loader uh that iterates over the input data set and Returns the inputs and targets as P torch tensors So currently we have got the input output arrays right but they are not tensors why do we need tensors because all the optimization procedures which come later we we are going to use py torch and py torch works with tensors so we need input tensors and we need output tensors no need to worry if you

don't know what a tensor is you can just think of it as a two-dimensional array for now or multi-dimensional array no need to worry about this this will not stop you from understanding this lecture so our goal is this we want to implement a data loader which creates two tensors an input tensor which contains the text that the llm sees and the target tensor that includes the targets for the llms to predict that's it basically we have to create something exactly like what I showed you in the code before but we need to create it

in a tensor format and we need to do it in a structured manner so that's why we are going to use something called as data set and data loader so the these are data sets and data loaders which are in Python and here you can just see some examples which have been done for some classification data sets but essentially data sets and data loaders enable you to load or process the data in a much more efficient and compact manner as we'll see right now awesome so now what we are going to do in the next

step is we are going to Implement a data loader and uh for the efficient data loader implementation we will use the pytorch inbuilt data set and data loader classes so these are the data set and data loader classes link which I've shown right now I'll also attach the link in the video description before going into the code further I just want to show you what we expect the data loader to do so that you have a visual understanding I have seen that until you get a visual understanding sending the code becomes very difficult to really

Master but if you know what you want to implement it's really very easy so in this section what are we doing we are implementing a data loader that fetches the input output Target pairs using a sliding window approach let's see what this means so uh here so what we are going to do is that let's look at this sample text in the Heart of the City stood the old library A Relic from a Bagon era let's say this is the kind of sentence and we want to create input output Pairs and we are going to

create input output pairs with a context size of four okay so let's say if the input is in the heart of let's say the input is in the heart of the output tensor will be shifted by one right as we already saw so the output will be the heart of the correct so the first input pair is in the heart of and the first output is the heart of the now in this input output pair there will be four prediction tasks the first is that if the input is in the prediction should be the if

the input is in the the prediction should be heart uh let me switch to a different color if the input is in the heart the prediction should be off and if the input is in the heart of the prediction should be the so uh first of all we have an X which is the input tensor and Y which is the output tensor now let's see what the row of every tensor are representing uh so we collect the inputs in a tensor X so we collect the inputs in a tensor X where each row represents one

input context so let's look at the tensor X if you see each row each row of this is one input context in the Heart of the City stood the so each row represents one uh input context so the first input output pair will be in the heart of and the first output will be the heart of Thee the second input will be the CT stood the and the second output will be CT stood the old so basically what I earlier in this lecture I showed you one input output pair when we look at tensors if

you look at each row each row of the X tensor is an input each row of the Y tensor is the Corr oning output so the 50th row of the X X tensor and the 50th row of the Y tensor will be the 50th input output pair and in each input output pair there are four prediction tasks here as I as I told you because the context size is equal to four so essentially we are doing the same thing what we did in the earlier part of the lecture but we just take the entire text

uh we put it in tensors in the rows of the tensor so we split the text into four words so the first four words are the first row the second four words are the second row and if you look at the output tensor it just the input tensor shifted by one that's it if you look at each row of the output tensor it's just the input tensor row shifted by one so the second tensor y uh the second tensor y contains the corresponding prediction targets next Words which are created by Shifting the input by one

position this is very important for everyone who is watching this lecture to understand uh it all we are doing is next word prediction task so let me again explain this so that I I really want this concept to be understood because it's the heart of everything which we are going to do but let's look at the second row in the second row there will be four prediction tasks the first prediction task is when the input is the the output is City when the input is the city the output is stood when the input is the

city stood the output is the and when the input is the city is stood the the output is the old the output is old so basically each input output pair corresponds to one prediction task and corresponds to four prediction tasks and we are just predicting the next word that's it it's as simple as that and that is exactly all we are actually doing in the code also so now I'm returning back to the code and uh This what I showed you here right now on the Whiteboard uh creating such kind of input tensor and output

tensor is exactly what we are going to do in the code so if you if you have understood this the code will be much easier to understand so we are going to implement a data loader which creates these input and the output tensors and this will be done in four steps the first is tokenizing the entire text because uh we are going to deal with token IDs right now I'm showing you words over here but actually we'll deal with token IDs so we need the encoded text uh then we'll use a sliding window and now

do you understand why is it called sliding window because uh look at this blue look at the blue window and the red window here so the blue is the input and then you slide it by one row slide it by one world and that's the output so each input and output pair is just sliding so you slide the input and then you get the output pair uh and then finally what we'll be doing is that we'll return the total number of rows in the data set and we'll return a single Row from the data set

I'll show what this means so here we are going to define a class um and this class is going to take a data set now uh for this what we are doing is from tor. utils we are importing data set and data loader the data loader will come a later right now we have our goal is to define the data set and the data set cannot just be a bunch of tokens we need to make sure the data set is in input output pairs so what do we do here first we see what are the

arguments which are taken when we create an instance of this class so when we create an instance of this class we basically need to specify four things we need to specify all the text file which we have so this is the data set the txt file then we need to specify the tokenizer so here we are using the bite pair tokenizer then we need to specify the max length the max length is the context size here we are going to use a context size or context length of four and then there is something called as

stride so what stride means we'll come come to that in a moment but right now just know that there are four arguments which this class actually takes now what we'll be doing is that we'll be creating two arrays the first is called input IDs the second is called Target IDs what we'll be doing is that in input IDs uh we are so here you see we are going to uh loop over the entire data set so when I is equal to 1 the input ID the input chunk will have token IDs which are from 0

to 4 and the target will be token IDs from 1 to five which is just shifted by one and then you append this to the input ID's tensor and the target ID's tensor so when I is equal to one what will be appended is the first row of the input and the first row of the output so let me actually rub rub erase this a bit okay so in the code we saw the iterations right so when I is equal to 1 the first row will be appended to the input tensor and the first row

will be appended to the output tensor so if you see the input ID is is the input tensor and this input chunk is being appended that's the first row now when I is equal to 1 we slide over to the next so if I is equal to one we uh we we look at the next chunk so the first chunk will be in the heart of in the heart of the second chunk here is the city stood the that's the third chunk the fourth chunk is old library a so we are going to divide the

entire data set into chunks like this and then we are going to append that to the input tensor and just we are going to slide the chunk by one and then that we are going to up to the output tensor this is how we are creating the input and the output tensors and then we are going to Loop till we reach the end of the data set so length of data set minus the max length why minus max length because the last thing will include the context size and so we don't want to spill over

the data set essentially if you are finding this section a bit hard to understand just remember that all we are doing here is that we have a data set we are chunking the data set so we have in first we have the input chunk and then we have the output chunk actually I think let me explain this using a Hands-On example over here so let's take this this example itself which I have written here right now uh what we are essentially doing here is that we are first going to have an input chunk so two

Implement efficient data loaders that's my first input chunk then what I'm going to do then I'm going to shift it by right shift it to the right by one and then I'll have my output chunk so the output chunk is Implement efficient data loaders V so that's my first input output chunk so the input chunk will be the first row of the tensor the output chunk will be the second row of my tensor and then I'll slide so the way I'll slide will depend on my stride so if my stride is equal to let's say

one it means that my next input will be Implement efficient data loaders and my next output will be sorry my next input will be Implement efficient data loaders V and my next output would be efficient data loaders V collect so at each time the output is just the input shifted by one and stride determines how much we slide so in the example which we took here if you see the in the first input is in the heart of but the second input is the city stood the so the first input is in the heart of

the second is the city stood the which means that the stride is also equal to four so the first first input was this and then we moved and then the because the stride was four the second was the second input was the it stood the if the stride was equal to one the the next input would have been the heart of the city but that is not the second input here which means we have used a stride of four in this example that's why we also need the stride because we need to know how much

to slide when to create the next input output batch I hope you have understood this part this is a bit of a tricky portion so please ask in the comment section if something is unclear over here so until this part we have created the input output tensor but now we have to define a method which is called as get item what this method does is that based on the index which we provide it just Returns the that particular row of the input and that particular row of the output so if the index is equal to

zero this will return the first row of the input tensor and the first row of the output tensor why is this method get item needed because when we create a data loader the data loader will look at this method and then only it will create the input output pairs so if you look at the data loader what uh it needs the data set in this map style or the iterable style right now we are using a map style data set which means we need to have this get item what the get item will do is

that it will actually tell it will tell the data loader uh what what kind of input and Target should we have so here we are clearly saying that if it's based on the index if the index is 50 the input will be the 50th row of the input tensor the target will be the 50th row of the output tensor so this G item is finally what the data loader will be using now let me recap what all we have learned so far so this GPT data set origin class we have just implemented is based on

the pytor data set class it defines how individual rows are fetched from the data set each row consists of number of token IDs assigned to an input chunk tensor this basically means that each row the length of or the number of tokens in each row is equal to the context length so here if you see the number of tokens in each row in the heart of four tokens because the context size is four the target chunk tensor contains the corresponding targets so uh as I've mentioned over here I'll actually upload the links to this data

set data loader and also maybe uh a small tutorial so that uh you know this thing becomes a bit more clearer and I think you should have a stronger understanding of uh this data set and data loader portion because many people just skip over this completely now what we'll be doing since our data set is ready and this format is ready we will feed the data set into the data loader so this is where the data loader comes into the picture and what we'll be doing is we'll be doing the four things we'll initialize the

tokenizer create the data set why because this data set needs this class needs four attributes the the text the tokenizer the max length and the stride then what we'll be doing is that we'll put drop last equal to true because if the last batch it's shorter than the batch size uh it's dropped to prevent loss spikes during training you don't need to know about this too much but just remember that when we do batch processing if the last batch is shorter than the specified batch size uh it is important to prevent loss spikes during the

training process awesome so now we Define this function called create data loader this is very important because this is the one this function will implement the batch processing the parallel processing which we will need uh and that is governed by the batch size but more than that what this function will do is that it will actually create uh it will help us create the input output uh data pairs from the data set which we defined earlier awesome so let's see the arguments which this function takes it of course takes the text file which is the

data set which we have then it takes the batch size which is how many batches how many CPU processes we want to run parallell if you don't specify anything this will be by default four so if the number of threads on your CPU are four or eight you can run those many processes parallely max length is basically equal to the context length so we here I showed you a context length of four uh but when gpt2 or gpt3 those high level llm models are designed they usually Implement a context length of 256 which means they

are so strong that the model can look at 256 words and predict the next word during training then stride is one 28 so stride as I mentioned is when we create input output batches how much we need to skip before we create the next batch a number of workers is also the number of CPU threads which we which we can run simultaneously awesome so the first thing which we do is Define the tokenizer and here we are using the tick token which is the bite pair encoder used by GPT and then we create the data

set so here see we are creating an instance of this GPT data set V1 class which we defined over here uh and here we provide the input text the tokenizer is this tick token which is the bite pair encoding the max length is 256 and the stride is 128 awesome so an instance of the GPT data set one is created and we are calling it data set this data set is then feeded or loaded into the data loader look at this data loader method it's it's it takes the data set as an attribute basically what

is happening in this step is that this data loader will just check this get item method in this in this class and then it will return the input output pairs basically based on what is mentioned in the get item that is exactly how it's going to work and then we just return the data loader which are the input output pairs that's it which is happening over here so uh essentially in this part what we did was we implemented a a data set which is a method or rather I would say it's a library python and

then what we are going to do is that we also implemented the data loader so the data set was initially implemented and we created a class called GPT data set V1 an instance of this class was created and then it was fed to the data loader method what this method essentially did was it accessed the get item and then essentially what it's going to do it's just going to create those input output tensors which we have defined in the GPT data set V1 class why did we do this data loader between because it will really

help us to do parallel processing and it can also uh analyze multiple batches at one time so here I want to explain the difference between batch size and number of workers so there is a difference between batch size and the number of workers batch size is basically the number of uh batches the model processes at once before updating its parameters so to make sure that the model updates its parameters quickly the data is usually chunked into batches so that after analyzing four batches in this case the model will update its parameters rather than going through

the entire data set num workers is different it is basically for parallel processing on different threads of your CPU uh and data loader enables us to do all this if we did not do this then defining the batch size num workers would be very challenging now what we are going to do we are going to test the data loader with a batch size of one and a context size of four this will develop an intu of how the data set V1 class and the create data loader function work together so if you found the previous

part a bit challenging to understand please try to focus on this part where we are going to show the Hands-On implementation of the data set class and the create data loader function so first what we do here we just read the text uh and then we are going to create a data loader and convert the data loader into python iterator to fetch the next entry in the data let me show you what this means so here what we are doing is we are going to create the data loader but with a batch size of one

and a context size of four then what we are going to do we are going to iterate through the uh data loader and we are going to print the first batch and I I want to show you what this batch looks like so I'm just going to print this uh and here you see this is the printed answer so what the first batch basically gave me is the input tensor and the output tensor that's it so this is the input tensor and when you shift it by one you get the output tensor that's it essentially

we did all this to get this input output pair which we had already seen before so if you look at the input output pair it's very similar to um the input output pair which we had looked at the beginning of this lecture so if you remember at the beginning of this lecture we had looked at this input output pair 1 2 3 4 uh and then 2 3 4 5 right what we have essentially obtained right now is exactly similar but through the data loader so it's much more structured and here we can specify a

lot more parameters like batch size maximum length stride so here the stride is equal to one we uh we can even show you the effect of stride so let me actually show you the figure to explain uh what actually changing the stride really looks like so um here is the figure for explaining The Stride in some detail yeah okay so if you look at this figure okay so let's look at this figure the first part of this figure shows the input stride of uh one and the second part of this figure shows the input stride

of four so if you look at the first figure the first input of batch the input of batch one is in the heart of and the input of batch two is the heart of the so see there is just a stride of one year between batches between inputs of different batches but if you look at the second figure the input of batch one is in the heart of which is this and then you stride by four 1 2 3 and four so the next input will be the city stood the see so typically a more

stride means that you you move over the data in a faster Manner and so less computations will be there that's what you specify When you mention the stride so here stride is equal to one which which means that we'll be doing something very similar to uh what is mentioned in this uh example which I'm highlighting with a star right now awesome so here is how we create the input and output batches so what I want to mention here is that the first batch variable contains two tensors the first tensor stores the input token ID these

and the second St tenser stores the target token IDs which are these and since the max length is set to four each of the two tensors contain four token IDs that is also very very important to remember note that an input size of four is relatively small uh or the context size it is common to train llms with input sizes or context sizes of at least 256 now you can also see the second batch see the second batch is 367 2885 and this is the output of the second batch so the second batch input is

this the second batch output is this so if you compare the First with the second batch we can see that the second batch token IDs are just shifted with one position see this is the first batch input token IDs this is the second batch input token IDs now can you think what would be the stride in this case based on the explanation which I provided to you the stride will be equal to one because because if you look at the first input batch and the second input batch it just shifted by one um so the

stride setting dictates the number of position the input shift across batches emulating a sliding window approach that's why it's called as the sliding window approach batch sizes of one such as we have sampled from the data loader so far are used for illustration purposes if you have previous experience with deep learning you may know that small batch sizes require less memory but lead to more noisy model updates just like in regular deep learning the batch size is a trade-off and hyperparameter to experiment when training llms so as I mentioned the batch size is number of

data the model has to process before updating its parameters so if the batch size is very small the parameter updates will be very quick but the updates will be noisy if the batch size is very large the model will be going through the entire data set before or the model will be going through the large batch before making the update so the update will not be as noisy but it will take a lot of time so you need to make sure this hyper parameter is set correctly before we move on uh and end this lecture

the last thing which I want to show you is the effect of batch size uh so let's actually take a brief look at that essentially uh what happens when the batch size is more than one so the batch size so here you see I'm creating a data loader with a batch size equal to 8 and then I'm running this so see the input tensor and the output tensor and here I'm also incre increasing the stride to four so if you look at the first input and the second input you will see that there is not

an overlap because the side stride is equal to four and here the batch size is equal to eight so see the input has essentially eight uh input tensor has eight inputs which means the batch size equal to 8 and the output tensor will also have eight rows because the batch size is equal to eight so the model will essentially process this batch before making the parameter updates uh okay so now here you can see as I mentioned we have increased the stride to four this is to utilize the data set fully um we don't skip

a single word but also avoid overlap between batches since more overlap could lead to increased overfitting so that is one advantage of having a more stride so here you see in these two examples if you look at the second if you look at the second example over here the stride is equal to four so there is no overlap between the input batch one and the input batch two which might be good for overfitting per uh which might be good to prevent overfitting whereas if you look look over here the stride is just equal to one

so the input batch one and two have a lot of overlap that might lead to overfitting which is not great for training this actually brings us to the end of today's lecture where we covered number of things first we covered what does it mean to uh break down the text into input Target pairs input Target pairs are definitely needed for training of llms and we started with this example uh this example right here on the screen let me just take you to that yeah so we started with yeah so we started with this example to

illustratively explain what the input Target pair looks like so the target pair is just the input which is shifted by one then we saw how to convert this into X and Y pairs so if the input is 1 2 3 4 the output is 2 3 4 5 here the context size is four which means that if one is the input the output should be two if one and two is the input the output should be three if 1 2 3 is the input the output should be four if 1 2 3 4 is the

input the output should be five so in one input Target pair there are four computations or four predictions because the context size is four the rest of the lecture was devoted to creating input Target pairs like this but in a structured manner for that what we actually did was we used something which is called as data loader and then we paired the data loader with the data set so first we created this GPT data set V1 class uh we created a data set and then we fed that data set into the data loader what this

data loer essentially did is is just it created these input output pairs but in a very structured manner why structured because we can Define many things we can Define things like stride batch size number of workers Etc uh so there are essentially three things which are very important for you to remember and let me take this example by rubbing some things of the screen so the first thing to remember is the uh stride so stride basically dictates here yeah so this this schematic here shows the difference between stride so if you see the top top

level schematic here um this shows a stride of one which means that the input of batch one and batch two will just be differing by one word so there is a lot of overlap which might lead to overfitting what people people usually do is that they keep the stride length equal to the context length so that you don't even miss any word but there is not an overlap between different input batches that's the meaning of stride the second thing is the uh batch size so what the batch size lets you do is it tells the

model how many batches of data you want to process at once before making the parameter updates so towards the end of this lecture we showed an example where the batch size was equal to 8 which means that all these eight input output pairs will be processed before making the parameter updates and the last parameter which we did not explore too much is actually number of workers what this means is that you can exploit the parallel Computing facilities in your own computer because all of us have multiple threads on our computer so you can make sure

the Computing happens on Parallel threads by specifying number of workers so what the data loader essentially does is that ultimately we can extract input output pairs through the first batch second batch Etc in the data loader and then this now will be converted into Vector embeddings which then we will feed into our model training the next section or the next lecture is going to be a token embedding or vector embedding I know everyone that today's lecture was a bit more complex especially because of the data set and data loader but this part is very rarely

covered by anyone and I really wanted to make sure I cover it in a lot of detail thank you so much everyone I hope you are enjoying and understanding these lectures I'm deliberately making them a bit long so that everything is covered from scratch thanks everyone and I look forward to seeing you in the next lecture