Lecture 7: Code an LLM Tokenizer from Scratch in Python

[Music] hello everyone welcome to this lecture in the build large language models from scratch series my name is Dr Raj dander and I'll be your instructor for this lecture today we are going to cover a very important topic which is called as tokenization basically when we build large language model Model S the data needs to be pre-processed in a certain Manner before it is used for pre-training and uh in this data pre-processing pipeline the first step is tokenization at its really basic form tokenization is just the process of breaking down a sentence into individual words

but it is a bit more detailed than that the way it is used to act build large language models in today's lecture we are going to understand everything there is to know about Tokenization and we will build a tokenizer fully from scratch we will also build an encoder and decoder from scratch and we'll code it out in Python so today will be a Hands-On session if you have a laptop in front of you U that's great so that you can follow along in Google collab or jupyter notebook or V s code to implement the code

along with me so let's get started with today's lecture building a large language model Usually proceeds in three stages especially if you want to build it from scratch in stage one we have to understand the nuances or basic mechanism of building an llm and that includes data preparation attention mechanism and understanding the llm architecture stage two involves pre-training and building the foundational model so that involves the training Loop model evaluation and loading pre-trained weights and stage Three involves finetuning training on smaller very specific data sets to build applications which are actually useful to you such

as a classifier or a personal assistant in this lecture we are going to look at the first aspect of stage one which is data preparation and pre data preparation and sampling so uh tokenization comes under the category of data preparation and sampling and that is what we are going To cover in today's lecture you can think of this as the first building block for building a large language model so let's get started with today's lecture so the main question which we are going to answer today is how exactly do you prepare input text for training

large language models at the heart of it llms are just neural networks right so you need data the parameters of the llm are optimized and then we have some Output the question is that this data which comes in as the input what form should it take how should we prepare the input text we have a huge number of documents right which the llm is trained on we have seen that in previous lectures the llm is usually trained on billions of documents but do we feed the document Direct ly as the input text do we feed

the sentences of the document as the input text no it turns out we have to tokenize the document and then feed Individual tokens uh there is one more step after this which is called as Vector embeding but we'll have a separate lecture for that today we are only going to look at tokenization so the process of tokenization can be broadly broken down into three steps this is the key takeaway which you should learn from today's lecture the first step is that you have to initially split the text into individual word and Subo tokens that is

Step number one imagine you have huge amount of text you break it down into individual words that's the first step of tokenization the second step of tokenization is converting these tokens into token IDs and the third step is basically encode these token ID into Vector representation we are not going to look at step number three in today's lecture because it comes also under Vector embedding but essentially we are going To look at step one and step number two so let me repeat step one and step two step one is basically imagine you have a big

document which is let's say you're training on Harry Potter books you take this books you divide it into sentences and then you split the text into individual word and subword tokens and then you convert these tokens into token IDs that's the uh two major steps involved in tokenization which we are going to look In today's lecture um so I just want to show you a visual for how the input text is given to the large language model so let's say this is an input text right this is an example remember the first step of tokenization

we break this into individual words so the first word is this the second word is is the third word is an and the fourth word is example so this is the tokenized text and this is my step Number one right now step number two is we need to convert each of these individual tokens into token IDs so this is the token ID 1 4013 token ID 2 2011 token ID 3 which is 302 and token ID 4 which is 1134 so every word or every token rather has a token ID associated with it I have

currently given these numbers randomly but you need to assign a token ID for each word and then the next step after getting these token IDs is to convert These token IDs into something called token embeddings or vector embeddings and then these Vector embeddings are then fed as input data to the GPT or the llm rather so even before coming to the training stage we need to do these many steps of uh pre-processing and today we are essentially going to look at step number one and step number two I hope you have understood the two Main

steps which we are going to look at in today's lecture so let us start by looking at step number one initially which is tokenizing the text or breaking down the text into individual words and you definitely do need to understand the nuances which are involved in this and that's why I'm going to take you through the entire course process in Python uh we could have just use a pre-built tokenizer but then you won't understand The nuances involved in tokenizing and uh just to give out a a shout out or rather a credits to Sebastian rashka

for hosting the this code publicly and I'll be heavily borrowing from this code so let's get started for the purposes of the demonstration today we are going to use a specific data set we are going to use this book book called The Verdict which has been written by Edith Warton so let's look at a few screenshots or let's look at a few data about this book So if you search The Verdict by edit Warton you'll get a image of this book which looks like this uh year of publishing let's look at the year when this

book was published so this was I think published in 1908 so almost 120 130 years back and it's available for free uh free to download I'll give the link to download this book it's also available in this repository by Sebastian you can just click on this download and then download this book so We are going to look at this book and assume that this is the only uh training data which we have for the purposes of today's lecture but whatever I show you today can be really applied to any book or even a huge number

of books I encourage you later after this lecture to try out some different book which is available for free online such as the Harry Potter book or whichever one interests you and apply the same code over There okay so this is the data set which we are going to look at and then what we have to do is basically download and load this data set in Python so let me show you how to do that so now we are moving into uh jupyter coding uh please comment out in the comment section if the code is

visible to you if I've zoomed in enough or if I should follow some different style okay so the first step as I mentioned we are going to look at two steps today Creating tokens and then converting these tokens into token IDs the first step is creating tokens right so what we are going to do here is that we are going to uh download the verdict. txt file we are going to open it in Python and we are going to read it and we are going to print the total number of characters so I'm just showing

this uh when you download it it looks this much it's not that big um and in fact it's even available over Here so I have downloaded this book right now and uh in Python you can use this open command so with open the word it DOC text and then R means we are reading this and then raw text is that variable where we are storing whatever uh content python has read from this book so this raw text variable is going to be very important for us because we'll also refer to it later basically raw text

is just the entire text which we are reading and then what we are Doing here is that we are counting the total number of characters in this raw text so what I'm doing in this part of the code is that I'm printing out the total number of characters and then length raw text right so here you see the print command towards the end raw text colon 99 what does this do this print command prints the total number of characters uh so here so there are two print commands right the first print command essentially prints the

total Number of characters and the second print command essentially uh prints the first 100 characters of this file for illustration purposes so uh let's run both of these print commands right now I have already run this before but let me click on this run Icon again so okay the result of this first print command is this the total number of characters are 20479 20479 and the result of the second print command is is basically we are printing The first uh 100 characters of this file right so the first 100 characters are I had always thought

Jack uh gisburn Etc so this is the these are the first 100 characters so this indicates that python has been able to successfully read the text which we have downloaded and that's an awesome sign great now let's move further our goal in this lecture is to tokenize all of the characters which are in this book in this text uh in this short story Into individual words that we can later turn into embeddings for llm training remember uh embeddings is the last step so before you come to training the llm there is this thing called embedding

so the goal in this lecture is to do the tokenization and the token IDs for the entire text which we have read for all of these characters for the 20479 characters of this short story the goal is is to convert them into tokens and token IDs so one note which has been added over here is that note that it's common to process millions of Articles and hundreds of thousands of books which is a huge amount of memory when actually working with llms this is very important to note when you actually work with llms you don't

just download one book as I showed to you right now we actually work with millions and thousands of books uh which are a huge amount of data uh to be stored on your computer however for Educational purposes it's sufficient to work with smaller text samples like a single book and that's why we are using a single book in today's code to demonstrate the idea to you okay so the next question is how can we best split this text to obtain a list of tokens so I encourage you to pause here right now and let's say

I give you this problem that you have read this text now your goal is to split this text into individual tokens right how would You do this which python Library would you use uh so let me reveal the answer the python Library which we we will be using for this is called as regular expression re uh so this module Pro provides regular expression matching operations and let's see how basically this works we have to import re and then what we can do is that we can use this library to split uh to split any given

text Um based on the white spaces in that text or any other characters in the text and I'll explain what this actually means okay so okay so let's say you have a text which looks like this hello world this is a test right and you want to split and you want to split this into individual tokens what I'm going to do here is that from this Library I'm going to use this function re e dos split right and then there is this R and then The symbol is/ S what this basically does is that this

slash s means that we split wherever white spaces are encountered right so let me tell you what is meant by white spaces so let's look at this word let's look at this sentence you see there is a white space over here there is a white space over here there is a white space over here here and here so wherever there is a white space what this command will do is that it will split the text into Individual tokens and where the split will be wherever the white space is there so it will scan this sentence

from left to right now here it encounters the white space so it will split split here then again it will move forward here it encounters the white space so it will split here so let's go from left to right so until hello and comma it has not encountered a white space right so hello and comma is one token and then it encounters a white space so it splits so Then it outputs the white space here when you print out the result then it again goes on scanning and here until world and full stop there is

no white space so it prints Out World and full stop then after this point there is a white space right see this so then it prints this out then it again keeps on scanning from left to right so till this and comma there is no white space so it prints out this and comma then it prints out another white space over here so it Keeps on scanning from left to right and then splits wherever white space is encountered right that is the advantage of using re do split U and then this character indicates where you

want to split so here I have indicated that you split wherever you see a white space right so the result of this is a list of individual words white spaces and punctuation characters so see white spaces are also the results of This now uh what we want to do is we want to also uh split commas and periods so here you see the comma is included as part of the word word itself we the full stop is included as part of the world but we want to have comma and full stop also as separate tokens

how do we do that it's pretty simple in the re. split command you also include comma and you also include the full stop along with this SLS command and then when you print the result you will see that along with White spaces so white spaces are of course uh printed out because we are splitting on white spaces also but comma is now a separate token full stop is now a separate token which was not the case before hello and comma were one token but now you'll see hello and comma are separate tokens awesome so now

we have split based on comma and full stops also correct now the main another remaining issue is that our list still includes space characters right see these the White space characters are still counted as tokens we can even remove these red redundant character safely so here what we are doing is that uh you first scan the result scan each item in the result so we are looping over each item in the result so this is the result right so we are going over each of the individual items in this result and if item. strip is

equal to true then only we return that item so if it's a y space then item. strip will Actually be false so white spaces will not be returned Whenever there is actually a full world full word like hello or comma or world or this or full stop then only item. strip will return true and those words will be returned in this result so using this statement uh loop over item in result and then find out item. strip so for white spaces item. strip will anyway be false so that will not be returned so in this

one line of command we can actually get rid of White spaces completely from the sentence and now when we print out the result we'll have the tokens as hello comma world full stop this comma is a test and full stop we won't have white spaces so one more consideration which I would like to mention over here is that removing white spaces are not this is an important question to actually discuss because uh when you develop a simple tokenizer whether we should encode white spaces as or as separate Characters or just remove them depends on our

application and its uh requirements so what are the advantages of removing white spaces removing white spaces reduces the memory and Computing requirements which is great however keeping white spaces can be useful if we train models that are sensitive to the exact structure of the text right for example python code is sensitive to indentation and spacing so if python code is used as the data set for Training and a large language model it makes sense to keep the white spaces right because the white spaces have some meaning in in the case of python code as the

training data set in this case we are going to uh until now we are going to remove the white spaces just for Simplicity so that we have memory advantages but remember that it's not obvious to always remove white spaces uh as has been mentioned uh uh over here if you have python code AS Training data white spaces are important because indentation matters so several things need to be considered when you're building an actual llm whether you should remove white spaces or whether you should not remove white spaces uh that's great so the tokenization scheme which

we deviced about works well on simple sample text so we can modify it a bit further because we also want uh to have question marks quotation marks and double dashes As separate characters or separate tokens currently only full stop and comma are separate tokens right but we also want question marks quotation marks double dashes Etc separate tokens so let's look at this document again see there are these double dashes then I'm sure there are question marks somewhere there are question marks there are exclamation marks in this document we all want them as separate tokens right

and we need to include that in our Tokenization Command and the way we do this is pretty simple again we use this re. split and along with comma and along with full stop we add all of these other special characters like colon semicolon question mark underscore exclamation uh quotation marks bracket Etc and uh we say that whenever these characters are encountered also do a split on these characters so that they are considered as individual tokens right um so so now when you take the Sample text hello comma world is this dash dash a text all

of these will be uh sample tokens this dash dash will also be a separate token the question mark will also be a separate token because we have included that in our split command and again we'll strip the white spaces like the same command for item for item in result if item. strip remember this was the command we used to filter out the white spaces or to remove the white spaces Uh awesome so now these two lines of code are our tokenization scheme so this is the two lines of code in which we have built the

tokenizer in this first line of code we take a sentence and we split the sentence wherever there is a white space wherever there is a comma full stop colon semicolon question mark underscore exclamation quotation mark or bracket we split it and then in the second sentence we just remove the white spaces and then we print out the result It's as simple as that this is the simple tokenization scheme for building large language models a different tokenization scheme is used but we'll come to that in the next lecture it's called bite pair encoding but for now

I just wanted to give you a sense of what all nuances need to be considered for building a tokenizer like this now you see we have we have tested these two uh sentences which are our tokenizer scheme for a sample sentence right now what we Can do is apply the same to two statements on the entire raw text so we had the raw text right defined before let me go to that U so see this was the raw text which was basically the book which we had read now we'll use these two statements to essentially

uh convert the entire uh raw text into individual tokens so what we'll do is that we will split the entire raw text based on these tokens which we had seen before and then We will store it in a variable which is called as pre-processed so pre-process pre-processed is essentially a list of all the tokens and then in the second statement exactly same to what we saw before we'll split or we'll get rid of the white spaces awesome right so now let us print the length of the pre-processed so we have 4690 tokens in the original

uh book so if you go to this book the simplest way to think About this is this will be one token height will be one token this inverted commas will be one token these two double dashes will be one token so this entire thing is broken down into individual words or tokens which are 4690 in number and here we have just printed out the first 30 of these tokens and you see I had always thought they are individual tokens and dash dash comma they are also counted as tokens over here because we have uh mentioned

That in this split where all do we want the split to happen awesome so we have now got our basic tokenizer working and we have applied it to the entire short story which we are considering considering as the input data great now we have to come to the Second Step so we have now finished the we have now finished the first step which is uh tokenizing the entire short story and now we will come to the next step which is converting tokens into token IDs now What is very important is that python works with numbers

right so even if you look at these individual tokens which you have obtained right now these are not numbers they are still words we need to convert them into token IDs so that they are numerical representations so let's see how to do that so we have the complete training data set here for illustration I have just taken a sentence and until now we have converted this sentence into individual tokens now What we will do is that we'll take the tokens and convert them into token IDs and then there is a very specific way of doing

this first we build something which is called as vocabulary vocabulary is just all the list of our tokens but it's sorted in an alphabetical manner so if the if our data set or the tokens are the quick brown fox jumps over the lazy dog if this is the training data set the vocabulary is the list of tokens in alphabetical order so Brown comes first Then dog then Fox then jumps then lazy easy then over then quick and then the this is my vocabulary for this training data set eventually we'll have billions of data sets billions

of files so then the vocabulary will be huge but I'm just showing a simple representation here for how vocabulary is constructed vocabulary is just a list of tokens which is sorted in alphabetical Manner and then what we do is that each unique token is mapped to a unique integer which is called as The token ID so it's as simple as that you map these tokens in alphabetical order and then to each token you assign a number so since they are arranged in alphabetical order Brown will be zero so you start from zero since it's python

dog will be one fox will be two jumps will be three lazy will be four over will be five quick will be six and the will be seven so the vocabulary always contains unique tokens remember although the appears two times in the vocabulary It only comes once so we make a list of these unique tokens and we assign token IDs to all of these and it's important to arrange the vocabulary in alphabetical order so that assigning token IDs becomes very easy so the process of assigning token IDs is actually very simple each unique just remember

that each unique token is mapped to a unique integer which is called as the token ID right so now let's see how this is implemented in Python uh so in the previous section we talk loaned edit worthon short story and assigned it to a python variable called pre-processed so remember pre-processed is the final list which contains all the tokenized words this these words uh now what we'll do is now we'll create a list of all unique tokens and sort them alphabetically to determine the vocabulary size exactly what I had mentioned over here so what we

are doing is that we are taking pre-processed we Are converting it into a set and we are going to sort it into set so this will sort it in alphabetical order and then we are just going to print the vocabulary size so we can see that the vocabulary size is 1 130 right remember the vocabulary only consists of unique words so the vocabulary size is less than the number of tokens which are there uh so now what we can do is after determining that the vocabulary size is 1130 via the above code we create the

Vocabulary itself so when I say you create the vocabulary remember that a vocabulary is not just these tokens but every token needs to be assigned to a token ID a vocabulary is a is like a dictionary of tokens and Associated token IDs so now we have to create that dictionary right and so it's very simple you just all words consists of all the unique words in the vocabulary right in alphabetical order what you do is that you Just uh you take all these tokens and assign an integer value to all these words that's it which

means the token which comes first that will be zero the token which comes second its integer will be one and it will proceed like that uh let me print this out for you to show you so our tokens are also exclamation marks commas Etc right so by default they are assigned the first priority so they will be 0 1 2 3 4 5 6 and here you see everything is arranged In alphabetical order so a will be 11 ah will be 12 among will be 13 and will be 14 R will be 15 a r

RT will be 16 as will be 17 at will be 18 Etc now all of these exclamation all of these I would say uh words are used because this was written in 1908 this a rrt is not used as an exclamation typically now but this book is written in 1908 so it shows up right so this is the ulary which we have and we have printed the first 50 items of this vocabulary so it looks like a Dictionary pretty much and remember that every element in this dictionary uh which is a token has a token

ID now associated with it that's it a vocabulary is as simple as that and the simplified code for this is for every token assign an integer and how do you get the integer and tokens you just enumerate all words so this enumerate command in Python what it does is that it takes all the words and then it assigns an integer to each Word in alphabetical order so that way python is actually awesome and you can write very complex things in in one line of code actually so this is how you create the vocabulary and this

is how you assign token IDs to every individual token right so as we can see based on the output above the dictionary contains individual tokens which are associated with unique in integer labels exactly what what we saw here the vocabulary which is a dictionary consists of Individual tokens assigned to a unique integer label which are tokenized s now one more thing which I want to tell you is that currently we converted the words or tokens into token IDs right you can think of this process as encoding now later uh we will also need a decoder

which means that from the token ID you need to convert the token ID back to the word because when the llm gives the output that will be in numerical form now you need to convert It in word form so you also need a mapping from the token ID to the Token the vocabulary is a dictionary which gives you a mapping from the token to the Token ID but we need a reverse mapping which is called as a decoder so this is what uh I have mentioned here later when we want to create the output of

an llm from numbers back into text we also need a way to turn token IDs back into text and for this we create an inverse version of the Vocabulary that Maps token IDs back to the corresponding text tokens and we'll see how to do that for now what we are going to do is we have understood enough about tokenization that we are going to implement a complete tokenizer class in Python this class will have two methods it will have an encode method and it will have a decode method let me show you what it actually

means so this tokenizer class which we are going to Implement in Python it will have two methods the first will be the encode method and the second will be the decode method in the encode method what will happen is that sample text will be converted into tokens and then tokens will be assigned token IDs based on the vocabulary exactly what we saw like over here in the decode method exactly reverse things would happen so in the decode method we start with token IDs we Convert it into individual tokens and then we get back the sample

text so I hope you understand the difference between the encode method and the decode method here because this is the exact same difference which will show up in the encoder and decoder block of the Transformer architecture or rather this is the this is a good way to understand that intuition in the encode in the encode block what we do is we take sample text we convert it into tokens And then we convert it into token IDs that feeds as the training data to the llm but when the llm gives its output in the form of

token IDs we need to convert it back to tokens and and then back to the sample text so that we know what the output is in terms of sentences so that's why when we implement the tokenizer class we need an encode method uh and also the decode method so the encode method will take text as an input and give token IDs as Output the decode method will take IDs token IDs as input and will give text as an output now let's see how to actually create this simple token izer class in Python first so there

are if you see three methods the init method which is called by default when an instance of this class is created and let us look at the arguments of the init method it takes wcab so when you create an instance of the tokenizer class you have To pass in the vocabulary right and remember the vocabulary is nothing but a mapping from tokens to token IDs right so then after this uh instance is created St Str to in which is string to integer will just be the vocabulary because the vocabulary is already a mapping from string

to integer or tokens to integer and then the integer to string is basically reverse so what you do is that you take the string and integer in the vocabulary and then for Every integer you uh mention which token it is so for S you can think of as the token and for I you can think of as token ID so what we do in this int to string variable variable is that uh so we take the token and we take the token ID in the vocabulary and then we just flip it then we say that

for this token ID this is the particular token remember this into string will be needed for the decoder method when we have the token IDs and we want to convert it back to Tokens so uh in the encode method the exact same pre-processing steps will happen as we had seen before for tokenization what we'll do if some random text is given to us we will take that text we will split it we'll split it based on the comma based on the full stop based on the colon based on semicolon Etc into individual tokens and then

we'll get rid of the white spaces what we had seen before and then these will be individual tokens right so up Till now we are at this part where we have converted the sample text into individual tokens and then we'll convert these into token IDs so that's the last step over here after you get the individual tokens in this list called pre-processed what you have to do is that you have to use this St Str to in dictionary which is basically just the vocabulary and then you have to uh assign a token ID for each

token so remember the Str str2 int is basically Just converting tokens to token IDs using the vocabulary which is passed into this tokenizer class so you will just take the uh tokens you'll take the tokens which are in this pre-process list and you'll convert them into token IDs that's the encoder method that's it in the decoder method what you are actually doing is that you are uh you are using this reverse dictionary which is integer to string which is basically token ID to token and then you are Converting the token IDs into individual tokens that's

the first thing so let's see the decode method you first convert the token IDs into individual tokens and then what you do is you join these individual tokens together so this this join is used so first you convert the token IDs into tokens and then you join the individual tokens together that's it and then here what we are doing is we are going to replace spaces before the punctuations so an example here would be Let's say if the tokens let's say in the decoder if the tokens are the fox chased and question and full stop

right now if these are the tokens and if we convert it into sample text by using the join method the final answer will be the the the fox chased and full stop now see the problem here is that there is a space between the there is a space here between the full stop and the chased so we need to get rid of this space so then the final answer would be the fox Chased and full stop and this is the the same for question mark Etc so in this second sentence here what we are actually

doing is that we are getting rid of all the spaces before the punctuation so that it becomes a complete sentence so this is the decoder method it's actually very simple we are just going to uh use the encode method to convert uh sample text into token IDs and we are going to use the decode Method to convert token IDs back into sample text and for that we needed to write two methods the encode method and the decode method so remember this tokenizer class takes the vocabulary already as an input so that so we already have

the mapping from tokens to token IDs because that is inherently present in the vocabulary we just need to construct a reverse mapping from the token IDs back to the tokens and then use that in the decode method that's it And to convert the text into tokens we use the same re dos split and item do string which we had seen earlier in today's lecture so this is essentially the tokenizer class which we have created awesome now let's move to the next step so what we can do is that now we can instantiate a tokenizer object

from this class right and uh tokenize a passage from the short story which we have downloaded to test it out in practice so Here you see I'm creating an instance of this class I have passed the vocabulary as an input what is this vocabulary this vocabulary is basically the uh vocabulary of tokens and token IDs which we have converted our input text so this is our input text and we have converted this into tokens and assigned the token ID to each this is my vocabulary now what I'm doing is I'm actually uh creating an instance

of the yeah I'm creating an instance of the simple Tokenizer version one class which we have defined over here by passing in this vocabulary as an input right and this is then defined as tokenizer great and uh so the text which I'm going to pass in now is this it's the last he painted you know Mrs gburn said with pardonable pride this is the text and now I'm going to test out this encode so remember this encode takes in the text as an input so you need a text right to convert into IDs so what

the encode Method will do always remember this schematic always remember the schematic the encode method will convert the text into token ID is so this is the text and then we apply the method tokenizer do encode and text so what when you print the IDS you will see these IDs of the text which means that our encoder has successfully converted this text into token IDs that is exactly what we wanted right so the code above prints the token IDs next let's see if we can turn these Token IDs back into text so now what we'll

be doing is we'll be using tokenizer do decode and use these IDs so tokenizer decode and IDs uh and pass IDs as an input remember tokenizer do decode takes the IDS as an input so I'll pass these IDs which we have printed over here and let's see whether it recovers this text so the text which is recovered is it's the last he painted you know Mrs gburn said with pardonable pride amazing right because it's exactly the same text Which we had given uh to the encoder and now it has been decoded by the decoder So

based on this output above we can see that the decode method successfully converted the token IDs back into the original text which is exactly what we wanted which means the encoder and decoder are working right so so far so good we have implemented a tokenizer which is capable of tokenizing and DET tokenizing text based on a snippet from the training set so this sentence which We gave to the encoder was from the training set so we knew that the words will be in the vocabulary but what if the sentence which is given is not present

in the vocabulary so we have a vocabulary here which is 1130 the size of the vocabulary is 1130 right what if I give a sentence to en code which is not present in the vocabulary so let's try this let's say the text is hello do you like T now uh hello is probably not there in this hello is not there in this Short story so let me search hello it's not there let's see if T is there t is actually T is there but hello is not there so hello will not be there in the

vocabulary right so now let let me see how what's my answer here because now I have asked the tokenizer to encode something which is not present in the vocabulary okay so if I run this you see you get an error here because hello it does not know what to do with this word hello because it is not present in the Vocabulary so the problem is that the word hello was not used in the verdict short story and hence it is not contained in the vocabulary now this actually highlights the need to consider large and diverse

training data sets to extend the vocabulary when working with llms we do not want this problem right if our training data set is small and if some user gives a new word to chat GPT which it does not even know we don't want our We don't want an error that's why huge number of data set is used for training large language models in fact llms like chat GPT use a very special thing they use something which is called a special context tokens to deal with Words which might not be present in the vocabulary so that

an error message is not shown such as this and we will come to that in the next section but here I just want to illustrate that the reason for having a A large and diverse training data set is because uh we we want many words to be present in our vocabulary it does not make sense to have a smaller and a shorter vocabulary because what if the user gives a new word which is not present in the vocabulary we don't want that there is a way to deal with this by using something called special context

tokens so let's look at this in the next section now let us have a look at the last section in today's lecture which is Dealing with special context tokens these content are not usually covered in other lectures but they are very important especially since we are going to be building llms from scratch we also want to understand how real life llms work and special context tokens play an important role so until now we have implemented a simple tokenizer right and applied it to a passage from the training set but the main question we encountered before

is that if there Is a word in the text which is not there in the vocabulary how do we encode this word um so we will be implementing some uh things which are called as special context tokens what we will do is that we will modify the tokenizer to handle unknown words uh and we will Implement a python class which is called a simple tokenizer version two so we have already implemented uh simple tokenizer version one but here it did not have the Provision to handle the unknown tokens so in this version two we will

also uh Implement uh the simple tokenizer version two to have the provision to handle the unknown tokens in particular we will be learning about two main tokens the first is this this token which is unknown for an unknown word and the second token is for end of text so let me again go back to this uh whiteboard over here and write a bit about why we are exactly using these Tokens so uh let's take a sentence which is the fox chased the dog let's say this is my sentence right now and I have tokenized it

into the Fox Chase the dog and here's the vocabulary so it's arranged in alphabetical order and there are token IDs right now to this existing vocabulary we are going to add two more tokens the first is the Unk which is unknown and we'll assign a token ID and then we'll also have end of text and then we'll assign it a token ID These two are the last two uh tokens in the vocabulary so the token is corresponding to these two will be the largest so what will happen is that let's say if some new sentence

or some new word is given so let's say uh the fox chased the dog quickly if that is the word all these other words like the Fox Chase the dog will be converted into token IDs but for quickly it will have a Token ID of 783 which is the token ID of unknown and why because quickly was not in the vocabulary our vocabulary only consisted of chased dog fox and the so quickly is an unknown word and so it will receive a token ID which we have reserved for unknown words and then you might be

asking what about this end of text so end of text is something which is a bit different uh when we are working with multiple text sources we typically add End of text token between the text so let us look at four text sources here this is the text Source One let's say it comes from one book let's say this is the text Source two let's say it comes from another news article text Source three which comes from an encyclopedia and text Source Four let's say which comes from an interview let's say these are are our

training sets usually all of these are not just collected into one giant Document or all of these sentences are not just stacked up together we usually after after this initial text is fed as an input we have this end of text token which means that the first text has ended and now the second text has started after the second text ends then we again have this end of text token then the third text starts and after the third text ends then we again have the end of text token and then the fourth text starts so

the end of text token is Basically added between the texts so essentially the end end of text tokens act as markers signaling the start or end of a particular segment this leads to more effective processing and understanding by the llm it's very important to add these end of text tokens because then llm treats the initial so let's say end of text there is before the end of text which is text Source One and after the end of text which is text Source two if end if end Of text was not there the llm would have

mixed all of this together right uh but end of text tokens allows the llm to process the data and understand the data in a much better Manner and in fact when GPT was trained the end of text tokens were used between different text sources this is very important to note and only students who try to understand tokenization in detail know about such specifics so these are the two tokens which we will be considering in this Section the first is the unknown token and the second is the end of text token so let's read bit here

we can modify the tokenizer to use an unknown token if it encounters a word that is not part of the vocabulary great furthermore we also add a token between unrelated text which is the end of text token so for example when training GPT like llms on in multiple independent documents it is common to insert a token before each document or book that Follows a previous text Source basically this is exactly the end of text token which has been written over here I'll be sharing this notebook with all of you so no need to worry if

you miss certain portion or want to revise certain portion again now what we'll be doing as I mentioned over here uh is that we'll be modifying the vocabulary we'll be augmenting the vocabulary and uh how will we modify or augment the vocabulary we will include Special tokens we will include two special Tokens The Unknown and the end of text token and we will add these to the list of all the unique words that we created in the vocabulary in the previous section so let's look at this pre-processed so remember pre-processed is the vocabulary uh or

U so preprocessed was our list and then later we converted this into vocabulary which was stored in vocab right and remember the size of the vocab is 1130 and now we Are going to add two more tokens so the size will be 1132 so let us again start with pre-processed we will sort this and then we'll add two tokens so we'll add the end of text token and then we'll add the unknown token and that is done by using the python command which is called extend so what this does is that it just adds two

more additional entries to the list the sorted list in this case and then again we'll write the same Command which we did earlier right we first enumerate all the tokens in the pre-processed list and then for each token we assign an integer and this enumer it will make sure that the tokens are anyways arranged in alphabetical order and then for each of these token we assign an integer which is the token ID that's how we create the vocabulary now if you print out the length of the vocabulary you will see that the length of the

Vocabulary is 1132 and if you remember without adding these two tokens the length of the vocabulary was 1130 so now the length has increased by two that is great right uh that makes sense because two more tokens have been added so the vocabulary is extended great so based on the output of the print statement above the new vocabulary size is I should mention here actually 1132 uh so this should be 1132 and in the previous section it was uh 1130 and let me run this yeah so based on the output of the print statement about

the new vocabulary size is 1132 and the vocabulary size in the previous section was 1130 good so what we'll do as an additional Quick Check we'll also print the last five entries of the updated vocabulary so if you print out the last five entries in the updated vocabulary you will see Let's look at the last two entries these are the end of text and the unknown so the end of text has a token ID of 1130 and the unknown has a token ID of 1131 awesome so now the vocabulary is updated and now what we'll

be doing is that we'll be extending the simple tokenizer class uh this is the version two basically many things will remain the same this initialization will remain the same it will initialize two dictionaries the string to integer dictionary which Is the vocabulary itself which converts the tokens into token IDs and the integer to string dictionary which has token IDs and then a string mapped to each token ID this encode now let's look at this encode the first two sentences are very similar to what we saw before we split on the comma full stop colon semicolon

question mark underscore explanation quotation and bracket and then we remove the white spaces using item. strip but We add one more thing what we add is that if the item or if the particular entry is not present in the vocabulary uh the token which is assigned to that entry is unknown so let's say you are scanning the text right and if you come across a word in the text which is not in the vocabulary so if the item is not present in this string to integer vocabulary which is the vocabulary which we have passed uh

then it is it should be the unknown Token we replace that with the unknown token and then we convert all of these tokens to token IDs so in this step what will happen is that all the words in the input text which are not in the vocabulary will be replaced by the unknown token and then in this step that is the main encoder step where the tokens are converted into token IDs in the decoder part part everything will uh almost stay exactly the same nothing changes we first convert the Token IDs back into tokens and

then we join them together and uh before the punctuations there are spaces so we get rid of these spaces and then we return the decoded text so this is the change in this tokenizer simple tokenizer version two the only change is uh adding this so if some word is not known you encode it with unknown and then assign the correspond oning ID to it so if something is unknown the ID will be 1131 great now let's actually try some Things okay let's say the first text source which we have is hello do you like T

the second text source which we have is in The sunl Terraces of the palace these are two text sources now what we'll be doing is we will construct a text which joins these two and we will add an end of text at the end of the first text Source why are we doing this because we saw that this is usually how it's done in practice in GPT Etc if you provide one text source and if you have Another text Source you don't just join them together you split them with this end of text token so

this is what we are exactly doing we are now constructing a text which will feed as an input to the encoder right but we add this end of text so the text Will which will essentially feed to the encoder is hello do you like T end of text in The sunlit Terraces of the palet s this is the text which we are feeding to the encoder and now we are asking the encoder to encode These into token IDs so let's look at this sentence in detail this word hello is not present in our vocabulary so

if if we pass it into this tokenizer it will hit this this statement where the word is not present in the vocabulary so it will replace it with the unknown token and this end of text again end of text is actually present in the vocabulary because we have added it right now so the token ID corresponding to end of text will be 1130 and the Token ID corresponding to hello will be 1131 right so let's check if this is actually true so I'll now run tokenizer encod text and see the token ID for hello is

11131 this is exactly what we had expected right because uh hello is not present in the main vocabulary so it's actually an unknown word so it token ID should be 1131 awesome and then for end of text let's see the token ID yeah this this is the token ID for end of text the token ID is 1130 so now there is no error which is coming here earlier when we passed this hello do you like T there was an error which we are getting that hello is not present in the vocabulary but now this error

is not coming because we have taken care of it we have added the unknown token in the vocabulary and so now the tokenizer encodes the text uh in a correct manner that's awesome so now what we'll be doing is that we'll be actually using The decode function now and we'll pass the encoded text which are the token IDs these IDs into the tokenizer do decode so let's see what the tokenizer decode so when these are passed into the decoder the decoder is unknown do you like T end of text in The sunlit Terraces of the

unknown so actually there are two unknown text here hello is an unknown and Palace is also an unknown so the token IDs for both of these are 1131 so when you decode these IDs you Will get unknown do you like T so which because hello is not known then end of text and then in The sunl Terraces of the unknown so here again unknown means the palace which was not in the vocabulary so the encoder and decoder are working perfectly and we are able to handle the unknown words now uh we are able to handle

the unknown words actually quite effectively because we replace them with the unknown token and the end of text is also being Captured pretty effectively so when we actually make the text itself we have to add end of text before we pass it to the encoder and subsequently to the decoder So based on comparing the det tokenized text with the original input text we know that the training set which is the verdict book did not actually contain the words hello and the palace because both of them are replaced by this unknown token here so we will

not get errors this way once we take into Account the special uh context tokens awesome so uh let me just add a last note about the special context token so up till now we have discussed tokenization as an essential step in processing text as input to the llms however along with these Special tokens there are some other tokens which also researchers consider so this is the beginning of a sequence so BOS token so we saw the end of text right some researchers also consider BOS so this token marks the start of a text it signifies

to the llm where a piece of context begins or where a piece of content begins right then the second token is eos which is end of sequence right so this token is positioned at the end of a text and especially useful where concatenating multiple unrelated text it's similar to end of text uh then the third token which is important is the padding token so when training llms with batch sizes larger Than one the batch might contain texts of varying lengths so to ensure that all the texts in different batches have the same length the shorter

texts are extended or padded using the pad token up to the length of the longest text in the batch so imagine the different text is being put in batches to the llm for parallel processing we'll look at this in detail so no need to worry about this right now but but just remember that for efficient Computing llm parallely Processes the batches so each batch contains text which have different sizes the shortest Texs are augmented with this pad token to match uh their length with the largest text and now we know how to add these special

tokens right we just augment the vocabulary with these tokens and whenever we pass text to the encoder we we may add things like beginning of sequence end of text the pad token Etc so we only saw uh the unknown and the End of text special context tokens but there are three other BOS beginning of sequence and EOS end of sequence and Pad which is padding now here I want you to note that the tokenizer which is used for GPT models does not need any of these tokens mentioned above but it only uses the end of

text token for Simplicity so gpt3 gp4 when they are trained they don't use BOS uh padding they don't even use the unknown token they only use the end of text token for Simplicity and this I also shown over here so the end of text which is used is exactly how it is used in GPT training as well okay and the last sentence is that the tokenizer used for GPT models also does not use an unknown token for out of vocabulary words then you might be thinking how does GPT deal with unknown tokens right so then

there is something called as the bite pair encoding tokenizer which GPT models actually use Which breaks down words into subword units So currently what we have done is each word is essentially one token right and each punctuation mark each colon semicolon is one token but what actually GPT does for tokenization it it uses something called bite pair encoding or BP tokenizer and that automatically deals with unknown tokens because in that tokenizer one word is is not essentially one token but even the words are broken down into subwords to Generate individual tokens uh we'll come to

this in the next lecture but I wanted to show this lecture specifically for showing you how you can do your own tokenization from scratch I could have directly showed you bite pair encoding with GPT users but then that you would not have appreciated uh if you were to tokenize yourself how would you do it from scratch of course we'll look at the bite pair and coder tokenizer in a lot of detail in the next lecture to Understand how GPT actually tokenizes but I wanted to give you this intuitive feel so that you don't think tokenization

is a hard process I'll be sharing this code file with all of you so that you can run this and I actually highly encourage you to run this for a different uh book or a different txt file so that you get the hang of it and you become better and better and better at this um just before concluding the Lecture I want to go through a quick revision of what all we have learned in today's lecture so in today's lecture we essentially looked at this data preparation and sampling stage in building large language models from

scratch in particular we took a look at tokenization and uh the question is that how do you prepare input text for training llms and we saw that this is divided into two steps in step one we split the text into individual words so You have a essay or a book or a huge number of books you split that text into individual words and then you convert these tokens into token IDs now when GPT actually uses tokenizers it doesn't use individual words as tokens it even does subwords and we'll look at that in the next lecture

so if you look at how llms are actually trained you take the input text you tokenize it into words you convert these words into token IDs and then the Later step is we have to convert these token IDs also into to Vector representations which we will come to later but in today's lecture we mostly looked at step one which is tokenizing and step two which is converting tokens into token IDs right so initially what we did is we downloaded and loaded the data set in Python we tokenized the entire data set using Python's regular expression

library and we used re. spit we first used it used it at White spaces And then we added colons semicolons question marks Dash Dash character Etc because we wanted to split all of those punctuations and even if you search the dash dash in this text you'll see dash dash appear so many times so even we wanted to split that into individual tokens awesome so this is what we did and then what we did is we maintained the vocabulary we converted these tokens into token IDs so we saw that what a vocabulary is that is it's

Simply a dictionary um and it's a mapping from the tokens into the token IDs so the tokens are arranged alphabetically and then each token is mapped to a token ID as you can see over here so each unique token is mapped to a unique integer called token ID and why are these token IDs necessary because you will see that these token IDs are then converted into token embeddings which are fed as input to the GPT um great now then after that we implemented a tokenized class in Python so that whenever we create a vocabulary we

can create an instance of this tokenizer class uh but we did not just Define the encode method in this class we defined one more method which is the decode method so what the encode method did is that whenever sample text is provided it converted it into tokens and then converted these tokens into token IDs great but the decode method did Exactly opposite it took the token IDs as an input it converted the token ID into tokenized text and then it recover the sample text from the tokenized text why is the decoder important because the GPT

output is also in token IDs so we need to convert it back to the sample text to to make sense of what the output is so the tokenizer class which we defined had two methods the encode method and the decode method finally we saw a problem that if some word is Passed to the encoder which is not present in the vocabulary then the encoder throws up an error to avoid this we need to add special context or special text tokens to the vocabulary and uh one such special text token is unknown so if some word

is encountered which is not known we add the unknown token uh the second special context text token which is also used in GPT is end of text so whenever there are multiple text sources when we feed the text to The encoder we can separate them with the end text tokens we later saw that many researchers used other to other special tokens also like beginning of sequence end of sequence padding Etc actually when GPT was trained it only used the end of text uh token to distinguish between the different text sources awesome so then we then

we saw that with these Special tokens even if you are given a new sentence whose words are not known uh such as the quickly Year which was not known it is just replaced by the token ID for the unknown token no error is shown up uh now we saw that uh GPT did not actually use the unknown token so let me take you to the end of this book end of this notebook so GPT models did not use the unknown token then you might be thinking then how did GPT models deal with Words which are

not known they deal with Words which are not known by using something called bite Pair encoding in bite pair en en in bite pair encoding every word is not a token words themselves are broken down into subwords and then these subwords are the token so let's say you have a word which is uh which is chased right in the vocabulary which we have developed this chased itself is one token but in bite pair encoding it might be possible that this chased itself is broken down into three sub tokens CH as this is just for example

but the this is what I mean by subw tokens uh so subword tokens means breaking down one word itself into subwords and then using the subwords as the tokens uh so GPT models use a bite pair encoding tokenizer which breaks down words into subword units and we are going to cover that in a lot of detail in the next lecture so uh thank you so much everyone For uh sticking with me for this lecture and it's been a long lecture I think it's more than 1 hour but uh I think it is worth it I've

not seen this much of a detailed treatment on tokenization in any of the lectures many of the lectures are toy lectures which means they just give you the basics but then don't show you things like end of text dealing with unknown words padding then pite pair encoding or some things which are actually used in llms and that is what I Wanted to cover so my lecture style will be a mix of whiteboard writing on whiteboard and uh also showing you code here in Jupiter notebook I'll be sharing this code file with everyone so you can

run this code and I highly encourage you to run this code on your own thank you so much everyone and I look forward to seeing you in the next lecture