PandasAI - Generative AI Capabilities in Pandas via OpenAI ChatGPT / LLM models

in this video we're going to take a look at pandas AI which is a new python library that extends pandas with some of the capabilities of large language models such as chat GPT I've got the GitHub repository open and as you can see in the top right at the description here it makes data frames conversational you can ask them questions and you're going to get answers based on the data and based on a request that's sent to the openai chat GPT model and this can cut down a lot on the amount of coding that you need to do you can just take your data frame and you can ask it a question essentially and it's going to return some sort of answer and that's a text based answer that's coming back from chat GPT so if we scroll down we're going to see some of the details of this Library as you can see it's designed to be used in conjunction with pandas it's not a replacement for it and if we scroll down further we get an installation section you can install pandas AI using pip and we're going to focus on this usage section at the beginning of the video so what we're going to do if we look at the codes here you can see that we're importing Panda's EI as an object from this library and creating a panda's data frame and this is just a normal data frame that you would use in pandas it contains columns and it contains rows of data underneath that what we're doing is we're instantiating a large language model on llm so we import this openai object and instantiate it and store it in this variable and then we pass that to this pandas AI object coming from the library and that gives us back an object here that we can then call a run function on and we can pass any pandas data frame to the Run function along with a prompt and this prompt is similar to what you may enter on the chat GPT web interface for example from this data set the prompt is asking which are the five happiest countries from that data and you can see below that it returns a series of countries from top to bottom of the happiest countries in that data so you can ask pandas AI to perform these queries and they can be more complex queries such as what is the sum of the gdps of the two unhappiest countries and that will give you back a number which is just the summation of those two values so the pandas AI object smart enough when you call this run function to actually be able to interpret the data and give you back a textual or numerical response and we can also ask pandas AI to draw graphs as well in this one we've asked it to draw a histogram of countries showing for each one the GDP and to use different colors for each bar in the histogram so we can get some output that's quite complex including visualizations from this pandas AI Library we're going to see how we can do this with a different data set in this video and what I'm going to use is this Titanic data set here and I'm going to leave a link to this below the video and you can grab this data set if you want to follow along with this tutorial I'll leave a link to this page if you want the raw data you can click this RAW button here and then you can copy the code here or copy the text into a local CSV file and that's exactly what I've done to prepare this video I've got a local file called titanic. csv and it contains the data and this data represents the passengers that were on the Titanic ship and it has a bunch of columns which will look more closely at very soon so what I'm going to do do is go back to this page and I'm going to create a new python notebook here within that directory that contains our CSV file and to get started I'm going to install pandas Ai and we can do this with this command here pip install pandas Ai and don't forget the exclamation mark which will tell the Jupiter notebook that we're running this command here we can run that command and install it into the Jupiter environment once that's been successfully installed I'm going to clear this out and I'm going to import pandas as PD that's a typical import of pandas and then I'm going to go back to the documentation and we're going to scroll up here and I'm going to bring in this pandas AI object from this new library here let's execute that and if that completes successfully you know that the installation has been successful now to begin with we're just going to use pandas in a normal way to read in the CSV of data so let's go back to the notebook here I'm going to create a variable called Data frame and that's going to be equal to pandas dot read CSV and we can pass the name of that file it's titanic. csv and then we can check the heads of that data frame that's the first five rows and you can see we're getting back the data now let's have a quick look at the columns we've got a column here called survived that's a key column in this data it tells us whether or not the passenger survived the disaster that happened on the Titanic and we also have columns for which class they were in on the ship as well as their six and their age and also the fear of the ticket the actual price that they paid to be on the ship so we're going to use Panda's EI to ask some questions of this data and also to plot some graphs later on in the video so I'm going to create a new cell below here and what we're going to do in a second is actually create an open AI object from the library as you can see here we create this open AI object but I'm going to scroll down a little bit I'm going to go to this section here on environment variables now in order to use this and call the openai API we need to actually get an API key from openai now I'm going to go to the open AI webpage here and you can see we have the ability to sign up on this page and we have a page here for API keys and I've already created this key here and I've got that copied to my local directory you can create a new secret key using this button here if you need to but once you have that key you can go back to the Jupiter notebook and we're going to create this object Now by copying the code that's defined here so I'm going to copy this line of code that creates a large language model object and we're going to replace this with our own API key so I'm going to paste a key in here and I will remove this key after the video we also need to import this openai object from pandas AI so I'm going to go back to the documentation and let's scroll up again and grab this import here and we can go back to the notebook and above this line of code I'm going to paste that in and then we're going to run this code and hopefully that'll complete successfully so we've instantiated an open AI object what we now need to do is create another object and we'll call this pandas EI and that's going to be equal to the object that we copied in from the top here Panda zi we're going to instantiate that and pass the large language model to that object so let's run this code and below that I'm going to create a new cell and we're going to now run queries against this data frame and that's the data frame of Titanic data that we read in above here so what I'm going to do is use the pandas AI object and to that we can call our run method and we pass as a first parameter the data frame itself and the second parameter is a keyword argument called prompt and the prompt takes a bit of text like the text you might provide to chat GPT so what I'm going to ask is which sex was most likely to have survived let's now run this query against the data frame and hopefully we're going to get some output below this cell and you can see the output here below it's likely that females have a higher chance of survival so we're going to check if this is an accurate statement in a second but we can also add more content to this query for example we could ask how much more likely were females to survive in this data once that's completed running we're going to see some output below the cell and you can see it has output some text it's more likely that females survived with a survival rate of 74.

2 percent compared to males who had a survival rate of 18. 9 percent it's actually giving us some numbers back now so what Panda's AI is doing is taking the data frame of data and it's taking the prompt that we're giving it and it's then using both of those to call the openai large language model which is returning a statement that contains some raw numbers it must somehow be reading the columns from this data frame and sending them to chat GPT so let's now run some code below this cell to check whether these numbers are accurate so I'm going to create a data frame of just the males in the data set so we're going to take the original data frame and we're going to index in at all rows where the sex is equal to mils and we can do that with that syntax there that will give us back another data frame and in the sex column you can see that every one of these rows has the sex of male so what we can now do is check how many of these meals survived the disaster so let's index in at that survived column and because these are 0 and 1 values we can simply use the sum function to get the number that survived and then we can divide that by the length of the whole data frame of males and that will give us a number you can see it's 0. 18 and essentially this is 18.

9 which is the number that was given back by Panda zi when we sent that prompt so the statement that we got back was accurate for males let's now check for females I'm going to create a new cell here and paste this code in and we can change the name of the data frame to females and we're also going to change the Boolean expression of course we're looking for a female now so if we run this code you can see we get back our number 74. 2 percent which is exactly what the prompt gave us back or rather Panda's AI gave us back so you can see that this Panda's AI Library it's intelligently Computing these numbers and it's also returning back some text from the language model and it provides an easier alternative way of querying your data frames using text prompts and it integrates pandas with the openai chat GPT models so let's ask a few more questions then I'm going to create a bunch of cells here and I'm going to paste in some questions so for this one I'm asking the data frame what can you tell us about the class of the passengers is there a correlation between the class and the survival rate so let's run this code and see what it outputs and the answer it's giving us here is that based on the data there is a correlation between class and survival rate passengers in first class had a survival rate that was higher of 0. 63 in second class that was down to 0.

47 and passengers in third class they had the lowest survival rate of 0. 24 so we're getting this useful output here from just asking a text-based question to this data frame let's see a few more examples of that if I've pasted in this code here and the prompt in this case is what is the youngest and oldest age in the data so that's just kind of some summary information and you can see from the output that the youngest person was 0.