Local GraphRAG with LLaMa 3.1 - LangChain, Ollama & Neo4j

hi everyone in this video I want to show you how to perform graph with Lama 3. 1 a local running model so first what is graph graph is an approach to perform retrieval alment Generation by taking the relationship of entities and documents into consideration key concepts are nodes and relationships nodes represent entities or concepts extracted from data chunks such as people organizations events or locations in Knowledge Graph each note contains attributes and properties that provide more context about the entity we then Define the connections between nodes these can include various types of associations such as hierarchical like parent child temporal before and after or causal cause and effect relationships also have properties that describe the nature and the strength of the connection when you've got a lot of documents you end up with a nice graph describing their relationship of all documents let's have a look at a very simple example in our data set nodes could represent entity like apple and Tim Cook while a relationship might describe Tim Cook as the CEO of Apple that approach is really powerful but one massive drobic is that it's very expensive to compute since you have to extract entities from each document with an llm and then compute a relational graph this is why it's great to use that approach with a model that runs locally like Lama 3. 1 we're going to use llama in combination with olama in this video and Ne forj as graph database we will create a graph about information of of a large Italian family who owns multiple restaurants in different places so many many relationships to model there to use llama in combination with olama please go to ama.

com create an account it's totally free and then click on the top right on download then you can download olama for your operating system after downloading the installer or if you use Linux then it will automatically install then open a terminal I use the Powershell for that and then you can run the AMA CLI this will be automatically installed and if you run AMA minus minus help you can see that I get an output so AMA was correctly installed the next step is now to run a model so you can get models here in this model section and this in this example we use Lama 3. 1 so we click on llama 3. 1 and then here is the command we can just copy we can choose between three different mods one with 8 billion one with 70 billion and one with 45 billion parameters in size so of course the size differs the smallest one is 4.

7 GB the 70 billion is 40 GB and the largest one is even 231 GB so I will stick with the smallest but to be honest if you really want a good result then try to run it with the 70 billion parameters model but my computer to be honest is a little bit too old to run that model locally okay so to run the model just now click on the model you want copy the command and go to the CLI copy that and this will now download the model if you don't have it yet and then just wait a few seconds so I could now use AMA like this hello how are you and communicate with the model like this so this is not what you want so we just going to close it and use it in combination with L chain okay I'm now in vs code and here on the left you can see multiple files that we will use so first before we go into the code we will set up new forj I created a Docker compos yaml for you so what we're going to do is that is that we will use the ne forj folder inside here is a jar file this is used to make use of a plugin that we need to create our graph so to create our database just run Docker compose up and this will set everything up for you and will just work out of the box Okay this may take a few seconds and after a few seconds you will see that the database is running so we can already go to our uh notebook and first install the required packages so we need Len chain of course we need L chain open ey Leng chain AMA this is just use as fallback and also L chain experimental because the graph solution is currently in in the langin experimental um package we also install new forj and here this wi fi Jupiter graphs is very well suited to display graphs in a jupyter notebook so execute that line of code okay so after installing the required packages we can now import the required classes so we use multiple classes here from length chain like the runable pass through chat promt template an output pass and so on we also import Neo for graph this is in the length chain Community package here in this graphs module we also import chat openi as fallback model for chat o Lama in the langen chain experimental package we've got a graph Transformers module and from there we import the llm graph Transformer that makes use of quite some complicated prompt to convert the data in a way that we can store it in a graph database so we will also import the graph database from NE forj and not only use new for JS graph database but we can also use it as a normal Vector database so we make a hybrid approach we use graph knowledge and also the more standard way to search documents which is currently just use an embedding model and then search for the most similar documents to a special query so this is why we make this hybrid we're also going to use the N package and load our environment variables here in our jupyter notebook in the end file there is an openi API key there is a NE forj URI a NE forj username and a NE forj password you can just use that as it is but in the repository it will just be called n.