Run your own AI (but private)

I'm running something called private ai. It's kind of like chat GPT, except it's not. Everything about it is running right here on my computer.

Am I even connected to the internet? This is private contained and my data isn't being shared with some random company. So in this video I want to do two things.

First, I want to show you how to set this up. It is ridiculously easy and fast to run your own AI on your laptop computer or whatever. It's this is free, it's amazing.

It'll take you about five minutes and if you stick around until the end, I want to show you something even crazier, a bit more advanced. I'll show you how you can connect your knowledge base, your notes, your documents, your journal entries to your own private GPT and then ask it questions about your stuff. And then second, I want to talk about how private AI is helping us in the area we need help Most.

Our jobs, you may not know this, but not everyone can use chat GBT or something like it at their job. Their companies won't let them mainly because of privacy and security reasons, but if they could run their own private ai, that's a different story. That's a whole different ballgame and VMware is a big reason.

This is possible. They're the sponsor of this video and they're enabling some amazing things that companies can do on-Prem in their own data center to run their own ai. And it's not just the cloud man, it's like in your data center.

The stuff they're doing is crazy. We're going to talk about it here in a bit, but tell you what, go ahead and do this. There's a link in the description.

Just go ahead and open it and take a little glimpse at what they're doing. We're going to dive deeper, so just go ahead and have it open right in your second monitor or something or on the side or minimize. I don't know what you're doing.

I dunno how many monitors you have. You have three Actually, Bob, I can see before we get started, I have to show you this. You can run your own private ai.

That's kind of uncensored. I watch this, So yeah, please don't do this to destroy me. Also, make sure you're paying attention at the end of this video, I'm doing a quiz and if you're one of the first five people to get a hundred percent on this quiz, you're getting some free coffee network.

Chuck Coffee. So take some notes, study up. Let's do this now real quick, before we install a private local AI model on your computer, what does it even mean?

What's an AI model? At its core, an AI model is simply an artificial intelligence pre-trained on data we provided. One you may have heard of is open AI's Chat GBT, but it's not the only one out there.

Let's take a field trip. We're going to go to a website called hugging face. co.

Just an incredible brand name. I love it so much. This is an entire community dedicated to providing and sharing AI models and there are a ton.

You're about to have your mind blown. Ready? I'm going to click on models up here.

Do you see that number? 505,000 AI models. Many of these are open and free for you to use and pre-trained, which is kind of a crazy thing.

Let me show you this. We're going to search for a model named Llama two, one of the most popular models out there. We'll do LAMA two seven B.

Again, I love the branding. LAMA two is an AI model known as an LLM or large language model, open AI's Chat. GPT is also an LLM.

Now this LLM, this pre-trained AI model was made by meda, AKA Facebook and what they did to pre-train. This model is kind of insane and the fact that we're about to download this and use it even crazier, check this out if you scroll down just a little bit, here we go. Training data.

It was trained by over 2 trillion tokens of data from publicly available sources. Instruction data sets over a million human annotated examples, data freshness. We're talking in July, 2023.

I love that term. Data freshness and getting the data was just step one. Step two is insane because this is where the training happens.

Mata to train this model put together what's called a super cluster. It already sounds cool, right? This sucker is over 6,000 GPUs.

It took 1. 7 million GPU hours to train this model and it's estimated it costs around $20 million to train it and now made is just like, here you go kid. Download this incredibly powerful thing.

I don't want to call it a being yet. I'm not ready for that, but this intelligent source of information that you can just download on your laptop and ask it questions, no internet required and this is just one of the many models we could download. They have special models like text to speech, image to image.

They even have uncensored ones. They have an uncensored version of a llama too. This guy George Sung, took this model and fine tuned it with a pretty hefty GPU, took him 19 hours and made it to where you could pretty much ask this thing.

Anything you wanted, whatever question comes to mind, it's not going to hold back. Okay, so how did we get this fine tuned model onto your computer? Well, actually I should warn you, this involves quite a bit of llamas, more than you would expect.

Our journey starts at a tool called O Lama. Let's go ahead and take a field trip out there real quick. We'll go to O lama.

ai. All we'll have to do is install this little guy, Mr Alama, and then we can run a ton of different LLMs Llama two Code Llama told you lots of llamas and there's others that are pretty fun like Llama two Uncensored or Llamas. Tdrl.

I'll show you in a second. But first, what do we install alama on? We can see right down here that we have it available on macOS and Linux, but oh bummer, windows coming soon.

It's okay because we've got WSL, the Windows subsystem for Linux, which is now really easy to set up. So we'll go ahead and click on download right here from os. You'll just simply download this and install like one of your regular applications for Linux.

We'll click on this. We got to fun curl command that will copy and paste now because we're going to install WSL on Windows. This will be the same step.

So Mac OS folks, go ahead and just run that installer. Linux and Windows folks, let's keep going. Now, if you're on Windows, all you have to do now to get WSL installed is launch your Windows terminal.

Just go to your search bar and search for terminal and with one command it'll just happen. It used to be so much harder, which is WSL dash dash install. It'll go through a few steps.

It'll install Ubuntu as default. I'll go ahead and let that do that. And boom, just like that.

I've got Ubuntu 22 0 4 3 lts installed and I'm actually inside of it right now. So now at this point, Linux and Windows folks, we converged. We're on the same path.

Let's install alama. I'm going to copy that curl command that alama gave us, jump back into my terminal, paste that in there and press enter. Fingers crossed, everything should be great.

Like the way it is right now, it'll ask for my pseudo password and that was it. Oh, LAMA is now installed. Now this will directly apply to Linux people and Windows people.

See right here where it says Nvidia GPU installed. If you have that, you're going to have a better time than other people who don't have that. I'll show you here in a second.

If you don't have it, that's fine. We'll keep going. Now let's run an LLM.

We'll start with llama two. So we'll simply type in, oh Lama run, and then we'll pick one llama two and that's it. Ready, set go.

It's going to pull the manifest. It'll then start pulling down and downloading Llama two. And I want you to just realize this, that powerful LAMA two pre-training, we talked about all the money and hours spent.

That's how big it is. This is the 7 billion parameter model or the seven B. It's pretty powerful and we're about to literally have this in the palm of our hands in like 3, 2, 1.

Oh, I thought I had it. Anyways, it's almost done. And boom, it's done.

We've got a nice success message right here and it's ready for us. We can ask you anything. Let's try what is a pug?

Now the reason this is going so fast, just like a side note, is that I'm running A GPU and AI models love GPUs. So lemme just show you real quick. I did install alama on a Linux virtual machine and I'll just demo the performance for you real quick.

By the way, if you're running a Mac with an M1, M two or M three processor, it actually works great. I forgot to install it. I got to install it real quick and I'll ask you that same question.

What is a pug? It's going to take a minute, it'll still work, but it's going to be slower on CPUs and there it goes. It didn't take too long, but notice it is a bit slower.

Now if you're running WSL and you know have an Nvidia GPU and it didn't show up, I'll show you in a minute how you can get those drivers installed. But anyways, just sit back for a minute, sip your coffee and think about how powerful this is. The tinfoil hat version of me stinking loves this because let's say the zombie apocalypse happens, right?

The grid goes down, things are crazy, but as long as I have my laptop and a solar panel, I still have AI and it can help me survive the zombie apocalypse. Let's actually see how that would work. It gives me next steps.

I could have it help me with the water filtration system. This is just cool, right? It's amazing.

But can I show you something funny? You may have caught this earlier. Who is network?

Chuck? What? Dude, I've always wanted to be Rick Grimes.

That is so fun, but seriously, it kind of hallucinated there. It didn't have the correct information. It's so funny how it mixed the zombie apocalypse prompt with me.

I love that so much. Let's try a different model. I'll say bye.

I'll try a really fun one called mytral. And by the way, if you want to know which ones you can run with Llama, which LLMs, they get a page for their models right here and all the ones you can run, including llama two, uncensored Wizard Math. I might give that to my kids actually.

Let's see what it says. Now who is Network Chuck? Now my name is not Chuck Davis and my YouTube channel is not called Network Chuck on Tech.

So clearly the data this thing was trained on is either not up to date or just plain wrong. So now the question is cool, we've got this local private ai, this LLM, that's super powerful, but how do we teach it the correct information for us? How can I teach it to know that I'm network Chuck, Chuck Keith, not Chuck Davis, and my channel is called Network Chuck.

Or maybe I'm a business and I want it to know more than just what's publicly available because sure, right now if you downloaded this lm, you could probably use it in your job, but you can only go so far without it knowing more about your job. For example, maybe you're on a help desk. Imagine if you could take your help desk's knowledge base, your IT procedures, your documentation.

Not only that, but maybe you have a database of closed tickets, open tickets. If you could take all that data and feed it to this LLM and then ask it questions about all of that, that would be crazy. Or maybe you wanted to help troubleshoot code that your company's written.

You could even make this LM public facing for your customers. You feed information about your product and the customer could interact with that chat bot you make. Maybe this is all possible with a process called fine tuning where we can train this AI on our own proprietary secret private stuff about our company or maybe our lives or whatever you want to use it for, whatever use case is, and this is fantastic because maybe before you couldn't use a public LLM because you weren't allowed to share your company's data with that LLM, whether it's compliance reasons or you just simply didn't want to share that data because it's secret.

Whatever the case, it's possible now because this AI is private, it's local and whatever data you feed to it, it's going to stay right there in a company. It's not leaving the door. That idea just makes me so excited because I think it is the future of AI and how companies and individuals will approach it.

It's going to be more private. Back to our question though, fine tuning, that sounds cool. Training and AI on your own data, but how does that work?

Because as we saw before with pre-training a model with mata, it took them 6,000 GPUs over 1. 7 million GPU hours. Do we have to have this massive data center to make this happen?

No. Check this out, and this is such a fun example, VMware, they asked chat GPT, what's the latest version of VMware vSphere? Now the latest chat GPT knew about was vSphere 7.

0, but that wasn't helpful to VMware because their latest version they were working on chat hadn't been released yet. So it wasn't public knowledge was vSphere eight update too. And they wanted information like this internal information not yet released to the public.

They wanted this to be available to their internal team so they could ask something like chat GBT, Hey, what's the latest version of vSphere? And they could answer correctly. So to do what VMware is trying to do to fine tune a model or train it on new data, it does require a lot.

First of all, you would need some hardware servers with GPUs. Then you would also need a bunch of tools and libraries and SDKs like PyTorch and TensorFlow, pandas, MPI side kit, learn transformers and fast ai. The list goes on.

You need lots of tools and resources in order to fine tune an LLM. That's why I'm a massive fan of what VMware is doing right here. They have something called the VMware private AI with Nvidia, the gajillion things I just listed off.

They include in one package, one combo meal, a recipe of ai, fine tuning goodness. So as a company it becomes a bit easier to do this stuff yourself locally. For the system engineer you have on staff who knows VMware and loves it, they could do this stuff, they could implement this and the data scientists they have on staff that will actually do some of the fine tuning, all the tools are right there.

So here's what it looks like to fine tune and we're going to kind of peek behind the curtain at what a data scientist actually does. So first we have the infrastructure and we start here in vSphere, VMware. Now if you don't know what vSphere is or VMware, think virtual machines, you got one big physical server.

The hardware, the stuff you can feel, touch and smell. You haven't smelled the server, I dunno what you're doing. And instead of installing one operating system on them like Windows or Linux, you install VMware's, EA XI, which will then allow you to virtualize or create a bunch of additional virtual computers.

So instead of one computer, you've got a bunch of computers all using the same hardware resources. And that's what we have right here. One of those virtual computers, a virtual machine.

This by the way is one of their special deep learning VMs that has all the tools I mentioned and many, many more pre-installed, ready to go. Everything a data scientist could love. It's kind of like a surgeon walking in to do some surgery and like their doctor assistants or whatever have prepared all their tools.

It's all in the tray laid out nice and neat to the surgeon. All he has to do is walk in and just go scalpel. That's what we're doing here for the data scientist.

Now talking more about hardware, this guy has a couple Nvidia GPUs assigned to it or pass through to it through a technology called PCIE Passthrough. These are some beefy GPUs. I notice they are V GPU for virtual GPU similar to what you do with the CPU, cutting up the PU and assigning some of that to a virtual CPU on a virtual machine.

So here we are in data scientists world. This is a Jupiter notebook, a common tool used by a data scientist, and what you're going to see here is a lot of code that they're using to prepare the data, specifically the data that they're going to train or fine tune the existing model on. Now we're not going to dive deep on that, but I do want you to see this, check this out.

A lot of this code is all about getting the data ready. So in VMware's case, it might be a bunch of the knowledge base product documentation and they're getting it ready to be fed to the LLM. And here's what I wanted you to see.

Here's the dataset that we're training this model on. We're fine tuning. We only have 9,800 examples that we're giving it or 9,800 new prompts or pieces of data.

And that data might look like this, like a simple question or a prompt and then we feed it the correct answer and that's how we essentially train ai. But again, we're only giving it 9,800 examples, which is not a lot at all and is extremely small compared to how the model was originally trained. And I point that out to say that we're not going to need a ton of hardware or a ton of resources to fine tune this model.

We won't need the 6,000 GPUs we needed for MATA to originally create this model. We're just adding to it, changing some things or fine tuning it to what our use case is and looking at what actually will be changed when we run this and we train it, we're only changing 65 million parameters, which sounds like a lot, right? But not in the grand scheme of things of like a 7 billion parameter model.

We're only changing 0. 93% of the model. And then we can actually run our fine tuning, which this is a specific technique in fine tuning called prompt tuning where we simply feed up additional prompts with answers to change how it'll react to people asking you questions.

This process will take three to four minutes to fine tune it because again, we're not changing a lot and that is just so super powerful and I think VMware is leading the charge with private ai. VMware and Nvidia take all the guesswork out of getting things set up to fine tune an LLM. They've got deep learning VMs, which are insane VMs that come pre-installed with everything you could want everything a data scientist would need to find tune an LLM.

Then Nvidia has an entire suite of tools sensor around their GPUs, taking advantage of some really exciting things to help you fine tune your lms. Now there's one thing I didn't talk about because I wanted to save it for last. For right now it's this right here, this vector database, post gray SQL box here.

This is something called rag and it's what we're about to do with our own personal GPT here in a bit. Retrieval, augment the generation. So scenario, let's say you have a database of product information, internal docs, whatever it is, and you haven't fine tuned your LLM on this just yet.

So it doesn't know about it. You don't have to do that with rag. You can connect your LLM to this database of information, this knowledge base and give it these instructions.

Say whenever I ask you a question about any of the things in this database, before you answer, consult the database, go look at it and make sure what you're saying is accurate. We're not retraining the LLM, we're just saying, Hey, before you answer, go check real quick in this database to make sure it's accurate to make sure you got your stuff right. Isn't that cool?

So yes, fine tuning is cool and training an LLM on your own data is awesome, but in between those moments of fine tuning, you can have rag set up where it can consult your database, your internal documentation and give correct answers based on what you have in that database. That is so stinking cool. So with VMware private AI foundation with nvidia, they have those tools baked right in to where it just kind of works for what would otherwise be a very complex setup.

And by the way, this whole rag thing, like I said earlier, we're about to do this, I actually connected a lot of my notes and journal entries to a private GPT using RAG and I was able to talk with it about me asking it about my journal entries and answering questions about my past. That's so powerful. Now, before we move on, I just want to highlight the fact that Nvidia with their Nvidia AI enterprise gives you some amazing and fantastic tools to pull the LLM of your choice and then fine tune and customize and deploy that LLM.

It's all built in right here. So VMware Cloud Foundation, they provide the robust infrastructure and NVIDIA provides all the amazing AI tools you need to develop and deploy these custom LLMs. Now it's not just Nvidia, they're partnering with Intel as well.

So VMware is covering all the tools that admins care about. And then for the data scientists, this is for you. Intel's got your back data analytics, generative AI and deep learning tools and some classic ML or machine learning.

And they're also working with IBM, all you IBM fans. You can do this too. Again, VMware has the admin's back.

But for the data scientist, Watson, one of the first AI things I ever heard about Red Hat and OpenShift, and I love this because what VMware is doing is all about choice. If you want to run your own local private ai, you can. You're not just stuck with one of the big guys out there and you can choose to run it with Nvidia and VMware, Intel and VMware, IBM and VMware.

You got options. So there's nothing stopping you. It's not for some of the bonus section of this video and that's how to run your own private GPT with your own knowledge base.

Now, fair warning, it is a bit more advanced, but if you stick with me, you should be able to get this up and running. So take one more sip of coffee. Let's get this going.

Now, first of all, this will not be using a lama. This will be a separate project called Private GPT. Now disclaimer, this is kind of hard to do.

Unlike VMware private ai, which they do it all for you, it's a complete solution for companies to run their own private local ai. What I'm about to show you is not that at all. No affiliation with VMware.

It's a free side project. You can try just to get a little taste of what running your own private GPT with rag tastes like. Did I do that right?

I don't know. Now L Martinez has a great doc on how to install this. It's a lot, but you can do it.

And if you just want a quick start, he does have a few lines of code for Linux and Mac users. Fair warning, this is CPU only. You can't really take advantage of RAG without A GPU, which is what I wanted to do.

So here's my very specific scenario. I've got a Windows PC with an NVIDIA 40 90. How do I run this?

Linux-based project. WSL, and I'm so thankful to this guy Emelia Lance a lot. He put an entire guide together of how to set this up.

I'm not going to walk you through every step because he already did that link below, but I seriously need to buy this guy a coffee. How do I do that? I don't know, Emil, if you're watching this, reach out to me.

I'll send you some coffee. So anyways, I went through every step from installing all the prereqs to installing NVIDIA drivers and using poetry to handle dependencies, which poetry is pretty cool. I landed here.

I've got a private local working private GPT that I can access through my web browser and it's using my GPU, which is pretty cool. Now, first I try a simple document upload, got this VMware article that details a lot of what we talked about in this video. I upload it and I start asking you questions about this article.

I tried something specific like show me something about VMware AI market growth. Bam, it figured it out, it told me. Then I'm like, what's the coolest thing about VMware private ai?

It told me I'm sitting here chatting with a document, but then I'm like, let's try something bigger. I want to chat with my journals. I've got a ton of journals on markdown format and I want to ask you questions about me.

Now this specific step is not covered in the article. So here's how you do it. First, you'll want to grab your folder of whatever documents you want to ask questions about and throw it onto your machine.

So I copied over to my WSL machine and then I ingested it with this command once complete and I ran private GPT. Again, here's all my documents and I'm ready to ask it questions. So let's test this out.

I'm going to ask it what did I do in takayama? So I went to Japan in November of 2023. Let's see if you can search my notes, figure out when that was and what I did.

That's awesome. Oh my goodness. Let's see, what did I eat in Tokyo?

How cool is that? Oh my gosh, that's so fun. No, it's not perfect, but I can see the potential here.

That's insane. I love this so much. Private AI is the future and that's why we're seeing VMware bring products like this to companies to run their own private local AI and then make it pretty easy.

If you actually did that private GPT thing, that little side project, there's a lot to it. Lots of tools you have to install, it's kind of a pain. But with VMware, they kind of cover everything like that deep learning VM they offer as part of their solution.

It's got all the tools ready to go. Pre-baked again, you're like a surgeon just walking in saying scalpel. You got all this stuff right there.

So if you want to bring AI to your company, check out VMware private AI link below and thank you to VMware by Broadcom for sponsoring this video. You made it to the end of the video time for a quiz. This quiz will test the knowledge you've gained in this video and the first five people to get a hundred percent on this quiz will get free coffee from Network Chuck Coffee.

So here's how you take the quiz right now. Check the description in your video and click on this link. If you're not currently signed into the academy, go ahead and get signed in.

If you're not a member, go ahead and click on sign off. It's free. Once you're signed in, it will take you to your dashboard showing you all the stuff you have access to with your free academy account.

But to get right back to that quiz, go back to the YouTube video, click on that link once more and it should take you right to it. Go ahead and click on start now and start your quiz. Here's a little preview.

That's it. The first five to get a hundred percent free coffee. If you're one of the five, you'll know because you'll receive an email with free coffee.

You got to be quick, you got to be smart. I'll see you guys in the next video.