How to Build Your Own AI Operating System (Full Stack Explained)

So the whole developer community has been raging about all of these AI agent harnesses over the past weeks already. Tools like OpenClaw, Nano Claw, Zeroclaw, and all the types of claws that are currently out there. This technology is really one of the most exciting things and recent discoveries really that we've seen in the world of AI since pretty much the introduction of LLMs.

But there is one thing that concerns me about all of this and that is everyone seems to be jumping on all of these out of the box pre-made GitHub repositories. Just clone them, plug in your API keys, give them access to your WhatsApp and credit card and just go with it. And we've already seen countless of reports from leaked API keys, leaked data, and whatever can go wrong when you set up a powerful system like this and you give it access to a whole bunch of your personal information.

So my take on this is pretty similar to what I said when large language models first came out because back then you had everyone jumping on frameworks like lang chain and crew AAI and while these are great to start with often what is what is a challenge with this they're new technologies everyone's jumping on them and because of that they kind of like bloat they blow up and you introduce a lot of unnecessary abstractions so what I did and what I'm doing now instead with tools like openclaw I am looking at the repositories I'm trying to understand what can can I learn from this? What can I do with this? And then kind of like reverse engineer it to fit my specific needs.

Not only will this make you a better engineer, you also end up with a codebase and a repository that you can actually maintain and understand and tailor to your specific use cases. And that's exactly what I want to share with you in this video. Not a complete tutorial covering everything that may come later, but a conceptual blueprint of how I am building my own AI platform for my company, Data Luminina, from the ground up.

So you can start thinking about how to build something like this for yourself. And now if you're new to the channel, welcome. My name is Dave Ablar.

I'm the founder of Data Lumini. I hold both a bachelor's and a master's degree in artificial intelligence. And for the past 7 years, I've been building custom data and AI solutions for my clients.

And next to that, I run a community with hundreds of freelance data and AI developers. And I make these videos to help you become a better engineer. All right.

Now, before we dive into the architecture, let's just first talk about what an AI operating system actually is. So this week I saw a video presentation by Jensen Huang, CEO of Nvidia where he introduced this slide when talking about what an AI platform or operating system really is. So it starts with the multimodal input and prompts, right?

So text, voice, images, pretty much anything you throw at it, it should be able to interpret that. Now an AI operating system also has memory, both short-term and long-term. We have, of course, the models in there, the actual LLMs that drive the system.

Then the LLMs can also use sub agents which nowadays are pretty much skills in the form of markdown files. Now these agents can then also use all kinds of tools CLI commands, MTP servers through which we can interact with and integrate with external services pretty much anything we can take and bring into context or perform actions. Now generally we also see nowadays that these agentic systems can have some form of computer use.

So they can use a browser, they can uh run terminal or bash commands. So any anything you can pretty much do as a user on a computer as well. And then of course there's files that they can interact with and not only structured data for example in a database that we are pretty much used to when we're building software but now also unstructured data on a file system that the agent can literally crawl through and search and explore.

And now this is a conceptual picture but the question that nobody really has the answer to right now is how do we set this all up? Where does it live? How do we run this?

And most importantly, how do we set it up in a scalable and secure way so that every time we want to add new capabilities to the system, we don't need another GitHub repository deployment and configuration. So when you look at a traditional operating system, the one that you're on right now for example, you have all of these apps and pretty much whenever you want to add new capabilities, you simply install a new app or you reuse some of the other tools that you already have but just for a new use case. Now when you think about an AI operating system, you really want to apply the same principles and have the same mindset.

So you want to install new capabilities and don't rebuild the entire system every time. All right? And so the way I think about it beyond the core building blocks that make up such a system, there are also three architectural layers that you need to take into consideration.

So layer number one over here at the bottom is what I call the triggerbased actions. So these are the patterns where if X happens then do Y. So these are your web hooks coming in your API endpoints and your eventdriven processing.

So for example an email comes in, a new subscriber joins your list and you want to perform an action. Then layer number two are your scheduled workflows. So these are your chron jobs, your recurring tasks.

So for example, every Tuesday 9:00 a. m. I want to do a competitor analysis.

So I want to set off a series of agents that do research that create reports and then report back to me. And then layer number three is the agent layer. So this is user info typically via chat.

So layer one and two are backend processes automated. These run around the clock when you are on the beach they keep running. The agent layer is pretty much when you pick up your phone or any type of device and you give some type of input through whatever integration you set up whether that's cloud code, WhatsApp, Slack, Telegram, that doesn't really matter and that triggers off an agent.

So instead of a very direct if this then that the agent dynamically decides and maybe asks follow-up questions to decide okay what do we need to do? What information do I need to pull in? So it's dynamic and conversational.

And now each layer here of course serves a different purpose. So layer number three is where you spend the most time automating your personal task. So stuff that you normally do that you now do through an agent.

It's user infog. You give input ideas. The agent come backs to you.

You give feedback and you work on this within a loop. But when you're for example creating business automations and processes that need to run in the background, you really want to be looking at layer 1 and layer two because they are going to be more reliable. they can run entirely in the background based on triggers or schedules without needing human intervention.

So start with a layer that solves your most immediate problem right now. And then the beauty of this is if you bring this together in one operating system, they can all work together and your agents can also trigger the web hooks and they can run the chron jobs as well and then use that within their context again because you built it on the same infrastructure. And now before diving into each of the separate layers, let's talk about the big question like where does this live?

Where do you put this? How do you run this? Do I buy a Mac Mini?

Do I put it on a server? Do I run it locally? And the answer is it can all work, but it depends.

So here is how I've currently set up my AI operating system. So I've decided to put everything on a server in the cloud. Everything is managed through Docker Compose.

And as you can see, this is a Python-based back end. So at the core there we use fast API to create all of our endpoints and we use caddy as a reverse proxy and to create HTTPS connections for all of the incoming requests that come in. So this is how we set up our endpoints where every time we integrate with the new system.

Let's say it's email, a new meeting is booked, something happens, I publish a new YouTube video, those all come in via the web hooks, they hit our API endpoint and then once the events come in, we have a simple in-memory radius database that creates a queue for all of the tasks that come in and then we have salary workers actually executing the tasks. So as you can see we have set this up using an eventdriven architecture where we create a disconnect between the direct API requests that come in and the actual processing of the events. I'll talk more about this later.

But the beauty of using celery in this system that you can do through the Python ecosystem is that beyond the API calls and the regular events that you can fire, you can also use celery beat which is essentially a way to schedule chron jobs. And then our salary worker which actually executes everything that can integrate with for example our database where we store some information all of the external APIs and tools. We also have our context hub where all of the files and markdown files and skills live that the AI agents can actually use.

And then I've also actually integrated the cloud agent SDK. So we can actually spin up instances of cloud code and run that in the cloud to perform our tasks. And now the language and the tools don't really matter as much.

You can build this in Go, you can build this in TypeScript, pretty much anything that you are comfortable with. What's more important is that you understand all of the architectural layers that you need in order to orchestrate all of this. And now currently I've set this up in a single repository on a single deployment.

I don't need microservices right now. I just want a simple system where I understand all of the individual building blocks and I know exactly what I need to do if I want to add new capabilities to this system. But who knows in the future this might change because I'm also literally just trying to figure this out.

And then I've also added a simple CI/CD pipeline to the system because I constantly have new ideas and I want to build new workflows. So whenever I build something locally, I can just push it to the main branch. It will trigger the CI/CD pipeline through GitHub actions.

It will auto deploy on the server and then I will get a Slack notification saying that we're good to go again. And then real quick, if you're a developer and you want to learn how we build systems like this and then sell it as solutions to our clients, but you don't really know where to start or how to find that first paid project, you might want to check out the first link in the description. All right, so now let's actually talk about the three layers that make up this system.

And we're going to start with layer number one that is the trigger based actions. So these are your web hooks, API endpoints and eventdriven workflows. So how what this can look like in practice is you pretty much think about all of the systems that you use.

So for me that could be I'm using WhatsApp to send messages. Uh I can monitor YouTube. I have meetings.

Uh people can fill in forms. I have email. And I set up within my API endpoints within fast API in my case.

I set up the endpoints for the web hooks and then through the applications that you use, you can pretty much configure the web hook to say if this happens, send this payload to this API endpoint. That's how you set it up. Then I persist all of those events in the database and then I process them async.

So again, this is the event-driven architecture that we are using over here. And then inside the worker, this can be anything. This for me is Python code, but these are just your functions, your classes, anything, any logic that you want to apply.

Now, when you're building a system like this, it's really important that you have a good folder structure. So, it doesn't become this gigantic mess, but you want to have a way where you have workflows with subfolders and everything has its nice little place. And then one important thing is of course security, right?

So, you want to protect your web hooks. So let's say you have the WhatsApp h web hook where uh WhatsApp meta is sending you uh messages to you just want to add a signature verification on that. Here's just some pseudo code on how that works.

So whenever a new web hook comes in, you just verify the web hook if it's legit to make sure other people cannot send information to your system. Then you want to parse that. So you store it directly into the database so you never lose data.

And then in step number three, you dispatch it to the worker in the background. And in salary this would be salary send task and now this will be processed. So that's how you can handle trigger-based automations.

All right. So then let's talk about layer number two your scheduled workflows. So in the Python ecosystem I recommend to do this using salary beat but any chron job type tool can do this for you.

You pretty much want to decide look what are the actions that we need to do on a regular interval. So you want to generate a daily report. You want to sync CRM data.

You want to recover lost events in this system and then you set the trigger and when do it executes that code. Again, you can do this through a worker if you're using an event- driven architecture which is really well suited for this. And the beauty of this is that it follows the exact same pattern as the web hooks but now just a different trigger, a different starting point.

So instead of a web hook coming in, theuler goes off and it executes the code. But the way I've structured my workflow so the Python bas based code we use it a DAC based system. So a directly asyclic graph notebased system where we can decide first we execute this note then we go to that node and we can pass data along the way.

This is similar to how tools like N8N and Zapier are set up but we do it fully within Python. We've been automating and building client solutions like this for pretty much three years already. And this is how we create all of our AI solutions.

and we plug that right into the system. But again, you can use any type of function, class or structure that suits your way of working. So your scheduled task then basically become a configuration that says at this time you need to do this and then when you execute the code you pretty much import your class or function something like this where you say daily report workflow.

run and it simply executes it and it updates it into the database. So you can see the event goes from pending to completed and you can also see the actual data and the output of that. All right.

And then let's get into layer number three which is actually your AI agent layer to make things dynamic and conversational. So this is really what tools like openclaw made popular right a simple interface which you can interact with through your phone through which you can set these powerful AI agents to work. So here's the flow and how you could set something up like that and also how I how I've set it up via WhatsApp and also via Slack for example.

So it all starts with the message, right? So take your messaging platform and set up so that whenever you receive a message within that platform, you can trigger a web hook. So when WhatsApp receives a message, you send the web hook to and then you link it to your API endpoint that we just talked about, right?

So you want to have the verification process in there. You want to persist it to the database. This is all stuff that we all talked about.

And then you dispatch it to a worker. But this is not just a regular Python function or class, but this is actually your AI agent layer. And now there are various ways that you can set this up.

It can simply be an LLM API call with a series of tools that you can set up with Pentic AI, with Langchain, any of these agent frameworks really. So that could be your AI agent, but you could also actually use the claude code agent SDK for this to really spawn a subprocess and to let your agent run using the claw code harness. So there are various levels really to this all the way from kind of like simple LLM API calls all the way to really claim the most the most heavy AI agent that you can run nowadays.

And then when that agent is complete, it can simply reply back to you and you kind of like close the loop in that sense. So then you get your WhatsApp message back from your agent. So it's the same infrastructure as layer 1.

So web hook based triggers but with an AI agent in the processing step rather than a predefined series of functions and classes. And you can configure these agents however you like also depending on what framework you're using. But one thing that I really liked and that I also implemented in my system that I took from OpenClaw was the soul.

md file. So this is a pretty deep file really describing who you are at the core, your values, your missions, your goals, how you want to show up, why you're doing things. And you can go pretty deep into this and especially if you give this to like a heavy model like Opus 4.

6 six right now and you literally start the system prompt with that soul and then after that have the task instructions you'll find especially when you're chatting with it things are going to get really cool. So this is highly creative and highly personalized like depending on how you want to set this up but you could use something like a soul file and then you have some instructions and then you could just pick any model you have the system prompt and you can give it access to a series of tools. how I have set up my WhatsApp agent right now.

I've kept it kind of lean. So it can do a couple of things. So it can right now do a web search.

So it has web tools. So I can for example say hey do some research on XYZ and it will start to do this. I also actually have a tool to save content ideas.

So whenever I for example like hey this would be a cool YouTube video to make. I could open up WhatsApp literally talk to is like hey I have an idea to make a video about XYZ. Can you store it on the file system?

And then when I am behind my computer I can simply pick it up and start reading through it. But the most powerful one that I have in there right now is delegate tasks. So this is where you can spawn a full subprocess of cloud code and it can do anything that you can do on your own behind your computer with cloud code.

It can do that on the cloud in its own subprocess. Now you can even set like a max turns and max budget to this because this can get pricey. Like especially if you use something like Opus 4.

6, six. You could literally tell it to like go do research and it could go off like 10 minutes and it might cost you like five bucks. Like for some cases that might be totally fine, but already through quickly experimenting with this I hit like 50 bucks in API cost on this.

So that's also why I'm very careful with this and I'm constantly trying to decide like hey where does this like claw code type of agent fit into the ecosystem and where do I just want to use for example Pentic AI with an LLM and some tools that I use. So all decisions that you can make when you're working on a system like this. So you can set up these agents really however you like.

So if you want quick responses, you can for example use a model like set and just give it access to basic tools and that will work. But then if you want to do the more heavyduty task, you can give it access to cloud opus in a full cloud code runtime. But again be sure to monitor the costs.

So the patterns here are the same and the results are sent back to you via a notification. So that was the whole backend infrastructure of the AI operating system that I am building right now. But let's talk about another important component of this and that is of course the data and the context.

Right. So I have currently set up a both a Postgress SQL database for event data metadata as well as a file system that the agents can crawl through and this is still an area that I'm heavily experimenting with. But currently I've set up another GitHub repository with these files.

And this is the structure that I'm currently using for this. So it has five folders at the top level and it starts with an identity. So this has all the information about okay who I am, mission, goals, values, what it is that I'm doing both from a business perspective as well as personal.

Then there's an inbox folder. This should pretty much always be empty after you've worked on things. So this is where you can dump ideas.

It's pretty much your to-do list, your inbox. And then I have areas. So this is now growing quickly in size, right?

So here you can have content, products, clients, health, pretty much anything that you think about where you want to create AI automations around. I create a folder for that under areas. Now there's also a projects folder where if you're actually building something out or you're doing research, it can be nice to have it in a separate project folder before you put it into an areas folder.

And then there's knowledge. So this pretty much is your knowledge base. So any type of like research SOPs or documents that you gather and collect that don't really are directly rel related to an area that you need to maintain, you can put it into knowledge and then archive.

So pretty much any information or files that you don't really need anymore, you want to put into archives because you can then also instruct the AI agents to not really look into the archive, right? Because they are crawling, they're peeking at files and they're bloating the context window with that. So you want to make sure everything is marked down or close to markdown.

So the AI agent can actually crawl through it. And with this also everything is version control and everything is searchable. And now one interesting thing that I've added to this context layer is a tiered context loading system.

And I got this idea from a project called open Viking. You can look that up. And pretty much what I've done is to every folder I've added an abstract an overview and then you have the full f file system.

So there's this three tiered system that you can go through where level zero is pretty much an abstract. md that is in every folder and this is literally one line. What is this folder and it can scan the whole system.

So so everything that's in this repository in under 2,000 tokens right now because it can simply look for these abstracts and then as it is searching for context and for files it can go into the overview and this is a short description of look what is this about? what workflows does it involve and what are the relationships. So now instead of the agent pretty much crawling through everything like peeking into a file like reading everything bloating up the context window and then figuring out it doesn't need it and then going to the following with this system it will first peek into abstract then into overview and then it decides what it needs in order to get the full files.

Now this is something you can set up for example using your cloud MD files or whatever type of uh tool you're using agent. mmd or rules. So you can instruct agents that work with your file systems to say hey respect this and then this becomes your navigation through the whole file system.

So if we then zoom out one more time you can see that there are two systems but that it's part of one platform and they're working together. So we have the whole back end the AI operating system that lives on the server in the cloud. This primarily lives locally on my computer but I sync it with git repository so it also becomes cloud-based and now I can instruct my workflows whenever a certain chron job for example or a web hook based trigger needs context.

we can pretty much pull that into memory and start working on it while also being able to effectively work on this behind my machine, make manual adjustments and also work with local cloud code for example on this because when I am behind my computer I might also have ideas. I might also spin off some skills and some agents to work on something. So that is how I've created a sync between these two systems.

And now the cool thing is that within this context hub you can also start to heavily use skills. mmd. So the new capabilities that you want your AI agents to do.

So for example, I'm working on a LinkedIn writer, a YouTube packager, a thumbnail creator, an AI pull skill that will literally like scan the internet and everyone that I'm kind of like interested in to give me a report on, okay, what's going on. I have a slide creator. So the slides that you're looking at right now are actually created using a skill.

An agent created all of that. I created the story and the narrative, but I just said, "Hey, look, I just need visuals for this. " And it created this.

And for your AI operating system, you can of course start to think about what are the skills that you frequently use or want to create and start using in a system like this. So then to summarize and how I am building what I call data luminina OS which is my AI operating system. There are three layers.

We have web hooks, we have schedules and we have agents. Everything runs under the same infrastructure and we have a workflow engine that actually does the work. We have events, we have notes and we have results.

Then that whole system can interact with a database. It can interact with external surfaces and APIs. It can actually spawn an AI runtime and run the cloud code SDK.

And then I also have a full monitoring system hooked up to this via Sentry and Graphana. And this is super critical because when you are rapidly expanding on a system like this and a lot of data is moving through it and you're also using AI to build the system and as you know we don't read all of our code anymore like we're way past that point. So yeah, errors will happen and Sentry will catch that and Graphfana will make sure your server stays alive and up to date.

And then finally, we connect all of this to our context hub for our files, our skills, and our identity. All right, so let me wrap up with a few key takeaways if you want to build your own AI platform. So the first thing that I highly invite you to do is to start build from first principles.

And what I mean with this is instead of just going to the next hyp GitHub repository, clone it, download it, and put all your data in it, start to think about what do I actually need? What do I want to build? What stack am I comfortable with?

And then use some of the patterns that I've shared within this video to start and build that from the ground up. Now, of course, use cloud code, use AI, but be in the loop and be precise about how you want to build things. And then once you have a solid foundation, that's when you can go heavy and you can go allin with AI.

The second thing is really start with the layer that solves your most immediately problem. So we talked about the three layers, right? We have triggers, we have schedules, and we have agents.

Now this depends on do you primarily want to automate your own personal stuff that you do behind the computer? Then agents are really well. Use agents with skills.

Set them up so you can run them in cloud code. You may not even need a full dedicated deployment or do you integrate with a whole lot of systems where you have web hooks and triggers or do you want to automate something for a business for your department or for your team? Then you want to be looking more at layer one and two.

Now then the third thing is that persistence is something that is really important when you do this because these agents they produce things events run and you want to make sure you hook it up to a database that leaves traces in the form of in my case events that you can get back to because these processes they will fail they will go they will get stuck and if you have a persistent storage layer that's just going to help you debug and maintain the system over time. All right and then number four is thinking about this long term. So this is not going to be some weekend project that you set up once and then start using it.

No, you want to set something up maybe in a weekend, but then that you can start to build on top of. So that's how you want to think about it. With all of the stuff that's going on in AI and all of the new tools that come out, you want to be flexible.

You want to be dynamic, but you don't want to reinvent the wheel every time a new tool or technology comes out, right? So build the foundation that can grow with you. And a final tip, of course, is context is king.

So your AI is only as good as the context it has. So, make sure you set up a good context hub or system, whatever fits your specific scenario where you can use the skills, the markdown files, it has access to what it needs to know in order to perform the tasks you wanted to do. So, those are the five key takeaways.

And with that, I also want to be transparent to say like look, I'm also literally just figuring this out. It is highly experimental. The setup that I'm building right now is not something we implement for our clients just yet.

There are parts of course of it. the web hook based automations, the triggers, that's all pretty much standard software engineering. But where it gets into the whole interconnectivity of things, having cloud code agents run with skills and markdown files, that is all highly experimental.

And it's also something that I notice within my own processes that I'm trying to automate is that it's in the beginning tricky to get it right and will often kind of like produce stuff that's suboptimal. So you really want to be in the loop there and improve the skills and then also the tools that they are using. Now of course as I continue to experiment and learn I will share my most important lessons here on YouTube with all of you.

But let me already know in the comments is there a particular component of this system that you want me to dive deeper into in a particular video. Maybe the infrastructure layer. Is it how I set up the agents or the file systems that I'm using right now?

let me know down in the comments and I'll consider it for a future video. All right, and with that we have come towards the end of this video. Now, I hope that you find it helpful.

If so, please leave a like and consider subscribing. And if you want to see more of my work and work more closely together with me, you can check out the links in the description. We have our Genai Accelerator course where I teach you AI engineering in 6 weeks and we also have our freelance course where I help you to land your first paid project.

So, if those are things you're interested in, make sure to check those out.