Dr Karpathy, one of the leading AI researchers, recently launched Auto Research, a framework that literally makes AI become self-improving. And when applied well, it lets your AI agents and your skills autonomously optimize itself against real world data to optimize anything from cold email campaigns to landing page conversions to content engagement or anything else that can be measured. So in this video I'll show you how to actually use it with best practices, show you how it works with plenty of examples and a simple way you can get started with this in cloud co-work or cloud code today.
So Karpathi introduced this auto research concept or framework a few weeks ago and he built it specifically for uh machine learning but this can be applied to many other concepts in AI too. We've, for example, built our own version of this framework with a plug-in for cloud co-work and cloud code. And we've been using it to autoimprove our skills, our cloud MD to make our second brain more efficient, and we're now testing it on our newsletter subject lines to improve open rates, uh, LinkedIn post to increase engagement, and landing page copy to improve the CTR.
Of course, this framework could be applied to many other things. Now, before showing you exactly how it works, let me show you some examples. So, I'll start with a simple one so you understand the concept.
So here I ran the auto research skill that we created to optimize my LinkedIn writer skill for the following criteria. Optimize a LinkedIn skill to make sure hooks are always under 136 characters and make sure bullet points are always done with hyphens. It then ran one test to identify the baseline score which in this case was 80% or four out of five that passed the two criteria and it then started the optimization loop with a hypothesis on how it could improve the baseline.
So in this case it started with an hypothesis on how to improve the hook criteria. In this case add explicit 136 character limit for hooks in step four in the skill. It then ran the first iteration and the pass rate went up to 100%.
So it decided to keep uh the hypothesis because it was a significant improvement. It then ran iteration two to improve the second criteria with a new hypothesis. So in the end it ran five iterations completely autonomously and the final output was 100% score with a 25% improvement for both the criteria and of course the new optimized LinkedIn writer scale and a dashboard where we can see all the iterations that it ran which ones it kept which ones it discarded and also the overall improvement from where we started.
Now this was a simple optimization of course but we can make this much more extensive and in-depth. For example, here I ran it again on making sure it uses the correct hook templates according to my hook template reference file. And another criteria to make sure it adds nuance in my sentences when I make bold predictions or claims.
Another one to optimize the skill to always follow a proven writing framework like PAS or IDA and making sure that my post contains a personal story for my personal background reference file. It then again first established the baseline which is 68%. It started running iterations with hypothesis.
It then ran 10 iteration loops completely autonomously to get to a 27% uh improvement from the baseline. Again, it gave me the new updated skill and a dashboard with the overview. Now, this can not only be used for skills.
I've also used it to optimize my second brain performance. If you've watched my last video, you'd understand this. For example, here I ran the loop on optimizing the cloud MD for knowledge routing to make sure it saves the data in the right folders to optimize it for wiki link creation and also for knowledge retrieval.
Again, it ran five iterations and got to a 9. 9% improvement and the updated cloud MD file. Now, coming up with a few of these criterias yourself and just listing them out is a great way to improve your skills or your cloud MD fast.
But this becomes especially powerful when we start optimizing and running the loop against real world data. Firstly, of course, we can do this based on past data. For example, in this chat, I pulled my top performing LinkedIn post and my worst performing LinkedIn post together with engagement data.
And then I let Claude identify patterns to come up with a hypothesis or criteria on what to optimize the scale for based on this real world data. So did a LinkedIn pattern analysis um comparing my top performing versus my worst performing ones and then compared a few things like hooks and the hook type versus metrics, the CTA analysis versus metrics and what kind of writing framework was used versus the metrics. Based on patterns it found in these three things hooks, CTAs and writing frameworks.
It then created a new hypothesis on what kind of post would work best. But the next step is of course that we can start using a schedule task together with auto research to potentially make it autonomously optimize it on a weekly basis. So I've set up three of these tasks uh like the LinkedIn auto optimizer, the email sequence open rates optimizer and the auto landing page optimizer.
Now they haven't actually run because I started this this week, but the idea is of course that every week it can set an up hypothesis. For example, this combination of hook, writing framework, and call to action gets the best results. So we'll test those types of posts.
Then the week after it scrapes the data from the performance, keep the change if it led to better results, and keep iterating until it hits a goal. For example, until it reaches an average of 250 average likes. Now, of course, you can also build some human in the loop in here if you're not comfortable with an AI actually running this autonomously.
But for example, in the email sequence open rates and auto landing page optimizer where you can test, for example, H1 copy until it gets to 3% CTR. And you can see in the instructions, it's a set of steps. So, it first has to pull last week's landing page metrics, right?
Then define a new hypothesis for how we can improve the landing page or H1 copy, then make the change on my website and do the same next week. Now, of course, there will be other variables when you're testing this. So, it's not going to be perfect, but you can see the potential for a framework or setup like this, especially with agents becoming more powerful and getting more context around our business.
So even if you're not ready to use it like this yet, it's still great to optimize your skills or cloud MD really fast. Now before showing you how it works and how to use this efficiently, how do you actually set it up? Now the way you can use Karpath's framework is by going to his GitHub repository which I'll make sure to link in the description below too.
And then you can basically download the zip file and once you've downloaded it, you can just unzip it. Then go to cloud co-work or cloud code and just give it access to that file. And then once you have the folder selected, of course, Karpathi's framework is built for machine learning optimization.
So you would have to um go through it with cloud and adapt this to make it applicable and usable for uh skills for example or for the cloud MD or any task or process you want to optimize it for. So you basically just ask something to cloud like can you apply this framework or loop um to optimizing skills etc. And you can start building your own version of it.
You'd need to go back and forth a little bit to actually make this work efficiently. This is how we built a plug-in out of this that helps it optimize for the most common use cases. You can see we have a skill here with auto research and also the agents that do the evals, the judge and our test runners.
So, you can definitely create something like this for yourself if you go back and forth with Claude a little bit. But if you're interested in our plug-in, uh you can also check out my AI accelerator uh where we list all the plugins and skills we're building out for ourselves. We also have multiple weekly Q&A and AI workshops where we dive a lot deeper into these tools and unlimited one-on-one live tech help to help you set up anything you need in Cloud Co, Cloud Code or anywhere else.
So, if that's interesting to you, um, definitely check it out in the first link in the description below. Now, whether you want to build this out for yourself or just use it, it's still important to understand how this actually works. Now, it's actually pretty straightforward.
All it is is just a loop. First, you define the criteria of what you want to optimize. for example, optimize uh my LinkedIn skills so that hooks must always be under 130 characters.
Then auto research will first establish the baseline criteria by running a sub agent who tests the current version and gives a score. Usually it runs a few tests. So for example five you can see here that in co-work it used a sub agent here the auto research test runner to establish the baseline and give a score in this case 80%.
And then the main agent, the one you interact with on co-work creates a hypothesis on how it can improve that score. For example, by adding a role in step four in the skill. Then another sub agent runs a test with the updated skill.
Basically, in order to make sure it's unbiased, for example, here it started testing this hypothesis with another sub aent here. Then there's another sub agent that again is unbiased that evaluates the outcomes of each of the test runs. Now there are two ways he can evaluate the outcome.
It can be through a script which just defines if something happened or not through deterministic logic or a Python script or it can judge the outcome through another sub agent if it's not possible to do it through a deterministic code script. For example, in this case is a simple criteria. So the eval runs through a script.
But if we look at another example where I try to optimize the skill to match the hook with one of the hook templates that I have in my hook reference file, it actually is harder to do it through a script. So you can see here first ran a sub agent to run the test and then the outcome is not judged through a script but through another sub agent which is the L&M judge, right? Auto research judge.
He judges the outcome if it passes or not. And then the main agent gets the results back and decides to keep the change or discard the change depending on if it improved upon the baseline score. It then keeps setting new hypothesis until it reaches the goal or it reaches the amount of iterations you've defined.
If you see my previous video on skills 2. 0 or evals, which is a built-in feature of the skill creator skill in claude, the difference is that this basically automates the entire optimization process autonomously. While with evals uh we have to constantly judge the outcome of an eval and give it feedback for the next improvement.
So instead of that we'll just keep running in a loop until it actually meets your criteria. Now of course this comes with a cost too because if you let it run for hours in terms of optimization you can imagine that the token cost will go up quite a lot. So it's important to actually optimize things and skills that you use frequently and make sure you optimize it for things that make a real difference.
Which brings me to the next point which is the importance of defining good criteria because the skill or the process you're optimizing with auto research will only become as good as the criteria you define. So what's an easy way you can think about formulating good criteria? First of all, whatever you're testing against, keep in mind that the outcome should always be true or false.
That's how the system works. So first uh then you have to state the exact condition, not just the goal. So don't make don't say something like make sure the hook is short enough.
say the first line must be under 136 car characters character. So generally you need a specific number a specific format or a specific pattern it can actually test against and then second of course you need one variable um per criteria. If you need the word and to describe what you're testing then you have to split it into two criteria.
You can still give it multiple of these criteria at the same time because it will go through all of them but each one should be well testable. So separate them into two. Now how do you come up with these criteria?
This of course becomes a lot easier if you actually have real world data experience or expertise around the process task or the skill that you want to optimize. And an easy way to come up with some of these criteria is by either feeding it some of the real that real world data like I did with the LinkedIn post and asking uh clots to spot the patterns and come up with criteria to test against based on good and bad performing posts. But even if you don't have that real world data, just try to feed it good examples or good outputs and tell cloud to identify patterns in them and potentially criteria to improve the skill upon.
Now, if you don't have that data, of course, you can also look at industry best practices or other people's posts or data. Now, there's also a big difference, of course, between optimizing for clear-cut deterministic skills or processes and subjective or creative skills or processes. For example, for an accounting skill that goes through my Stripe, my invoices and uh logs it into my Excel sheet at the end, it's a pretty clear-cut deterministic process that just needs to work well.
And there's not really any external data we can test it against. So, in those types of skills or processes, we can often just run an eval from the skill creator skill. If you haven't seen that video, I'll make sure to link in the description below, too.
Um, to basically make sure that the skill actually works the way we intended the skill to work. But I think the real unlock is using auto research specifically to optimize um processes and skills that have some subjectivity or creativity embedded. And if you think about it, most processes and skills across marketing, sales, customer support or other business departments have some level of subject subjectivity embedded.
But that of course brings up another question. And if we have to give it these very clear true or false criteria for auto re search, how do we actually use it to optimize it for these more subjective or creative processes like a LinkedIn skill? For example, how do we optimize it to match our tone of voice?
Now, if you think about it, even subjective or creative skills like copywriting can actually for a large part be defined by true or false criteria. And the way you can think about it is through this three-level framework. And the first level is what are the clear-cut rules.
It always has to follow according to your or general best practices. For example, on LinkedIn copywriting that could be the hook must always be under 136 characters. Uh second, the sentence sentence length should always be between five and 12 words or a paragraph shouldn't consist of more than three sentences.
These are kind of like hard rules and well-definable and general best practices for LinkedIn copyrightiting. But then the next layer becomes a bit more subjective, right? And this is where we start looking uh more for patterns of copy or your unique way of writing.
For example, I try to optimize my skill to make sure that bold predictions and claims are given with nuanced words like I think, I believe, or I predict because I noticed that AI makes very bold claims in a in a too convincing way. And for these types of tasks and optimization, we generally need that AI judge, right? Not just a script.
Another example for that could be for example to make sure that your copywriting follows one of the major writing frameworks like PAS, IDA, CPF etc. And then for this layer which is a bit more subjective I think uh AI can really help you spot those patterns or criteria I could optimize it for. Now of course there's always going to be some level of creativity or subjectivity in these types of skills that you cannot optimize it for together with auto research.
But the point I'm making is that if you can articulate these roles or let AI analyze your post and identify these patterns or criteria, you can get it to match your tone of voice to a very high degree by just defining these true or false criteria. So that's the way I generally recommend you optimize these skills or processes with together with auto research is by first thinking of those hard rules or best practices, run it through an EVA or an auto research uh and make sure that it consistently nails those. And for some more deterministic skills like my accounting skill for example, this would be all you need.
And then for more creative skills, you can go uh to the more subjective layer while still trying to get those true or false criteria. And once you've done this, of course, the real meta is to start optimizing against real world data. Now, of course, this becomes a bit trickier because there's a delay in the feedback loop which you can solve with the scheduled tasks.
But you have to be comfortable with AI autonomously doing this for you. And that's again why you need to make sure that the skills already really well optimized before you go into this stage. So how do we actually start using it?
Now if you're going to use the plugin that we've built, you can just go into my AI accelerator and download the zip file of the plug-in. Once you have the zip file, you go to customize. You go here to plugins and click plus.
You can click uh click create plugin here and upload the plugin where you upload the zip, right? You just upload the zip and then it will appear here. As you can see, we have auto research.
Now mine was installed of course already. Then we have the skills. Here we have the auto re search skill and three sub agents.
One for the evals, one as the L&M judge and one is the test runner. Then you just go to a new task and you run the skill auto research. It will then ask you some questions if you haven't already specified what you want to use it for.
So in this case, I'd say optimize my LinkedIn writer skill. We've embedded in the plug-in the also the roles for the criteria, right? state, exact condition, one variable, etc.
And then they actually propose me some things to optimize for. But of course, here's where you want to start thinking, okay, what you want to optimize this for. Again, here's also the point where you can start feeding it real world data.
For example, let it analyze your top performing LinkedIn posts. Um, but let's say I just define four criteria here. I want to optimize the LinkedIn skill for these criteria.
So, I just add in four criteria. I let the skill do its work and it will already know what to do. Then it defines your criteria a little bit more in depth.
It's good to read them to actually make sure that they make sense. In my case, they do. So, I just tell them they look good.
And then it asks me which evaluation mode should we use. Now, this depends on if this is a really clear-cut rule or if it needs an AI or an L&M as a judge. Now, you don't really have to make this decision.
You can make the decision, but the system itself will give you a recommendation. In this case, for many of these, we'd need AI as a judge. So, that's why it's already recommended that.
So we just click that and we can decide how many iterations we want to run this on. Now generally there's two ways you can do this is you can either define how many iteration loops it should run or you can tell it to keep running until it hits a certain goal or number. Now generally what I recommend also to in terms of saving token usage is to uh start with five iterations maybe 10 if you have many instead of letting it infinitely do an iteration loop because in my experience if you let it run for too many iterations it actually gets worse after 10 or 15 iterations.
So generally that's sort of the cutoff that I recommend. So in this case I I'd say five. It then scored the baseline results and prioritized the weakest areas, personal story, specificity, and nuance on bold claims.
So that's the first iterations it will try to optimize. You can see that the first hypothesis it's testing is uh add explicit instruction in step five to always include a specific personal anecdote from Ben's background. And as you can see in this first iteration result, it got an improvement on the personal story specificity.
So it decided to keep the change. In the second iteration, it then tried to optimize for nuance on bold claims. It got a 6 improvement.
And again, the result was keep. And it keeps doing this until it hits the five iterations. Now, this is going to keep looping until it does all five iterations.
And then, if you want to take this up a notch, which I can't show you because it hasn't run yet, is to use the schedule task. But in this case, it would work something like this. You're running the weekly LinkedIn optimization loop for BMW.
This is an automated AB testing system that changes one variable per week. hook typewriting framework or CTI CTA type to maximize average like post target 250 average likes. So we just give it some context on the folder structure because we have to save some data.
Um so it actually understands what tests we've run in the past. So first it's reading that context to get the current active experiment details to read the optimization rules to understand allowed hooks frameworks and CTAs. Then of course scrape last week's posts, extract the engagement metrics, calculate the metrics, update the context.
So we save that in the folder and then of course make the decision based on the results. If the average likes went up versus baseline, keep average likes are down, revert and then generate the next iteration or hypothesis which will then be of course scheduled for next week. Now, I'll probably do another video on this when I've experimented a bit more with these scheduled or autonomous optimization loops, but uh I think it's interesting to experiment with.
Now, that's it for this video. Again, if you want access to this plugin on all the other skills and plugins that we're building out, you can check out my AI accelerator in the link in the description below. Um, we also do these weekly Q&A, multiple weekly Q&As, workshops, and unlimited one-on-one live tech help.
We also have a full guide on setting up your AI operating system or your uh second brain. Um, so if that's interesting to you, uh, you can check it out in the link in the description below. Also, if you want to learn more about Claude and skills, etc.