Agent Experience with Matt Biilmann, Sebastian Witalec, and Charles Pierse - Weaviate Podcast #116!

hey everyone thank you so much for watching another episode of the weeva podcast we've got another awesome one diving into agent experience firstly I'm super excited to be joined by two members of the wva team making his debut on the weeva podcast I'm super excited to welcome Sebastian whtc Director of Education and developer experience at weeva Sebastian's been one of the core leaders of we8 from an extremely early stage doing all sorts of things the development of we8 including a super awesome collaboration with Andrew and G uh Sebastian thank you so much for joining yeah

absolutely yeah thanks for having me I've always been watching your podcast but this is my chance to actually say something smart hopefully uh I will succeed we'll see awesome thanks so much Sebastian and I'm also super excited welcome Charles Pierce director of The Innovation Labs team at WEA back to the podcast again Charles thanks so much for joining thanks Conor good to be back uh I think I'm I'm starting to to rack up some appearances now so I'm going to go for that second place medal after you yeah our analytical query on awesome and rounding

out the roster for our podcast on agent experience we have the man of the hour Matt bman co-founder and CEO of netfi Matt's article introducing ax why Agent experience matters really struck ACC chord with us and we couldn't be more excited for this opportunity to discuss it with you and learned more about all the awesome work you're doing at netfly Matt thank you so much for joining thanks for having me great being here awesome so uh I know uh probably we a bit about how Charles and Sebastian are approaching this but Matt I'd love if

you could kind of kick us off by explaining your journey at netfi and yeah your perspective on on these things yeah I mean for for for context ntif is a is is a front-end web platform that that helps developers build deploy and operate any kind of web property right like we we launched now almost 10 years ago in in in March 2015 at a time where the way you typically build web properties with these monolithic applications where the front end layer was kind of like an afterthought inside a a backend framework at sometimes right like

an back in that time I point this term jamack for this idea of decoupling the front end from the back end and starting to to to treat the web experience layer as its own application and um and we saw that that that if you did that and started using headless cms's and different headless apis for Content data and commerce and transactions and everything then you would have like dedicated teams just building the the the frontend experiences and there would be an opportunity to build a whole new set of tooling for that workflow just centered around

building the web experience so nlii kind of invented this whole frontend Cloud approach of coupling together the idea of of global CDN not just as as a proxy in front of something but as like the hosting and the deployment layer tightly integrated into git operations platforms where developers would naturally be working and we we focused really up Ely around the idea of developer experience right like of if you are building this kind of front-end experience how do we make the path from writing your code code to having something running live in production on a URL

the shortest possible Right like and how do we really make that experience feel unintrusive and intuitive how do we meet developers in the tools they're already using in GitHub and vs code and their command line and and and and really like give them this sense that that that netlify is frictionless for them and and it's there to to help them and that has sort of powered our our growth from when we launched till today where we are we've unboarded almost six million developers to to to the platform with more than 5,000 developers signing up every

every day we are like the S stores and apps running and our Global Edge Network reaches more than a billion unique visitors every month is spanning every kind of of web experience you can imagine from from hobbyist portfolio sites all the way to um to to to massive e-commerce properties things like vs North Face or like global companies like un l.com or um application software like Navan software right like so any any kind of use case any kind of scale and so on and as we started seeing the the whole change happening in the world

around geni and I started formulating sort of our our strategy to that um a a lot of companies started to to to focus obviously on on how GNA is going to start building more and more of the code and and and writing being playing a more and more active role in in in these web experiences and I always felt that nlii role had never been to like come to developers and say this is how you should build your stuff it had always been about like when when you when you build it we'll get it live

instantly we'll scale it we'll make that whole path frictionless right like so I started looking in the same way if if if AI start building this how do we do that for them right like I I started the strategy of like how do nlii become the best deployment platform for AIS and what what what does it mean how is it going to work right started out retooling some of some of our earlier flows specifically for for AI agents we we when when the GPT assistant ler emerged in in in chat GPT we built a nlii

GPT that you can install and if you've install that then anywhere in the GPT ecosystem you can just add mention netlify and deploy whatever you have to a URL and and and really focused on this frictionless path for an agent in the same way that we had focused on a frictionless PA for for user so it meant like no extra sign up step or anything like that just get it on a URL and if the user likes it they can go claim it and add it to their own account and if not it can just

be an Epal URL that goes away after a couple of hours right started giving tooling for for for for other people building agents to to to instantly like add that flows so a c of other gbts like gmir in the chat GPT ecosystem Devin from cognition lab was an early adopter and started deploying to netlify and and latest bold has has done a really great integration into into netlify that we are now also sort of really learning from and saying like Okay what what what do we need to to to improve and it was as

we were on this journey of of working with these agents that that that I co this term agent experience that has really helped us internally thinking about this as a new discipline in a way right like in a similar way that that that we had user experiences the first take on this right like that was really seeing okay as a user of a product there's a holistic experience of this product that's that's everything that goes into like how how how does it appeal to me how is the brand how is my interactions how does this

product serve me how does it work right like for developers we started focusing on the same way of a developer experience that's not just like the product itself but the documentation around the the CLI tooling how how intuitive is it to understand and build on on top of that product and we started seeing okay we really need a new term around agent experience that sort of really centered on the on the holistic experience of nlii right like how how easy are the flows to to act on behalf of a user right like because agents are

always they autonomous in their driving but they're hopefully always doing something because some human somewhere wants them to do something right like so you have to have some of those levers for for for agents to to to do work on behalf of someone in collaboration with someone they understand like it's interesting that we have this whole set of of apis that we think of as made for machines but in reality they're made for developers to to to to to write machine codes and they're not necessarily just they're not written today for for for agents um

so we need to start thinking about like how what is the best way to structure your your programmatic interface in a way that's really great for an llm driven agent right like and and it won't be it won't be the same as what's the best for for a developer to write a workflow based on coupling two tools together right like it's it's more like a UI in the sense that the agents want sort of to instantly work and do things and and they probably need to be structured and and and documented differently right like so

we've sort of eternally really aligned the next phase of netlify around the same level of obsession over agent experience as we had in sort of the the first 10 years of of nlii around developer experience and it's not to give up on developer experience but I just think that inherently in in in a few years you won't be able to deliver good developer experience if you're not using if you're not delivering great agent experience because every developer will be using different agentic tooling to to to build and work on their projects I I I really

love how uh how you explained this and um how how you look at at it right because on one side like me as a developer my thinking is like Oh Yeah you mentioned the the Char PD plugin and everything if I have a service you know it's not a big deal I could just go there and rewrite the plugin but I think the whole idea is that what if you don't have to rewrite that plugin and and there's not just Char PT there's so many other kind of llms out there so could I write my

software in such way so that an agent could come in learn from whatever I'm offering and go like oh this is how you want me to use it and then kind of like go through that layer of like okay I'm going to execute and go now right and you know of course netlify you you guys are a pretty big company and now so you have engineers and everything but there's plenty of small startups that don't have that Bond withd right so maybe they could follow your principles design their apis in certain way and provide this

metadata slash a agentic documentation so that the llm could go like oh there is this brand new service I'm GNA learn in one minute and start using it right and that's that's relevant to another part of our thinking right like so in internally at netlify we sort of have two two tracks of work around this one is netlify ax how do we make nlii better for for for agents the other is what we call industry ax which is really like what are all the stuff we need to figure out across the industry not just uses

right like in order to to build standards for for these kind of things how does better documentation for agents look like is it llm text files is it like a um is it a model context protocol is it like what what what are the pieces that goes into that how do we make the stuff you can do when you have a a website and like if if one of our customers builds a a web app what what are the ways that an agent can discover what that web app can can can do for for them

right so we actually just now today ER rolling out a agent experience. a as as as as sort of an open source uh industry resource where where where we hope we can collaborate across the industry and figuring out like what what is great ax right like and what goes into it and there's both practices more similar to ux or DX press practices around like understanding friction points and so on but there's also like a a pro call liver side to that on like what what tooling what Open Standards do we need to all agree on

to make life easier for these agents yeah I I loved hearing your mention of um lm. Tex because I guess in in a in a way I I I saw that as like maybe a precursor like an initial attempt at doing ax but it kind of still needs to go further and like I I really like this idea that all the standards that come from this like they have to be open source because if it's going to work it has to be like an open source initiative and something defined at that level because it's the

only way we don't know what models are what how this game is going to play out over the next 10 years but it's going to have to be open source with an initiative like this and there'll be this weird like everything in this space is so interesting and weird right like because there'll be this weird things that if we get a certain volume of of documentation around one way an agent should should use tooling and that starts getting indexed and including at all the agents training set they will be more prone to use that right

like so there all these weird random feedback loops going on in in in everything we do in in in this space yeah and that's actually something like you mentioned earlier so you kind of have there's like these two types of users now that you have to cater to there's like the developer or like user user like the human user and then there's the agent user and it's there's probably a balance that still needs to be struck especially in the short term where you satisfy both users like you have you have an you don't want to

if you make things too verbose or too too agent oriented it could degrade the performance for the human user or the experience and vice versa and and and I think it's really important to to always remember that again as I said there's obviously at a dystopian version of the world where those agents just run around and do stuff and and and and we've just given up in some sense right like but in any sort of like human centered version of the world we're building AI to do stuff for us right like and things we want

so there's typically for any agent that are out there doing something it's always doing it because some human is is like has asked it to to to figure something out or solve something or build something and and I think it it's one of the reasons why current apis for example might not be the ideal ways to to always do things because often you need the human to also be shown what is the agent doing right like what is it working with and and and the human might not get a lot out of seeing like a

bunch of like I'm calling all these apis and doing stuff that that sounds really mysterious right like I can try to describe you for it but it's kind of a metayer it's interesting to see right now that probably most most web applications that has a set of apis if you make an experience and on the one hand you just stuffers like the open API spec for that into chat tpt's context and start asking it to do stuff you will probably get much worse results from that than going to operator and asking it to do stuff

through the web UI which means like which in reality means the operator taking like a virtual browser taking screenshots of every step of the web UI and then having multimodal LM interpret that and give commands back and control the guidance and so on right like really R Goldberg's style system but I'm pretty convinced that for most apps that system right now will probably work better than just giving chat TBT and and and and open thei back and hoping it it it can figure it out and part of that is because the interaction with the with

with the users are much clearer and like the user can much better see and understand what's going on right like so I think it's again one of these examples of how as we build out this this agent experience layer we we're going to have to figure out what that that right intermediary like kind of experience for an agent that's still not just a raw set of endpoint but but an actual experience need to look like yeah maybe want to um yeah I've Loved all these I I love Matt your perspective you seem so on the

edge of all these like Cutting Edge interfaces like anthropics model context protoc call obviously the chat gbt plugin and or maybe it looks like an open a open API spec um I for me this kind of like tool definition the Jon like the function calling are are you familiar with that Matt and um yeah yes so I think a lot about like um in documentation will have kind of like a copy and paste Json tool definition that's kind of that's kind of like how I think about the yeah yeah internally like Sean Sean Roberts that

that that leads our um industry ex efforts at at at at nlii we we've been talking a lot about discoverability of those right like again it feels like some somewhere along the road it would we we need to find out like an a web standard for that right like it should be enough ideally it should be enough for for an agent to know a domain and if that domain has like actual like optimized tools for each in should be some way to just query that domain and get the list of of of available tools and

descriptions like ready ready to use for an agent right like um I think that's that that seems sort of intuitive that that that would be the right open way to do this right like tying it to the idea of every domain can have a have a set of tools and there should be an easy discoverable way for them but it's going to require a lot of like cross industry collaboration if we if we there right like because right now all the different agents have their own tool use implementation and there's not sort of like a

a clear standard for like what what what should what should that look like across any re agent and it might be too early to have a clear standard for it right like because everybody built it like everybody trying to get agents to use tools they're also trying to figure this out right I I guess in a sense there's like an incentive as well for for businesses and companies to actually like participate in the creation of this standard because you you you mentioned something about you know how to call and you can call there's calling an

API and then there's calling an API like efficiently and the right way so you know if we're going to enter into this agent agent world we might as well like participate because if you don't the agent might just try and figure out a way to talk to your API and like overload it with like really inefficient ineffective like requests or you could say here's how to like not crash our servers and do this respectively we we see a lot of of that at at nlii with with our customers right like we like if you go

go to developers ny.com you'll find a guide on how to block all AI agents for from from from from your s through Edge functions and so on right like and that that's not just because we think like hey it would be fun to write that it's because that's like really frequent request from customers that are experiencing like overrides because like these AIS just start downloading all kinds of stuff or like or or overloaded end points because they just start hitting them and all all these kind of behaviors that that can suddenly happen when you just

have like tons of different crawlers almost attacking your side right um and and and that definitely feels like an area where where where better standards can get much more control back to whoever builds that side in terms of how do they want agents to to to use it um that if Done Right can be a really like can be have a lot of benefits to both parties yeah but I think because also like quite often we thinking like there is an agent that then goes and talks to my resources right like whether he consumes apis

or metadata or documentation but what it could be uh what if we could have agents that talk to other agents right so you could have a netlify agent then the we agent talks to your agent you know the kind of story get your agent to talk to my agent and we got sorted out right and they could actually communicate with each other in a completely different language that is a lot more efficient or done in a very different way um yeah and there's so much we can do that we didn't think about in the past

when were like human interact with documentation or apis but agents to agents could be yet different thing right definitely I saw someone suggest that almost every R CPI would end up with a SL chat end point where where you can just use API by chatting to it right like that would kind of be also the agent to agent endpoint and so on right like so so I I think there's things there it gets hard with like some parts can get hard when you have two non-deterministic systems now starting to be non-deterministic together right it can

definitely become interesting but but I think there's something to it and I think also in smaller parts right like we've been figuring out we we've been considering doing stuff like this like for example we we we we have a little LM functionality in in in in in nlii workflow that's centered again on on helping reduce friction of of building and deploying which one probably like the most common point of friction is when a build fails for some reason like a developer open support request we try building it it fails and now that developer has to

go look at build logs look at the code try to figure out like why did this build fail and then fix it right like just always can end up easily gulping up like 20 minutes or something right so we put a little bu and in our UI that just says why did it fail they'll take the the the build log the code context run through an llm and give you like a diagnostic and a potential solution um and that's sort of an identic feature for our for our side right like but now we're starting to

think is there a good way to make that available to the agents that are starting to run Builds on nlii right like because they might want to implement all of that them themselves to like if if they're doing Builds on a behalf of a developer to doing all of that debuging and so on right like it might be simpler if we can just give them access to our agent and ask our agent like why why did it fail right like so there you're starting to see sort of these steps towards like agent to agent communication

where you have more generalist agents that that that drive the whole workflow and then smaller specialized agents for solving maybe a specific problem yeah totally right and also like we used to communicate in words right whether they spoken or in writing or something but um like from my understanding like machine models they use Vector embeddings as a way to communicate each other they do they wouldn't even need to talk to each other through words they could use a completely different either language or format or you know it's almost like the six sense that they could

communicate a lot faster and also a lot more precise because that that's one thing I've always been saying like English is a very imprecise language there's so much ambiguity in it but then if you could have something that is extremely precise the the agents could actually use that to talk to each other and then if you got like an error you could actually very precisely go like you need to do this this is that that went wrong you should try that right um there's no need for politeness or anything it's just like let's get this

done you know but it's funny because like in in the end the agents are still like fundamentally some sort of like probability engine based on language right like so so so it's always like in the end they aren't like first principal reasoning machines or something right like they are really like predictors based on on on previous text and language or previous other data sets right like so so it's always interesting this degree to which the agents kind of still ends up meaning a step away from those kind of like when we traditionally have computers talk

together that's kind of how it works right like just some interchange format and and it all runs there right like but it it's it's a good question where it will end up with with with agents how how much of their like their structure coming from being trained in in humans ways of interacting will translate into like them carrying that into computers interchanging in in in ways that are a bit more like humans interacting than than computers interacting Sor oh go Conor yeah oh I a quick transition there like I think um like we these grpc

apis at we V8 that are like these like B streams to access and it yeah it was like one of our Journeys at we V8 is learning this Evolution from graphql to grpc and yeah exactly like Sebastian like this idea of outputting a grpc bite stream to query an API rather than a Json yeah yeah you need to think of like apis or anything just like straight send me the stream of uh yeah stream yeah yeah those kind of things are are getting really interesting as well I I I I had a friend a long

time ago that was working on this whole system for like be before gen right like but we're thinking it through the system of kind of like fundamentally changing from the concept of an API where today we have a set of we typically have some database and then we have like a program in front of that with specific endpoints to do specific things right like he was trying to invert it to like send a wassom to the to the data source and it'll run it against the like it'll run that program against the data and give

you back the the the the result instead right like um and again not sure like that never really became a thing right like but maybe if we start having the sayi driving the the interaction stuff like that that starts becoming really interesting of like here here's like this dense format of of of how you can write a program against our our our app send it to it we will execute it and give you back the result right yeah Tot but there's a scary thing about it though because if you can have agents communicate with each

other or with you in in either human language or through code that you understand you can at least have that false sense of security that you can supervise it so that they don't plot you know the next Matrix move or something while if they talk lit at the level that it's is just beyond us right that could be scary right like to and everything going and also because they're not necess correct right like I think that's another part right like as humans we kind of also in many of these things want to be able to

see if if if the agent SE are doing the right thing right like because they they they're not like often they're not right like often they make mistakes and and do the wrong thing or ask for the wrong data or misunderstand something right like so so if we if we don't have ways of understanding what what is the agent actually doing right what is going on we we we have a very hard sense of like figuring out is it actually like they will always give us answers with a lot of confidence is like this this

is the truth it's like they're not necessarily right yeah I might say maybe you could have like a structured output where the language model outputs the query in a grpc form but then it also outputs a rationale and natural language and I maybe the grpc is so efficient like outputting the you machine or however you encode it right but like some you know bite sequence that's then decoded and like maybe that's still a more efficient amount of tokens right because like guess it's all about like how many tokens you're getting in and out from yeah

yeah cool one other topic I really wanted to talk about was um this kind of like workflow flow with a like like I guess there's kind of like two apis like there's either you're say like you're querying WEA and it's like boom here are the responses you're finished but like you mentioned Matt with like uh you're uh building a website and you're going through like the cicd or it's like it's generally like an API that's now going to run for like 10 minutes or longer yeah yeah I think that's a that that's the thing right

like that a lot of our interactions with with agents it it changes the whole interaction Paradigm in many ways right like from from this click something get a response right like to everything tends to become more async and more um more of these kind of feedback look I'm doing this like now this is streaming back now I'm waiting for this now I'm doing this and so on right like so so that's definitely like one one one of the followup articles on on my ex article was from the founder of Life blocks sort of really sharing

how they've been thinking about that right like because from their perspective part of part of this is going to mean that there's often like in in in lots of the interfaces we build there's going to be agent interactions and and for that holistic experience we're going to have to do much more like almost every app is going to turn into some kind of multiplayer app right like because you're going to have not other people but other agents working and doing stuff that we need to see updated and reflected and so on right like at at

at nlii we we launched this feature called async workloads on top of our cist functions again because we expect that we'll see more and more of that style of programming of like spin up a like a request wait for this process to finish then when that is done do this but also you got to think of retries with exponential back offs for those kind of stuff with one of them fails and so on right like and there's like just all of this bming that that that we're going to need to do to create great experiences

around the way agents work which is different from from from the way most of our API calls andern traditionally have worked right like that tends to be much shorter and more like do this show if you did it right yeah and that motivates well I also just think about how much AI generally is helping educate me on how to even create the software that like runs every so off yeah like async like is like pretty Advanced like you're pretty Advanced programmer if you know how to do this kind and i' I've been for for a

couple of decade now yeah yeah yeah I guess like like how how to set up like a Cron job with with kubernetes said like yeah and then yeah just this whole way of how you then integrate with the so I guess kind of maybe one other question I want to ask about is your view on like the um uh like chat gbt might recommend a particular kind of software like if you don't like so we're trying to like bring our software to chat gbt but if I say hey chat gbt how do I set up

say like a database and it's like hey use postgress because it is like hey but what if they want to use WEA like like how do you think about it yeah I mean that that that is really interesting like for not just for software for e-commerce for everything right like there's kind of like there's two sides of it right like one one one is kind of a whole new kind of SEO right that's really how do you get agents to know what you are and why someone should should should use you right becomes can can

end up becoming really important for for for businesses and the the the other layer maybe the two other layers are ax on the one hand and maybe brand experience on the other hand right like that also both both becomes interesting right like because on the ax side my thesis is that that that tools that invest a lot in in ax and agent experience and make a material difference in the sense of like now they are actually better to use for LMS and agents than than than than other tools they will start getting like an outsized

effect of of of the agent ecosystem sort of making that the bless path that they've seen work for users and that they kind of end up getting trained at this like works when I'm trying to to to solve this task for someone right like and they they will start like that in itself will start being almost the type of of of SEO right like but where where instead of a developer going and researching a bunch of tools if if the agent knows how to work use and provision and set up and interact with one of

them really well they might just ending up up up using that that tool for that for that reason right like because that's the point of lowest friction and the other question becomes around like more of the brand experiences right like you may be more in the consumer world and so on right like but how do you actually make it work worthwhile for people to go discover your web property on its own right like how do we start like there will be some pressure on us to uplevel the level of experiences we build there to make

it worth not just asking agents questions and getting answers but actually going and and and and and connecting with with with a product through for a human experience right like that that that needs to be worth our time to go there so that we build a brand relation and it's like hey I I really want agent like try out this thing because like I I think it look pretty cool right yeah and I think um we we sort of think of like those ages machines as this neutral entities right but there is bias in it

and to be fair there should be bias to some extent so if let's say somebody comes to like the netlify agent right or to the weit agent we have some biases we have certain Partners we like to work with all we have certain best practices that we want people to follow even though we allow you to do these four different things but we know that one way is way better the agent should be more prescriptive and and biased whether it's at the common sense and best practice level or sometimes sometimes at a commercial level I

think it's it's it's natural right that this happens right yeah will be yeah go ahead yeah I I'd say even what yeah like I'm actually more comforted but I it's like how how much more different is like the agent's ability to use your tool and then create something and then get that positive feedback that maybe you know somehow goes back to the gradients and causes it to use your tool more how much does that really differ from the SEO system we already have in place where humans come use your website and they like it so

that you have more you know lower balance rate and things it's kind of the same thing it sounds like it it's a very similar pattern right like but I think the the search experience is just so centered around like very shortlived metrics of of of of success in some way right like where the agent experience can really be centered around did I actually like accomplish something on a behalf of a user or not right that that is insane because in a way you could have like an agent that goes like hey uh I want you

to go and deploy a wiid instance and then they maybe go like yeah let's spin up kubernetes blah blah blah and then they were like to this huge process and then they were successful but the next time around they were like oh I'm just going to create a we cloud account set it up oh I'm done after two minutes so the third time you ask it to do the task they're like I know two ways but the second one was like way easier it was pretty cheap uh yeah I'm gonna go there but it's a

really common problem also to be able to tell like again that's where we probably in the industry need need better toolings for example really typical problems with a lot of these code generating agents is like not knowing what certain like they might have a larger data training set with an older version of an API because it's been around later and and not instinctively use the new version of an API that's typically like when there's a new one it's typically because it's better right like companies mostly launch new ones because they think they can improve them

right like um but often like the the the inherent behavior of the model will almost tend to the opposite right like there's a lot more in it it's training data from from from an API that's been around for a long time than from an API that's been around for a short time right like so how can we build standards like that allows us as providers to sort really guide the agents to like this this is the way you should use our system right now right like for get some of your training set right like and

use this way because like we we figured out better ways and we we sometimes run into that right like and and and um we we we of course working on supplying context files and Cru rules and all of this right like but um but it's still and you'll see the same like super Bas have there like Crystal rolling cont files like conx have done a really good job of both publishing like cursor rules and context files but also like an eval system to figure out if your system is running it right and so on and

I think a lot of us are thinking in V like how do we actually text to the models that can make them really do the right thing but again it's kind of chaos of like llm text Cur rules and just paste this into this file and ad mention it and like there there's there just there's a la of sort of Industry agreement on like what's the canonical way to solve this problem that that we all going to have yeah totally because I think what we're looking at now is like if one of the examples we

talk about was like hey we we used to have this way of doing it but now there's a newer way which is better and we want the agents to know it so we almost need this like a equivalent of a website redirect right like funny enough this week I was playing with because we use nlii we8 and I was creating a netlify redirects which generate a code 204 that then tell all the machines and everything that from now on this is the best way and what I love about it is like when I talk to

someone about 204 they mely know what it means and we need like equivalent of 204 for agents right like if an agent comes in and says like oh I get the a24 it's like there's a new way of doing it like I should abandon it and I try the new way right and I think that's sort of but we need to agree on it like what is that code right and how how you actually tell it to do it yeah there there there's a lot of there's a lot of work to be be done there

and then please please come participate like we we just rolled out agent experience that ex right like guess one of the places we we want to try to create like a a forum for for discussing like what what do we need for for research for articles and SS yeah I I'll definitely join because this is a thing that we are discussing right now right because uh so one of the things that my team is in charge of is our documentation and of course we had to put a lot of effort for on from the SEO

point of view but SEO only solves like a tiny problem in terms of how do I make sure that this specific link is then shown as like the top search result on Google search or like any type of search but that's not going to help me when it comes to agents because we've gone to an exercise where we wanted to use weit so that you could search through our documentation and it wasn't so straightforward because we realized that the way we built our code wasn't very helpful for llms right so it's it's just kind of

that thing is like what are the special maybe meta annotation or special tags that you could have to to help llms and agents to kind of understand and consume and you know like should they be reading straight up text what's rended or should there be even like a special version of something that is like almost compiled to a different language or different format uh and yeah we're like is anyone working on it and I guess like this is it right like this is what we need to do like how this is what we need to

do right like I think we all need that and I heard an interesting just like agent experience example of that from from from Fred the found of Astro the the front end framework right like that that are kind of saying like one one thing that turned out to be interesting is that because they have build a chatbot for their docks they've done a lot of iterations on their docks to make them work better with the chatbot um and make the chatbot get better answers and and if it doesn't give the right answers fix the duck

so when it inexes and that gets the right answers and that has a self-reinforcing effect of like they they can iterate on that to make their docks better for llms and now as they get indexed in training sets llm turns to start starts becoming really good at at using Astro as a as a as a framework right like an astro as a framework starts being probably often better for LMS than many of the other Alternatives out there right like and it's it's it's kind of an aspect of that but but on the third side I

also totally agree right like docs is one of the obvious areas where it feels like we need to figure out what what the best way what the best standard way for an llm to understand the current behavior of our system through our docks outside of just all the iterations they might have in their training set mixed up with other people's articles and outdated right like what is the the most is it some kind of of of of URL with the vector EMB beddings for of of the docks that they can just ask to get the

exact piece they need is it like what what is it like right like and that's something we got to solve more than at an industry level yeah there just so many interesting aspects of this I guess um I kind of going into the podcast I maybe wanted to brag about our PLL request to the gorilla repository which is part of the training data set for meta's llama models where we have our WEA apis and I'm like for in the trading set now as we're talking about I'm kind of like oh what if we change our

apis and and it's got that old knowledge and it's like I promise you we will change apis because like I keep bu bugging our engineers and we we do change our apis and always improve so yeah yeah I think that kind well maybe if I can continue on this point there's like um a lot of people they build new data quering languages and and they extend pandas and obviously like custom SQL dialect is a big thing like you have like data bricks SQL like snow SQL and um but the model already has some bu some

prior on how to use SQL and so it's like not GNA want to use your new SQL that it's just always also so interesting to go like go go go read like the opensource version of Bold and and go read like The Prompt in there right like there just something really fascinating about reading these prompt with all of their important you must never like do this and treat it as really important and I've mean it when I say it's important like all these like like interes ways you got to rankled the models to to to

do the right thing in a specific situation right like it's it is pretty fascinating and and and I still again I really wonder on like their ability for all of us is as as our tools gets more and more adoption from Agents to to be able to to give that kind of right context to those agents on on how to use them right like how that end up looking like and and how many death threats or Promises of tips will will be in those instructions yeah to but I think that this could be something interesting

right because to some extent that llm having a level of knowledge right or like baked in kind of information like you were saying Connor um like in know SQL but then there is another like database that does something slightly different and maybe instead of kind of going like hey this is learn all this SQL from scratch we could kind of go like oh you're an agent you're not a human being all right you know SQL it's just like that except for these three differences right and you could almost go like in in kind of like

just in time sort of uh fine tuning right like say like just just break this three rules everything else is fine and it's the same thing like if you come to we documentation because even for a person that comes in and learns about we8 you you come in and let's say you're a JavaScript developer you never touched an llm so there's so much you need to learn so much you need to read right but then you're an expert you've done this and everything you come in there go like oh I just want to know how

how do I put my vectors in and how how I run a new new vectors is completely different and the same thing I think applies to agents if an agent already has that knowledge then then we could always go like okay you know this thing yes you know this thing yes you know this thing yes yes you know oh this is new okay from now on I use this as a basis and let's move on right it's it's very very spot on and and something really interesting in how developer experience and how aen experience is

fundamentally different right like because with developer experience you always have to assume this like you you cannot assume a specific knowledge right like we have to work in a lot of paths around like what if the person doesn't know about this at all and and and this and Target all these registers from from from beginners in our areas to experts in our areas and so on right like and the whole and and with agent experience it's very different right like because you probably you you know that there's this massive Corpus of of of data and

docks and information that just part of that data set that you can kind of take take for granted and give instructions around you'll see that again like a for bold it's another interesting example of this right like because one of the things that's so strong around bold is is is this idea of web containers they build this like really complex technology to be able to like transpile noes apps to web assembly and run a whole node app directly in the browser that sort of that development environment and before AI in their product they had the

product stack plates right like as like team editing and so on and it was very fast browser based IDE because it runs Direct in the browser but with these like limitations of like node modules that depends on C binaries won't work like these specific like if it tries to run Gip in anel path it want won't H work like I think those limitations are really hard to teach every developer that comes and approach that tool right like you you got to like it's different from from other and some might get it easily right like but

you've got to find a way to explain all of this to developers that might have very different knowledge of like what works in that we simply environment and what not and so on right like whereas once they have bold as an agent they just need that prompt to again to take to to to do the minimal corrections to the knowledge of of the agents who say like can't use these type of things you have to do it's very important that you follow these rules and so on right like but but then it will do the

like once it get that right it will do the right thing and it'll work within that web container environment and they'll get all the benefits right like and that's great example of that level of difference in in in experience for an agent and experience for for for for generic developers that that that really changes yeah that's that that is so true that is so true um and I think the the one thing maybe that is better versus humans like when when we talking about agents um the there are no emotions involved right so they will

never get like annoy or like or anything right like they L just go like oh they're oh this is a stupid API there like okay you want me to do do this way is not how I was trained but I'm gonna follow this and then do it right while if you work with Engineers like some of us are quite opinionated and and then some of us actually hold to our opinions pretty strongly as well right um and that's why the agents kind of could be more malleable right like at the time they go like okay

I'll do it this one time and I'll stay quiet about it so so yeah that's kind of cool um I had also like a on a slightly different topic like a thought of what if you could also have an ability for agents to leave feedback so like you come in you try to do something and then they'll like 10 out of 10 would do it again or H miserably here is what I tried do better next time you know like no hard feelings the proper D way I'll wake up negative comments from Agents n PS

score for agents maybe that'll be the next thing yeah yeah orps rights I love that I love that amazing well Sebastian thank you so much for joining the wbaa podcast so awesome to have you on and Matt thank you so much for joining agent experience such a fun conversation uh to everyone listening I hope you check out the article Linked In the description thanks again see you [Music] [Music]