in this video I'm going to teach you how to do more advanced knowledge retrieval using some techniques in our tool Builder so your agents have access to data from knowledge table all right so we're in our tool Builder and the first thing you'll see is that I've called this tool better knowledge retrieval now what do I mean by better knowledge retrieval well you'll see this blue section at the top of your tool Builder called knowledge this is an obstruction we've built where you can easily do knowledge retrieval in your tool Builder but at a cost
the cost is you have less control so if you want to do some more fine grained finely tune knowledge retrieval if you have an agent maybe a customer support agent that needs to get really accurate answers then you might want to use a technique like I'm about to show you so let's have a look at this tool so this tool obviously as a user input receives a search query so a question from the user now to provide a bit of context here I've spun up a fake application I've called it Sparky CRM so it's a
CRM for electricians and I've generated a knowledge table which has 50 to 100 frequently asked questions from the users of Sparky CRM so we're imagining we're creating like a customer support agent this is the tool we can use to allow that agent to pull answers from their customers from the knowledge table with the frequently asked questions that we have so let's look at how we might do that in the most simple way possible so you might take this search query input you might add a knowledge search step so let's try that now and then we
could select our Sparky CRM FAQ as our knowledge Source we could pass in that search query and we can do a vector search so again the search query is this input up here where we've had this question hey bro wondering whether this can help me on the go so we have a user of Sparky CRM here or a potential user wondering whether they can use the application on site when they're working so they're probably wondering you know is there a Mobile app for example and so we can feed that question into the query pram of
knowledge search and do a vector search so let's run that while it's running I'll do a quick little introduction to Vector search so Vector search is a technique that allows you to do searching based on similarity not just text matching so the normal text matching search you're used to is if you put in a word like on the go it will search all of your fa use for specifically the words on the go Vector search will try to do a bit more of a semantic matching but you can see here it's found one result about
mobile apps so it's done well on that result but it's a bit lost the first result is about offline mode for some reason the third result is about tracking team performance okay two results on mobile but another time tracking one so you can see that some of the results are good it's picked up that semantic me meaning of on the-go this mobile app but a lot of the other results are just kind of nonsense and unrelated and this is sort of a limitation of vector search of knowledge retrieval because it's doing these similarity based matches
but you know it's it's to some degree something that you have to refine to really get high quality results from so how might we refine this well let's get rid of this step and have a look at the tool I built so the first first thing we'll do is plug our search query into an llm step so why are we doing this well when you think about it if we go look at what a question from a user might look like hey bro wondering whether this can help me on the go that's free form text
it's not exactly what you would put into Google search for example and this sort of free form text makes for quite a bad search query it's full of filler words and stuff that might distract the vector search algorithm so the first thing we want to do is use an llm step to convert this free form search query into a more refined search query designed for searching a database again similar to what you would do if you were searching Google so to do that I've done a bit of prompt engineering here you'll see that we have
a very structured prompt this is something we like to do if I toggle off markdown format you'll see I've used triple hashtag here to make these headers so here I've passed in the search query I've passed in some context so use users are asking for help about our software products soft Sparky CRM and then I've given it instructions this search query is free form text that we want to convert into an effective search query for our FAQ database this means it should be concise and remove all filler words that don't contribute to the search finally
I've used a technique that is very common in the prompt engineering World which is I've ended my instruction my prompt to the llm with the header for the thing that I want the llm to generate so I want the L to generate a revised search query so I've put that header at the bottom of my prompt now one little thing I did here a bit more advanced is if you click on this setting you can open the advanced settings for this tool step and here I've used a system prompt where I've told the llm that
it's an intelligent Millennial who is an expert at finding things on the internet so again a bit of prompt engineering magic you can iterate on this sort of stuff but this is basically a way to reinforce the llm what Persona it should try to adopt when completing this task I feel like telling it that's a millennial that's used to Google searching things maybe is a good starting Place back you can you can experiment here and have some fun and see what works for your situation so now if I run this step specifically we'll see that
it takes the search query up here hey bro blah blah blah and converts it into what is actually quite a good search query Sparky CRM mobile features so the L l m has interpreted that by asking about whether they can use it on the go that they are actually talking about mobile features and now this is a way better search query for our Vector search so if we step down into the knowledge search um this time we're going to plug in the llm step. answer so that's lm. answer we're going to use that as the
search query if we run it this time fantastic we'll see that if it has done a very good job now of prioritizing more relevant questions and answers you'll see the first one is about mobile the second one is about mobile this one's about desktop availability this one's about the ability to use it on multiple devices this is really cool these are way better responses now you'll notice two things I'm sure firstly you'll notice we have a second llm step and maybe you also have noticed there's one result that still Maybe isn't that relevant which is
just what is Sparky CRM now this is actually probably going to help the LM generate a better answer but this doesn't answer the question about mobile features so as a second pass essentially of quality assurance here let's use an llm to validate that these results are relevant to the question so here in our final llm step you'll see I tell it about the original search query the revised search query I've put in the results from this Vector search so again here we have the vector search variable and it pumps out a results output so here
Vector search. results and then I've given it instructions based on the search query analyze the information that might help and extract anything that is relevant to the query if nothing is relevant please feel comfortable to return no useful information as a response so I'll explain that piece in a second but essentially what this is doing doing is feeding in the results from the vector search query and telling the llm to evaluate okay which pieces of these results are actually relevant to the question let's summarize that so if we run this step now awesome so here
is our here's our answer Sparky CRM is designed to be accessible Via Mobile with a mobile app available on both IOS and Android devices you can download it from the app store or Google play this makes it suitable for professionals who are always on the Move additionally Sparky CRM can be used on multiple devices this is a really good answer and this is an answer it couldn't have got to based on just the raw knowledge search now again I mentioned that we have this little piece here about feeling comfortable if there's nothing relevant to return
no useful information so why is this important well in a vector search so in a knowledge search our knowledge is powered by Vector database essentially how it works is it will take all of the results in that database and it will rank them based on their relevancy to the query so what that means is it's able to do really cool things with semantics rather than just doing text search and just finding things with exact matches this ranking process means it can find things that have similar meaning very good the issue is it always just ranks
it doesn't know if there's a concept of match or not match and therefore even if none of the results are relevant it will still return the top X results based on some essentially arbitrary ranking so in a situation where you do have relevant results this is fantastic Vector search works really well but if someone asks a question where there's not anything relevant in your FAQ knowledge table it's still going to return data and we need to handle that case so what's an example of that well let me imagine if rather than asking about wondering whether
there's help on the go the user asked a question about whether there's an update coming out in September now I know that there's nothing in that FAQ knowledge table answering this question so the llm could hallucinate we could respond with some nonsense ideally we wanted to identify that it can't answer this question so now let's run through every step again let's generate a better search query cool Sparky CRM September app update that's a good search query let's do a knowledge search and see what it finds as you see it's kind of returned a bit of
nonsense like what is Sparky CRM how can I access Sparky CRM does it have a mobile app that's because it's not been able to find any results so it's just rank them as best as it can and return the top five results from that ranking but none of them have anything really to do with an app update in September so this is why that final step is so important it receives those sort of useless results it knows what the user asked and it should be able to evaluate does this actually answer the question if not
ideally it returns no useful information so there you go that's a technique that you can build into your agent systems using our tool Builder to give them really accurate nonhalogen knowledge retrieval
