Coding the future of AI | BRK115

[Music] all right welcome everyone thank you for joining our session today my name is Dan Taylor and I'm a product architect working on the Azure AI Foundry I'm Rob Chambers I'm a technical adviser working on the same team with Dan and despite what our titles say both Rob and I are actually developers and you've seen a lot of really cool stuff today and what we want to show you in this talk today is a more practical Hands-On guide of what it looks like to code some of these experiences so that you can build the future

of AI we're in the era of AI development every single application will be reinvented with AI and new apps will be built where things weren't even possible before and there's a lot of people building AI applications today and building with AI there's going to be you know up to a billion applications new built over the next 24 to 36 months uh 70% of organizations are accelerating modernization and bringing AI into their existing applications and 90% of developers are using AI in their tool set how many people here are using AI in their tool set right

now that looks like about 90% as far as I can tell good on you um so but the reality is that the conversion rate from building that proof of concept to getting to production is is still pretty low about 14% of applications make it and you know 74% of organizations are still in the experimentation and plan planning phase and of all the factors that hinder getting uh to ji production uh we're going to focus on the 41% of apps where they lack the developer skills and tools and so what does it look like to build

these applications well here at Microsoft you know we provide some of the world's most loved developer tools between Visual Studio GitHub and co-pilot studio and we want to start off by showing you what it looks like from what what one of our customers ABB has done to build their co-pilot let's go ahead and watch the video AT AP we provide Technologies for manufacturing and Industrial operations all over the world the industries are going through a significant transformation they have been challenged to be more productive efficient and more resilient our digital Flagship product is genic it

analyzes the functions such as asset operational and process performance emissions and Energy savings but one of the biggest challenge is pulling the data from the different Source systems and contextualizing them in a platform approach we chose azour open a service because of its customization safety and dependability and built genix co-pilot to help our customers ask a question in national language and get the answers and make better decisions immediately the customer might ask genic scope pirate what is my industrial emission level today it gives recommendation reduce the cooling fan speed to reduce the energy consumption by

12% customers are seeing the value of AI it's cutting the maintenance cost up to 30% improving the asset reliability by 20% reducing the carbon emissions by 15% bringing the energy consumption down by 20% by integrating azur open AI with our domain expertise and Technologies we are solving some of the most complex problems in the [Music] industries all right what I really love about the Gen X co-pilot that ABB has built is how polished that user experience is and how powerful it is and the the the productivity that it unlocks for their customers um this seems

like some pretty advanced stuff though so where do you start as a developer well I want to kind of break it down to a few simple Concepts and then we'll build from there so at a fundamental level building with AI is you write some code that takes some data it generates a response or a prompt you send that prompt into a model and you get a response out you do something with the response and you keep going right and so as you scale that up what this looks like it turns into having many different conversations

with many different agents with different pieces of code processing that data and extracting that information into that useful user experience that you saw in that video and so the way we're writing code is changing we're going from building deterministic applications with to probabilistic Applications with probabilistic outputs from writing code to writing prompts from building features to creating agents that do useful things and from testing those determin deterministic outputs to evaluating them and scoring them on a scale from one to five and the things on the right aren't replacing the things on the left but it's

a new tool set that you need to integrate into your development processes and so we have the co-pilot and AI stack can help you do all of this at the top we have the Azure AI Foundry which we'll be showing a lot today which Builds on top of your data and our AI infrastructure and integrates into the world's most loved developer tools and so you know as developers we like to start small and what does this process look like what's the road map for building the applications well you take some data and then first you

prove that you can get something to work with a proof of concept then you get your application into development and you're trying to get it to production quality and then once you're in production you're taking data from the actual production app you're feeding it back into development process so that you could continue to improve and optimize the user experience and so talking about building a proof of concept we're going to go through those three phases today but Rob is going to get us started off with what it looks like to build that proof of concept

that's great thanks Dan thanks for setting all that up so oops sorry too soon what I'm going to do is I'm going to talk to to you about how we're trying to meet you where you are with various choices at the different layers inside of the stack so we're going to start off by obviously developers need idees or portals or their shells who who here uses uh an IDE how about uh a a shell like bash or poers shell all right yeah that's great but what you need from us then is you need us to

meet you in those locations at the right times and so we're doing that by making sure we have the correct kinds of IDE extensions in the right shell in the right uh idees that we can meet you in the right places with clis in the shells that you've grown to love at the same time for that to work we need to build that on top of a great set of documentation and repost that have all the samples that you need and it's available in the language that you program in not some other language that you

don't program in all of that then is fundamentally based on top of SDK client libraries and packages and there's a lot of variation and choices there as well so we're going to go through those today with that I'm going to jump in and start showing some of the code so you'll see me at the screen my shell uh this is we're going to in this section we're going to talk about GitHub models we're going to look at across the various idees portals and CIS inside of shells so we're going to start out by looking at

the get GitHub model uh collection at the marketplace so if you go to GitHub Marketplace today you'll find all of the models described you can go and list through the different models to see all the different capabilities across providers you can even filter those providers down you can eventually select one of those and take it into a playground experience you've probably seen some of these earlier today but that you can ask a question something simple like how tall is the Willis Tower in Chicago press enter text in text out like Dan was talking about you

know we get a prediction back out this is really nice for developers both because it's free you can just use it right away if you have a GitHub account and everybody here has a GitHub account right yes yeah and so you can use it for free you can use it simply but it's not inside your IDE or it's not at your shell so what you can do is you can come over here to the VSS code extension that's called the AI toolkit it has a way for you to see these same models actually has a

few more models in it as well but if I click on this and filter down to the things that are hosted by GitHub I can then filter down on Publishers to things like open Ai and then I can take that and make a entry over here on remote models and come into a playground experience as well so in the same kind of way I can play here just experiment with it change different aspects of it the system prompt instructions or different parameters I can even switch off to other models like the 53 model run that

remotely I can even use the FI model running locally as well so all of this gives it to you right here in this shell or in this IDE we're going to ship back to the uh experience here GitHub has this new models extension to the GitHub CLI and with that too you can list all of your models just there they all are you can then do very simple things like checking out what a single prompt looks like but because it's in a CLI you can do neat things with that data in aspect anything that you

can produce as data in you can just pipe into the GitHub models extension and have it do something so very simple example here is I'm going to look inside this command which is helping me show what I'm going to be showing you but maybe you don't understand that what look what that looks like so I'm going to give that to the GitHub models extension with GPT 40 mini ask it how it works it's going to go through it very detailed and let me know what it is so you can do that with anything you have

very easily data in text out so with that I've shown you that you can use the GitHub models both at the portal you can use them in vs code and you can also use them at the CLI inside the shell of your choice so with that we're going to go into the next section here I showed you a CLI that CLI is specific to GitHub we also have a new CLI called the AI CLI which is an open source repository and to take a look at what that looks like there's this documentation site actually built

mostly by AI itself but it shows you how to install it it works across various things like GitHub Azure AI open AI you can use Onyx also Frameworks like semantic kernel or doing things with speech things like that so we're going to come back to the Shell window here and you can take a look at the various things you can do it's got built-in help so you can either use the website to get the help or you can do the help right here in the CLI at the Shell one of the things that's really neat

to do here that's not available in the gith head models extension yet is to do conversation inputs and outputs so I'm going to do this tell me a joke it'll tell me a joke I exit and I take a look at the conversation history this is that conversation in conversation out that we saw so then I can take that and just modify it from output to input and say another and it already has that context so you can actually save that off and then reuse it in different ways once you have that set up all

right additionally in addition to the the chat history there are these things called function calling who hear uses function calling today much smaller number you're all going to want to use this after this talk so what you do we're going to show it to you here in the CLI we're also going to show you how to write the code so you just run it like this you can see that it looked at the directory I asked it like what is in this directory I can say what is read me trying to tell me be brief

I love that the AI can overcome your typos you know exactly standing on stage right so it it actually described the various parts of what is in here and what we're going to try to do so one of them it tells us how we're actually can build a sample application using JavaScript let's go ahead and do that to do that you can list a bunch of different uh samples that are built into the CLI across all kinds of scenarios you can narrow it down to a particular language or a particular set of capabilities but what

we'll do is we're just going to focus here on creating one sample this one and then we're going to start installing stuff PM install while that's happening we're going to go back to the book of AI and take a look at the code that's in there so this is going to go through what code got just got created function calling basically is you create a function you actually create a a schema that describes that function then you can give it to the llm and it can take advantage of using that the rest of the program

is simple it just gets the connection inputs and information uh goes off shows you that you put the system message in when the user asks something you put the user message in dot dot dot Etc so we're going to go back here and run it and see what it does to do that I'm going to use this thing called AI Dev shell which then populates all that stuff so there's no copy paste for you and then I'm going to run it and ask it what time is it there you go you can see that it

called the function it just knew to call the function because we described it to it and that's another way to get data into the system but how easy is it to make those functions we we saw that a little bit but let's actually watch how very simply it is especially when you have co-pilot at your disposal so what if I wanted to be able to see what the user's name was let's do function get username return Rob Chambers and then copy that's actually going to predict the rest of it based on the context that's in

the file I think I type like 12 characters in that whole thing I think you press tab three times I press tab three times that's true I might not be counting that so I come back here and run it again oops sorry node main.js and I say what's my name and now it knows my name so what I've shown you here oops dot dot exit what I've shown you here is how to get started with the AI CLI the kinds of things you can do like chat history and also how to use it to build

a quick sample for you all right so we're going to go back over here go to the next one now once you've done that this has all been done using GitHub models but you can maybe take on even more capabilities and features by taking a look at the open AI assistance API or AI search or speech input and output things like that so what we'll do is we're going to jump right into that we've shown you a little bit about function calling there are these things called assistants if you wanted to create an assistant using

a bunch of files you can just do this it'll create the assistant it'll create the vector store it'll upload all the files it just goes through it it's going to take a little while I'm not actually going to do that in front of you but what I'll do is show you how it works once you've done it what I did here is I pushed all the source code to the this open source CLI up to it you put the source code for your C I did the C source code I just put it right into

the into the the system and then what I'll do is I'm going to take this question and and paste that in it's going to ask where in the source code does this happen and it correctly identified it's in the chat command. CS file and then I can ask it also when I'm using assistance you know what are the key functions in that assistance helper class that actually that thing uses and it goes through and tells me what all those were this source file has this collection of sources has about 450 uh source files so very

very simple to do uh sorry go back here and I'm going to show you a little bit about AI search basically uh go to this next tab over here oops one more going to show you you can initialize it you can create a new search index there's a lot of samples that Dan's going to go into about how to do that so in time we're going to skip past that last thing I'm going to show you is a little bit of stuff around the what you can do with the speech that capabilities that are in

here so with this we've got input we've got output I'm going to show this should come across the speaker hello ignite 2024 that was pretty easy if I wanted to see how how I save that into a file that's what this looks like now I've got a file and if I wanted to figure out if I had a wave file like that that I just created I can get the text back out of it I can also translate it into multiple different languages it's very very easy to do and all these things that I've shown

you here are things that you can build your own source code to that are included in this experience so you just pick one of these things instin it on your disc and then use it straight away with the same AI Dev shell so you don't have to copy paste any keys or anything Dan's got a better way to do these things without keys he's going to show you in a little bit so last one I'm going to show you here is the real-time API which is very cool you've probably seen a lot of demos about

this today because it's amazing Seth waras had a fantastic one with the phone in the keynote today we're going to look and see how that actually works so again uh you know you could build one directly from the CLI but what I'm going to do is I'm going to use one that I've already created here and I'm going to go back in and do a shell populate all the environment and then I'm going to do netrun now I've also integrated in the ability to turn speech on and off with a press of a key because

we're in this big Auditorium and that might not be great if it's just open mic the whole time it's also got a keyword spotting capability built into it so I can interrupt it with a key and ask things or I can use the the keyword so I'm going to show you the push to push to start talking what's the date today is November 19th 2024 well it came out on my computer hopefully some of you heard that it's supposed to come out on the other one saw some loud speakers man I know I tell you

I got to turn it up uh but it does the function calling in the same kind of way but it also has keyword spotting capability so I can just use the keyword to activate it Quincy what's the date sure I'm here what's on your mind what's the date what's up yeah these speakers right hold on one more time Quincy what's the date there you go told me what the date was okay and behind the scenes it had called that function so we're going to go ahead and take a look at what that code looks like

very simple basically we give you everything you need uh we've got capability examples to show you how to get audio from the right thing in this example this is a c example using the in audio package we also show you how to do keyword integration to get the audio off of the keyword these things that are in this directory here are basically the same as all the other examples that are in the CLI how to do function calling Etc and if you wanted to go here and look at the custom functions these are pretty cool

I made a new one here which I can launch a URL so I'm going to go back and show you what that looks like can you open the Bing homepage sure I'll open the LinkedIn homepage for you I've open well you know LinkedIn Bing we have both of those so okay anyway uh audio uh aside there you go that's how that all works Dan a couple signs so we saw that saw how to actually use GitHub we upgraded to use the capabilities of azure what else can we do well you know one thing Rob is

like that was a lot of stuff was a lot of stuff that was a lot of stuff right and you know to build oh we forgot to mention by the way oh sorry open AI uh as your open a SDK it's available in multiple languages we've got python we've got net we've got JavaScript Java and go it's stable in multiple of those and in preview in Java and go at the moment so there's a lot of stuff that we can pull together and you actually need to bring multiple of these things together to build a

single AI application as a developer you need your data you need your services maybe you need some speech maybe you need some function calling maybe you need some assistance different endpoints you might need to integrate with other capabilities in Azure and your developer tools and so that's where the Azure AI Foundry helps you as a developer bring all of these things together in one place that you can use to build generative AI applications and one of the things that we're excited about that we announced today is the AER AI Foundry SDK that let you take

all of the capabilities that you have that we have in the Azure AI Foundry and bring them into your application and do everything that The Foundry can do programmatically from code some of the key things that The Foundry SDK offers is it allows you to leverage multiple of our popular models through a single coding interface allowing you to switch between them very easily it allows you to easily integrate uh AI capabilities into your application and develop f with a simplified coding experience so I'm just going to drop right into a demo and show what that

looks like so I'm going to start off here in the Azure AI Foundry so in The Foundry I can get started by creating a project a project is something that allows me to organize all these things into one place for my application I can click the create project button I can just provide a project name I can provide a hub name to connect to my resources I'm going to show what it looks like you know after I've created one of these projects inside The Foundry I've got uh my project here and uh inside the project

I've deployed a bunch of models so I can go to models and endpoints and uh I've deployed the GPT 40 model the 40 Mini model so I can switch out these different models to see is 40 giving me better quality versus you know for how much I'm paying for the the extra quality or performance um I also have a couple 53 models deployed and some embedding models which will allow me to do Vector search the other capabilities we have here we have in our AI service Services a lot of the things that Rob is showing

speech we got language translator Vision uh and document intelligence and content safety all of these are built into this one uh project that these capabilities that you can leverage so if I want to jump to code how do I connect to all these different things well what I can do is I can just grab this project connection string which is the one variable that I need to connect to my AI Foundry project from my code so I've got gone ahead and copied that code and let's hop on over and play with some models from code

so I'm going to paste that project connection string here and here I'm making a connection to the project using the AI project client so no keys no keys there's no keys in my files and you know as hard as I've tried to not leak my keys when I put them in my developer environment it always happens somehow so I from this project I can get a chat completions CL this is our unified model inferencing API that allows us to use all those models that I had showed you that I deployed into my project we can

get started with GPT 40 mini and I'm just going to pop up in the terminal and run this python file and we're going to see the output I've made a kind of fun prompt here that says you're a techno punk rocker from 2350 uh be cool but not too cool uh and I asked a question about taxes and it's giving me some uh tax advice as a freelancer so this is pretty fun so I mean GPT 40 mini is a great model let's take a look at what a 53 model would output just by changing

the model name so I'll just come in here and say 535 mini uh Das instruct and I'm going to run this model and this model uh it runs a bit slower because this model is meant to be fine-tuned for specific scenarios well that was easy yep I'm just using a different model by changing one variable now we're seeing as it's streaming output it's running a little bit slow in this case but we can see what that response looks like and we can compare it to what we got from the GPT 40 model um but you

know so we can then fine-tune this model if we want it to to work specifically for this futuristic uh tax spot scenario okay so let's take a look at how do we have this model respond with context about our application our users so as I mentioned it starts with data and let's take a look at some data that we want to provide a chat experience uh over so here are some product catalog information about uh Koso retail and I want to take this data and I want to ingest it uh into an AI search index

so in my Foundry project if I go into the Management Center I've connected an Azure AI search index up to this project that's all I need to do and once I've connected it to the project I can then get the search index through the project client using that one connection string so I'll go ahead and run a python script here to create a search index by ingesting this CSV file and while that's running I'll just pop open the code quickly here and show you at the at the top of the code just like I got

the chat client before from the project I got a vector embeddings client which will be used to embed the documents and a search client which allows me to connect to that search IND so just using that one project connection string I can get access to those different capabilities and I've got ahead and created that products index now uh let me run a uh chat with products function which will then allow this AI to answer questions from the documents in here so I'm going to ask uh what kind of tent would you recommend for four people

and while it's running one of the cool things that uh that this leverages is it uses a prompt template this prompty file format which allows you to specify uh using a template engine how you want to take the documents and insert them into that prompt so here uh you can see that there's a set of documents that are going to be retrieved from the search index and passed into the prompt and we're just going to create a markdown document with those and we can see the answer here uh being output in the in the terminal

it says for a group of four I'd recommend the trail Master tent now you can also um that's interesting run this with uh tracing turned on sorry I the wrong terminal window so uh built into this sample if you do this Dash Das enable Telemetry um it's going to Output a link to it's going to log the tracing results to AI studio so I can actually see what's happening when I'm running that code and so here I get the trace of what happened and I can now show you uh the different steps that were involved

in answering that question so I've got uh first there's I can see there's two llm calls that were made the first llm call takes the question uh from the user with this sort of intent mapping prompt um and the user's question was I need a new tent um for four people what would you recommend and then the AI turned that into a search query of the best tense for four people we then embedded you know the best tense for four people into a vector uh representation and then we ran a search query to get the

documents and then the final prompt that we sent to the model was that one that we saw in the prompt template there where it's taken those results and it's generated those documents and it's passed it into uh the llm along with the user's question and there we got the answer back so that tracing is really helpful because it gets you it helps you get an understanding especially when there's all these complex interactions that are happening between you know your code and the models you can quickly understand uh what the code is doing and potentially figure

out where you went wrong so this is this is fun and all but you know I'm just playing with this app in the terminal and you know it's not really giving me that user experience that I want to deliver so how do we get this into an actual app where we can have a conversation rather than just a single prompt uh in andout so if you head on over to the code page in the AI Foundry you can see that we've provided you with a set of AI application templates that can help you get started

quickly there's three here that we're highlighting for different scenarios but you can click the link to view the full AI app catalog and we've got all sorts of different technologies that you can use for databases and Vector search and programming languages with front ends we got some C go Java JavaScript python typescript Developers use all sorts of different languages and so these give uh everyone a different starting point that they can use we're going to get started with the basic uh starter template here and what we can do is we can install the Azure developer

CLI which we can just run this command to in it a new app and it will clone this template down set us up a new git repo with all the code that we need to provide a basic chat experience to our users now let's give it an environment name and let's just take a look at what that code looks like so here we have a basic application with uh a backend you know some routes and we've got some HTML and what we can do is we can just take some of the code that we wrote

that custom rag code and we can just pop it in to the sample app I've done that already over here so that chatwi products function I just popped it into the the routes of this sample app and then uh we can just deploy the app so here if you look at the the commands I ran in the terminal I just use that project connection string to set an environment variable to say hey I want you to deploy an app that connects to that existing project I had created and then I ran ASD up which built

and provisioned the application and so after that application was deployed I just get a link to uh this external website that's been deployed I get this nice chat UI and I can say uh what kind of tents do you have uh would you recommend let's try that one and the assistant will come back can ask me with another question so now I can actually answer that question and say well uh I have four people and you know uh we can ask you know where should I go to see stars and because of the prompt that's

provided it'll tell us that it can't answer that question all right so that was a quick tour of how thank you how you as a developer can create a project use a common inferencing API to try out different models use Azure AI search to ground your applications responses in your data and then deploy applications to Azure using pre-built templates another thing that's coming soon that you'll be able to leverage next month is the Azure AI agent service which allows you to build unlock another level of intelligence in your applications doing a lot of the things

instead of having to write all that manual code that that I wrote in in this sample app uh you know there's a few hundred lines of code in there that with the agent service you can express that capability with just a few lines of code to create a vector store and uh run a chat function and you can leverage additional tools like code interpreter function calling to build these applications much more quickly I think there was a talk on that in the room right before this uh and I encourage you to definitely check that out

so I showed this uh application to my manager udy who's actually sitting in the front row and he said great can you ship it next month right never happens as a developer right so I don't know it worked for me did it work for you I'm not sure so this is where you know we're in that phase where we want to get our application to better and better quality and we need to use a more rigorous approach to uh measuring and improving application quality and so let me just drop into a quick demo on how

we can do that using evaluation now which okay so here in my code I have this set of test questions these are Benchmark questions I'm going to use to evaluate my application just a whole bunch of different things uh that that I'm going to ask and evaluate the outputs of my model on these are a lot like test cases but for non-deterministic testing of prompts exactly and uh sorry I get lost in the terminal scroll here um we can run that and then uh we can see what the answers are to all those uh questions

in here in the terminal um let me just take a quick peek at the code so inside the code we've got this evaluate function which uh I can use to run evaluation on that data set and I can specify different evaluators that are been specially purpose built by Microsoft and uh industry standards uh to evaluate and score how well the quality of those responses are uh based on these metrics and so groundedness tells us how well is it using that context and answering factual information and relevance tells us how relevant is that answer to the

user question now I'm sure you're all thinking this terminal is not a great way to view those results but after I run this python script it'll upload those results to AI Studio where I can get a much richer way to go through and understand the performance of my application and I got to say I did a pretty poor job on this one you know 1.62 out of five on relevance is not going to be happy it's not ready udy not ready so what do we need to do to ship this next month well let's take

a look at all the inputs and outputs that are coming into the application um so here we can see for each of those test cases uh what are the results that we're getting and um we can see that a lot of the answers to the question is just asking another question back right so I went into the application and I went and changed my prompt I noticed that inside the prompt it was saying that hey if you're getting a vag uh if you're getting a vag question ask for clarifying questions instead but hey this is

AI we can first respond with some suggestions from the documents and then answer the user's question so I reran evaluation with this uh updated prompt and I went back here and I can actually compare two different runs with that original prompt and the updated prompt and now I can go row by row and see the difference while we can see that instead of just answering with the question now we're getting you know a summary of different tents that are available and then you know the follow-up question so that's pretty cool um but why is it

only a three out of five instead of a two out of five I think we did a lot better this time yeah well this is where the reason comes in the reason will tell you that hey this response accurately reflects the information but the context is adding some unsupported details about waterproof ratings what do you mean unsupported details well it says the sky view and Trail Master tents both have waterproof materials and R lives for added protection is that not true well if we look at the actual context information we can see that um inside

of the context this tent is water resistant it's not waterproof and so these are the kind of details that the these finely tuned evaluators can pick up about your application as a developer I would totally miss this and so this is how you can develop these applications at scale using AI that would take a lot longer to find if I was just combing through that I just wouldn't found it to be honest all right so that was a quick demo about evaluations you can also uh run these evaluations in the cloud uh to if you

want to get it off your laptop so you can close the lid and get on the plane and then you can also have human feedback in the loop to to give thumbs up and thumbs down on those evaluation results so yeah very nice so how do we improve further from here Rob well fine tuning might be a way to go let's take a look all right so you've heard about fine tuning as well I think there was a talk earlier today that had a lot of great details there I'm going to touch on it just

really briefly but the idea is that you might start with using a very high-end model but maybe that's not hitting your cost curve and so maybe you're going to try to use a different model and or F tune it or distill it or maybe you started out with a more basic model and you need to actually add a lot more more to it using this fine-tuning mechanism and so either one of those directions you go whether you start high and come down or you start low and you try to go up optimizing via fine-tuning or

distillation is a really important part so one of the things that I'm going to show real quick here is that there are many different ways to do that you you saw earlier today in some of the presentations you can watch online if you didn't you know you can do that using some tool sets in a shell or you can use the the Azure AI Studios fan fantastic capability now called The Foundry Studio or we've also got capabilities built directly into vs code so it's again trying to meet you where you are you know with the

kind of development environment that you need to be able to do that so let's say we get it all fine-tuned then what do you do my chat website was pretty cool don't you think it was very cool but wouldn't it be cooler if it was in teams now we're talking now we're talking all right so in sat's talk you saw that we are describing the whole stack we've got co-pilot at the top you know we've also got the whole AI stack there's a there's a way for you to plug capabilities in using these capabilities and

so again if we switch back to actually let's go forward one more so as you can see on this slide this is that stack that we were talking about and so you can take your intelligent agent and get it into various levels of the system system any any way you want we're starting to run low on time one thing I wanted to show you on the screen uh is if you did want to actually put it into teams the way you could do that is we also have an extension to help you with that you

got a problem we got an extension we got an SDK we got the right thing for you in the right language and so here you see there's a bunch of different capabilities you can get to the documentation you can view a bunch of samples you can create the app it asks you some questions maybe it's an agent or maybe I want to create it differently I want to create a custom engine agent because yours was using chat protocol between your front end and your back end right so I could use one of these to then

modify it to have the use of chat protocol it also could be used over here in vs uh big vs itself not just vs code for those of you that are still using vs for all your C needs or other capabilities most C developers which is most C developers that's right so bunch of same kinds of samples same kinds of capabilities and I mentioned the chat protocol and so there's a website here you can go and learn more there's a great set of SDK libraries for you to do that with as well all right awesome

so after we've built our app let's talk briefly about production so on the road to production one of the things that you're going to need is a lot of automation right so those evaluations that I showed you I'm running locally we can wire those uh up into GitHub actions and let me just quickly show uh how you can do that so from that terminal where I ran a in it to create this app I can also run uh after I've you know pushed that git repo to GitHub I can run ASD pipeline config and it

takes all these set steps to set up credentials environment variables and everything so that I can run uh my code inside of GitHub actions to automate run evaluation as part of every code check-in to make sure I'm keeping that quality high right I like that no more it works on my box but not yours does it work in cicd so here uh this is one of those evaluation actions we've got an evaluation GitHub action uh where it can run you know all the evaluation on different inputs and outputs and we can see those uh results

here and we can see you know different test scores for different test cases um and something new that I'm super excited about is once we pass the evaluation step coming soon we're going to have the ability to kick off an experiment after that evaluation step runs so here I've got experiments integrated into my GitHub action and let's just go through and look at this experiment analysis so this is a case where we've tried out two different prompts we're pushing a new prompt we had a professional prompt and we're pushing an adventurous prompt and we can

see that the experimentation action rolled that out to a treatment group of users and we can see that there was some effect as a result and you can look at all the different metrics that we have and you can see that the average chat call duration uh increased as a result of this new prompt so it went it took 40% longer to answer questions it used almost 50% more tokens but was it worth it well we can look that from the thumbs up thumb down from users that it didn't make any conclusive difference in the

actual quality of the experience so cost more money didn't seem to make a difference maybe we'll hold off on that prompt and keep the one that we have good savings all right so all this data is going to Azure monitor which means that you can also build production dashboards that give you a sense of what your evaluation scores look like over time how much you're spending on tokens and what models you are being used and so this is the kind of visibility you're going to need when building these kind of applications and uh if you're

interested in trying out the Azure AI evaluation uh service you can sign up for the private preview uh using that link and that QR code that's up on the screen and with that why don't you take us home Rob all right we'll do so you've seen today a lot of different things you've seen how you can go off and explore the Azure AI Foundry in all different variations from using it at Terminal Windows to vs code to vs itself to the AI studio all over the place it's there for you there's a bunch of different

sdks but the one we want you to also look at is this AI Foundry SDK which is inclusive of our work on the Azure open AI sdks as well use the one that's right for you if you want to do multiple models definitely pick out the uh inferencing SDK portion of The Foundry SDK got lots of great documentation some of it's integrated into various extensions you can look on Ms learn you can look through our sample repositories we also have a lot of learn courses for you to take advantage of just go up to Ms

learn and check it out got a lot of different sessions some of which were today some of which are coming up uh just search online we I don't think you've noticed but there's a couple talks about AI here at the conference this year so it's not just these it might be another 200 or so uh to take a look at all right thank you so much everyone for coming and hope you all took something away from today's talk e for