Elevate the retail experience with the Personalized Shopping Agent | BRK259

Good afternoon. I'm I hope that you enjoy Chicago. It's a great city.

We are going to talk to initiative that we have been working on for the last 12 to 18 months, a personalized shopping agent. We do that with three speakers. We do that together with Avanade.

Fishhall is here. He will do a longer introduction later, but he will cover the kind of the practical approach towards these implementation projects. And then Andre, the architect that worked on this project for a very long time that will cover kind of the technical deep dive into the architect architectural design behind the agent and and I will do a a quick intro.

My name is Papayarta. I lead product management for the cloud for retail. So with that, these are the three sections.

We try to keep it short so that we have time for for Q&A after. After that, I wanted to start off with with some of the the opportunities that we see with AI in in retail. Again, we have been working on this for for quite a long time.

So our team develops scenarios on their kind of what we call the Microsoft clouds for retail. So basically what we do is we build an industry IP together with our partners on top of horizontal Microsoft services. So in this case, you will see that this agent is built on their Azure AI stack.

Again, I, I think there's a lot happening in retail. Retail's often the forefront for where things happen. Look at what what's happening around security and retail or consumer experiences in retail a leading a leading industry.

Now what is very interesting is that we have engaged with quite a few retailers over, over the past year or so around how to use AI and we see an enormous opportunity. For the first time, we see almost equal involvement from both the business side of retailers as well as from the IT side of retailers. And you will see the, the answer on this slide is that the upside for retail and as you can see here, so additional revenue to drive with AI is north of $600 billion in revenue.

So it's a, it's an enormous upside in, in retail. Now there is a benefit for both the retail side as well As for the consumer side. You saw Judson this morning when we apply AI, there are different business outcomes that organizations strive towards and we see that always also with retailers.

So there could of of course be kind of an employee productivity business outcome. We built a agent for store associates and managers as well. We have a session on that tomorrow morning.

But what is really cool about this one is that it actually elevates the consumer experience. So you can actually drive more revenue with these experiences on the consumer side is very interesting because it actually creates a more delightful, engaging and even more efficient experience. So you will see it in the example that we have, but you will see AI drive a lot of these experiences.

We have examples where AI might be used as a as a mirror to try things, things on. In this case, it's, it's meant as a, as a chat, chat box and especially to drive kind of the personalized experience. And what we have seen is that with the personalized shopping agent, we are trying to drive these business outcomes in terms of higher conversion ratios from online sites to transactions, higher basket values, lower returns because you get to the right product for that for that consumer.

So that's quick the setup for kind of the solution that we have. I just wanted to do a quick demo and then hand it over to to Andre. So, so So what is it?

It is a a chat experience, so a natural language experience that a retailer can embed on either the website or a consumer app that they that they have. And it provides that natural experience where in human like language, you can start a search without knowing the details about the product. For instance, if I were to search for a washing machine, So instead of diving into kind of the details of washing machines and asking for a comparison, I can say, well, I'm a family of five, I want to wash twice a week.

So what washing machines do I need? So it's a human like experience where the where the agent tries to understand what you're trying to do and then tries to match it with the product catalogue of the of the retailer. It's completely grounded in kind of the consumer information from the retailer and the product and brand information from the retailer.

So retailers can tweak it to have their kind of their brand tone and completely fine tune it to kind of their product catalogue. So let's let's unpack that just a little bit more. So the agent is a headless service, meaning that a retailer can actually embed it in any asset they have.

And we as Microsoft are not building a front end for it because retailers have put a lot of investments in their brand experiences. So if I do the demo stay with me, the user interface is a demo user interface, not as nice as you will see. Of course the retailers have in there, but it's a it's a headless experience as as one.

Then the first part is, is that we have a a workflow that we will go into that tries to understand the the intent, the consumer intent. So the large language models tries to understand what is this consumer looking for? Is it just looking for brand information?

So if you go to a retailer or you go to a large brand is maybe to know something about the brand history of that brand and where that brand stands for? Or is it looking for kind of a gifting scenario or maybe what we called an occasion location scenario. I want to I want to dress up for Chinese New Year and I want to do that in in Bellevue, WA.

So what, what should I wear? So depending on kind of the, the consumer intent, it tries to understand that. And then with that it will ask a few clarifying questions.

It will do a product search and do product recommendations from the product catalogue. So, so that in an, in a nutshell, is the is the agent if you put it in a little bit more abstract. So the template that we build has these components in it.

So first of all, we do quite a few things on the on the prompting senior side. And you can see we have a, a range of these consumer intents and you see them listed on the on the left hand side. Then we have kind of an orchestrator that orchestrates kind of the flows that you go through depending on the on the intent.

So if you just do brand history, it could be a very simple one, or it could be just calling a, a browser to search for it. But if you do occasion location where you maybe want to recommend a fashion outfit, it will go through a couple of iterations and we'll call the LM model a couple of times and then use AI search to find the right products from from the catalogue. So there's an there's an orchestrator there.

And of course you can embed it and connect it through an API with kind of a website or a consumer app. And then of course, it's it calls an LLM model. And because this is a template, we update the LLM models continuously.

So when Microsoft releases a new model, we actually use the new models and we embed that in in the template. So this morning I think you saw where you can use multimodal to interact with the chat. So these things we will embed over time in the personalized shopping agent to make sure that we just leverage all the new things for Microsoft in in that agents.

And then there's a, there's kind of a lot of rag documents that the retailer often provides to tweak towards kind of tone of voice or or other specific kind of retailer specific components. So that's the just in a nutshell. Before we go into the architects of the the agent, I just wanted to quickly show it.

As you can see here, this is our website again, a demo website, so not as nice as the websites that you will see an example later. So this starts with a kind of a with a chat based experience. So I can now start with kind of searching for what I want.

So instead of searching for products, I'm doing a different. So I'm planning to go on a trip. I'm planning to go on a hiking trip to Joshua Tree National Park.

I have not been been on a hiking trip before. What would you recommend me to bring? So again, it's a human like search in terms of what you know what, what is it that I I want to do?

So it will ask me a few clarifying questions. These clarifying questions the retailer or brands can can tweak some want it a little bit quicker to get to a transaction somewhat a little bit longer than you have a more accurate search search term. So next month, I have no patience.

So it's a little bit colder. So it would probably recognize kind of the temperature that we're doing great. So we yeah, we you'll want to consider.

So it remains, it keeps the context. So whenever you continue the the search, it will continue that that search. So it will what you see it, it will actually recommend me a couple of products and then at the end in the tone of ours of the the retailer, it will do a description based on the on the search term.

Now the nice thing is, is that you have this is complete completely integrated in the website and most of the retailers that we work with integrated in the web for the consumer apps to actually do that transaction right from there. So basically you can actually click on these and it will actually bring you to the product page. So if you're interested in a backpack, you can actually go back.

You can actually do the transaction right away. And of course you can go fragrate this in different way. In this case, it's you basically branch out to a new tab and and to the website.

And this is the same website as we as we pulled up. Let me go back to the search term from here. You can actually go and further, you can actually look at some of these items and actually go and and look for further details on these.

You can both use the chat, but the LLM actually provides some suggestions as well based on the context that you have given it. So you can actually ask a couple of questions. And again, these are completely tweaked towards kind of the conversation.

So this is a quick demo of how kind of these experience work on a website or a consumer app. So with that, I wanted to hand it over to to Andre to dive a little bit deeper into kind of the architecture behind, behind the agent. Thank you, Papa.

So hello everyone. My name is Andrea. I am an architect for industry Cloud for retail.

And I'm going to explain what happened, what has just happened with this pilot, so. On the component level, the shopping agent is pretty much similar to any other RAC solutions. So from the left to the right side, you can see the shoppers using either web application from the customer or the mobile applications which are in the box of experiences.

We provide a demo website and the retailers can be can use that website, can customise that website. But mostly as Papine mentioned, the website is usually only the existing and the retailers just want to embed the experience into the existing application of the website. Usually the website or the web application is connected to the commerce data and enterprise systems where the actual purchase or basket management would be implemented.

Then it goes to the right to the biggest block with the chat API. And that one is our compiler, pretty much our agent. It is fully built on the Azure stack, so it starts with the Azure AI management, which serves as a gateway for the application, which understands where to land the traffic more.

By default it of course goes to our engine, but the retailer, since that is a template, can reconfigure the gateway and navigate some calls to the API to a different experiences. If they want to combine 2 compilers, for instance 2 agents, one from Microsoft retail, another from from the other source or something developed by the retailer, then the central block which is with the mark of Azure Functions. This is our major IP where we implement the whole chat experience.

It is an orchestrator, all the plugins, pretty much everything that you see and everything that is retailer specific is implemented here. So on the top and right side there are dependency services that our agent leverages. Of course, on the right side we would use Azure Open AI service where we currently use GPT for all model.

But during the history of involvement of sorry of improving the agent, we use different language models. As Papai mentioned, we get the latest one, we test them, we see if it improves the experience. If there are benefits of that, we update the latest 1.

So on different levels of agent, we may use different models. Right now we use only single only single one, but we can use different ones. Then we use the Azure AI search right together with the LLM to ground the agent on the customer product catalogue.

And the AI search is getting the data index through the data indexing pipeline on the top. There are augmenting services with being custom searched. So when some information cannot be found in the general GPT sort of knowledge or cannot be found in the branding information, it tries to use the websites approved by retailer.

So the agent tries to find the answer on the websites and on the bottom all of that is, we said it all grounded on the data. The data platform is not part of the compiler, but we do have another data platform and we're going to have a session tomorrow for retail industry data solutions. This is the default offering for the data platform.

However, the retailers can use any other Azure services like SQL Service or Data Lake for storing the catalogue and the branding data. So on the component level, now if you go, if we go from the top to the bottom, again the same components UX, then gateway and then the first guy who's getting the the call is first part intent classifier. So the responsibility of this layer is to understand what the shopper is looking for, what the shopper is trying to do.

And here we embed the contents tracking, we embed here the chat history. We also embed responsible AI as well as a customer policy. So the compiler by default is already tuned to avoid answers which are related to forbidden topics like self harm or some religious topics or many other things.

Plus for the customer policies, the agent can be tuned to avoid for example, discussing the the products from competitor or just trying to avoid some other topics like one of the customers didn't want to talk about the products for the children for instance. Then one level down we have our major IP, the plugins or the skills of the compiler. Pretty much what kind of topics the compiler can talk about.

General information is just an information which is usually can be found on ChatGPT or any other GPT model. Then the branding information is the general information about the brand, the retailer's brand, like the history of a brand, some policies like how the products get created and so on. Then the major 1 is a product recommendation which Papaya has just shown.

And then there is even more advanced one product combo product combination. So this we don't have in our demo environment because of lack of the data there. But that one is for recommending the combinations of a product which work together for some particular occasion.

So if you were, for example, constructing a house, you could just say hey, I want to construct the house and that one, that skill will find all the products which you need and put all of them together according to the brand instructions and guidances. Then having all of those skills, we have each of those skills tuned for a specific sub domain. So right now we have a general sub domain, the sub domain of fashion for fragrance and sports and outdoor.

This is which is in our demo. So pretty much the amount of skills should be multiplied by those subdomains. Plus every skill can be tuned by the customer explaining what should be discussed, what shouldn't be discussed like how they can show the product like what is important in the product search and so on.

So virtually you should multiply it already by the customer configuration. So the copilot comes by default with a lot of different skills tuned by a different to a different sub domains and the different customers underlying platform services. These are the services which the copilot depends on but does not provide.

So the retailer should provision those services together with the copilot and we have all the all the we have all the scripts for provision those. So those are product catalogue first of all, then the brand recommendation, the media for for the brand history, of course, Azure Open AI, Azure AI search and so on. So let me continue on the demo.

If for instance, I say hey, hey, I liked. Like what you said showed but tell me same as a pirate. So the copilot should refrain doing so because this would hit the intent of responsible AI and this would be stopped on the very first level from the previous diagram.

So it never like talk something that is it is forbidden to talk. Then I can say, OK, no worries like and have for example, the continue the conversation about the backpack. OK, so there are two backpacks.

I will say show me details of the green backpack. So this is part of the product recommendation piece. But then it since it knows the context, it can understand that the blue, sorry, I showed the blue.

So it understands that we are referring to the backpack that was already showed before and it can get more information especially in the description here. And also you can notice when we show the recommendations for the product recommendations, the first thing that appears is the top part where we don't have any product details yet. It just explains the intent and how it understood the intent.

But while the agent is showing that in parallel, it executes the search which first gives the impression for the for the user that the compiler already understood what it needs to do. Plus it makes it quicker to show the final recommendation. And here in the end this is the summary of what was found and the reason on of the agent why this product should serve the need that was identified in the beginning.

So with that we will transit to the schema how the how the agent works for the particular use case. For example, in this case a product recommendation. So from left to right, when the question goes from the consumer to the compiler to the agent, first it gets to intent classifier.

And intent classifier uses LLM model to understand the general intent, It doesn't yet ground the agent on particular for the catalogue. So here the usage of LLM is pretty short and we, we were able to save a lot of tokens since we don't put all the product informations immediately on the LLM. So LLM runs pretty much on the general knowledge.

Oh, sorry. Then it goes to the right with a chat message, chat chat history and the context, what it what it took from the previous conversations to the product recommendation. And then product recommendation have different stages.

First it extracts the parameters like the occasion, location, important things that the shopper we're looking for immediately returns back the overview that I showed before and in parallel it gets run the real product search. So the product search stage already has product catalogue involved with AI search service. And then once AI search and it writes several, several recommendations, then we use LLM a third time.

So LLM selects the best products which are matching the the recommendation. So at every stage the customer is able to customize that. So for instance, the customer can use a different recommendation engine.

For instance, in this case we use pretty much LLM as a recommendation engine, but there might be some ML model or some homegrown custom recommendation engine. So it can plug, can be plugged here and the compiler will just give a different recommendations in the end. And then finally we use the LLM to generate the answer again following the customer configuration for the tone, for the length of the message, for the way how the product should be shown, like what the sequence of the most important attributes of the product.

So then it goes back to the application which consumer is using. So all of those customisations plus the open mode for the template for the agent template allows a heavy customisations for the retailers whether done by the details themselves or done by the partner companies by ISV source software integrators and we have one successful implementation of that compilot. So let me hand over Vishal.

Thank you. Thank you, Andre. Thank you.

Hi there. I'm Vishal Sarkar from Avnard. I lead business applications.

It's been a privilege to partner with Microsoft on this journey on the personalized shopping agent. And what I'm here to do today is actually tell you a story of how we're actually implementing this in real life, the real customer, and give you some considerations when you're starting to consider what you're going to do with the shopping agent. So let me start, you know, and recover the how.

I'm going to go back to the why, why this conversation is really important right now. Even though retail has evolved, there are a lot of underlying problems in the way and the way customers experience when engaging with the brand. You'll see a lot of statistics on the slide there.

You'll see there's inefficient discovery. Have you tried going to a website, trying to look for things and finding, you know, not finding the right answer, being overburdened with information? What do you do?

You end and stop your purchase process or you leave the brand. So as you can see, there's information overload. There's 76% of the customers that you know, they, they claim a bad customer experience and don't come back.

There's a brand gap. Brands are not able to offer customers contextual help in the purchase process. And there are about 59% of the customers say, looking for help, you know, when it comes to saving time and then the indecisive customers who are trying to make a decision during the purchase cycle that they need help during that that process.

And there are only 38% of brand experiences that are pursued by consumers to be individualized, which means that another 62% don't work. So these these are not very, you know, comforting statistics for retail. And so that's why whatever you know Andre and Pepine showed you around the first line shopping agent become even more important in today's world in retail.

So this is where we are going to talk about conversational commerce. This is human like interactions with customers that are coming to brand website or through multiple channels to educate the clients, work with them, understand what the needs are and make the right recommendations at the right time enabled with our seamless commerce experience to be able to complete the purchase cycle. If you do that well customers obviously that the F customer effort goes down, website bounce rate and the churn rates go down, need for customers to go into the store obviously goes down.

And then the brand conversions increase, retention increases, upsell cross sell opportunities increase and the lifetime value of the customer can be better realized. So with that I'm actually going to cover a story where we have actually done this with the customer. So there is a French hardware retailer called Bricorama and they specialize of all the other things selling paint.

Now buying paint and doing a paint project is a, you know, as you can see in the statistics here shows that 66% of people in France are unhappy with the result of their home decoration. So that's the problem statement that this retailer was trying to solve and to make the paint buying process and the project itself more seamless for the end customer. So they rolled out something called paint.

Paint is a Avnard shopper Copilot, which is powered by the personal shopping agent that Andre showed you. And there are some really interesting statistics that came out of after rolling out paint and you'll see some of the statistics here. For example, in less than three months, 11% of the web users go through the complete expertise expertise funnel right from decor inspiration to finding out the right paint, finding out the you know, the the technical recommendations and they go through the entire buying process 11% in three months.

What is even more important is within 24 hours of engaging with paint, people actually come back and they start engaging with lengthier conversations that they're having conversations an average of 17 message over 1 1/2 days, which means there's a lot of back and forth between the customer and the and the copilot either asking questions or looking at recommendations. And what that does is it actually creates a pretty channel less customer journey because now the consumer is operating on their own time. They don't have to go to a store, they don't actually have to make a phone call.

They have gone through a process with Copilot and the copilot is contextual. It also remembers where you left off in your project and helps you pick that back up at any time of the day that you would want to go back into it, right. So this also helped build Brickaramas brand for them and instead of just becoming a, you know, a hardware retailer, this just started injecting AI.

They got a lot of brand and, and coverage and media coverage out of this. And they also reinforced more and more of the Microsoft platform in their work. They're now working on, you know, Azure open eye services to increase more of the AI impact on the business.

So I'll show you a quick video here of what the experience looks like. Now just keep in mind this isn't French, but we have tags in English to be able to follow through right? Alright, sorry clicker, There you go.

All right. So just to reiterate what happened there, the the consumer came and started started interacting with paint, gave a number of questions, I mean gave a number of ideas about what the customer was looking for, what kind of paint palette the customer wanted to to look at. The copilot came and serve his bag, the different kinds of paints and the kind of textures that were available.

That engagement started with paint, which then ended up going down the path of purchase. And after the purchase, the copilot went back in and came back with product recommendations. This is a cross seller absolute opportunity and this is very similar to what Andre showed in the shopping in the in the shopping agent where you can go back into your data and you can take a look at what are the opportunities to bring in surface other products along with the main product the customer is looking for.

So how did we how do we do this right? I mentioned this is the Avanar shopper copilot. It is built on top of the Microsoft personalized shopping agent.

So similar diagram to what Andre showed. I'm going to walk you through it. So one thing that I do want to point out to you here is as a partner, when you're actually, when we're actually doing this with clients, it is not just about the technology.

The technology is really interesting and is really important, but there are a lot of surround services that go hand in hand to make this really, really successful. So if you look on the left hand side, he says Avanar Shopper Copilot powered by Microsoft personal shopping agent. The first two boxes are things that are outside of the core technology.

Number one, it is the AI readiness on top of retail operations. Super important that we engage with our customers to really figure out what they're trying to achieve. Where is the friction in the customer life cycle?

Where is the friction in the customer purchase cycle to figure out what are the problems that we're trying to solve? This is not, you know, typically a technical engagement. This is more of an advisory LED engagement.

Really understand the customer's business. The second part, the second box on your left hand side is conversational design. None of this is going to work if you don't have your data estate in order.

None of this is going to work if you actually don't engage in conversation design. Like how should the copilot react? How are you going to teach AI to react in a certain way when it when it goes down a different path?

All of this is a part of conversational design before you start actually getting into technology. And then right at the end of the bottom, you see retail ecosystem integration. So there's a lot of opportunity here, not only to work with the, the personal shopping agent from Microsoft, but how do you extend that?

There's an opportunity here to extend that into ERP. This is where the the product data lives. How are you going to do billing?

How are you going to integrate that into a commerce engine? So we did all of that for Bricorama for for that, for that example to actually work. And then at the at the end of it is, you know, this data structure, data work is an ongoing piece of work.

Do you have a strategy? And as your product catalogue increases, the different changes, what is your data strategy around keeping the data structured and feeding the copilot? So if you look at this diagram, we'll go slightly technical here on the left hand side are the components that Avanade actually built from the customer application front end website.

That's that's brand experience work, which uses user experience design capabilities. Then in the centre we use Microsoft Copilot Studio. So when you do you see the shop or copilot, it is actually built on Microsoft Copilot Studio.

It's a custom copilot, which also has integration back into different channels. It has an integration back into live agents because a customer can come in through a contact center. They don't necessarily need to come in through a website.

And then how does a contact centre agent seamlessly transfer that interaction back to a back to a copilot? So this is human and copilot working hand in hand in a very complementary way. And so that's also beyond what the beyond what a personalized shopping agent would do.

That's an integration that a partner or yourself you'll have to build that in. And then finally the commerce and data enterprise systems at the bottom, they are the actually the auto management systems. Now it could be a Microsoft based auto management system.

It could be a third party auto management system. But integration is key because that's how you're going to complete the life cycle of the journey and to get into customer conversion. OK, So what does a typical engagement look like when you, when you hear the term, you know, shopper copilot with Avant?

Like I said, we don't start, we don't just jump into technology. There is a 2 hour assessment that we actually end up doing with customers to really sit down and understand the business objectives. This is not a purely technical project.

If you look at our team at the bottom left, you'll you'll realize there are different kinds of capabilities that need to come together to be able to do this. You have advisory services, you've got organizational change management services, you've got data engineers on this project trying to figure out how to surface data, you've got data science and you've got technical architects. All of these need to work in tandem to figure out what the right architecture is before we jump in.

And then typically after doing a 2 hour assessment, learning objectives with the customer, we actually get into a hands on workshop, which is a design thinking. That workshop, which is more derivative from the business value the customer is going to derive and what the consumer journey is going to look like. We actually model out what the end customer journey should look like and then go backwards into fitting blocks of technology in it.

There's a quick readiness assessment. All customers today want AI. You know that they're all coming to Ignite here.

They're all listening to everybody talk about copilot. They're hearing everybody talk about AI. But you need to be ready for AI.

There's a readiness assessment you need to go through before you just start letting AI loose into your environment, right? Where does that readiness assessment start? Starts with, you know, what you want to do with your data.

What is the governance around your data? How much of data should should you really expose outside to be to to a copilot like technology? And all of this happens under readiness assessment.

Some customers, like in this case the the French hardware companies, you just can't jump into, you know, getting them started with the copilot. They have to get their data estate and all of us. So that's why you have a lot of data engineering and work that was part of the project.

And then finally, you know, there's a MVP implementation for the selected brand and the banner. You know, we need to back that up into the KPIs. You know, what kind of customer conversion rate are we looking for?

You know, how many touch points a customer is going to go through, etcetera, etcetera that actually make this a pretty unique, unique engagement. So with that, I'll pause. If there are any questions, I'm happy to take the questions or I'll can invite Pepine and Andre to join me back on stage, right?

So maybe, yeah, maybe let's close it out and then go to questions. I thank you Vishal and Andre for bringing this to life. Great example with Rickarama and and paint as an example.

A few things if you want to learn a little bit more about kind of this specific agent Beau Avanade. And we are at the Expo. So come, come and find us and we, we can talk through it one one-on-one.

And of course there's documentation on, on learn. microsoft. com.

If you want to see more, there's a link, a sign up link for the preview as well. So if you're really interested and you want to go to next steps, then we, we have a sign up link for that as well. And then last slide, I promise there are a couple of more sessions that we have and I want to call out at least two.

We have a retail store operations agents tomorrow morning, a session how we do it for to empower the store associate and store manager in the store. And then we have the retail data solution with the retail data models at the end of the day tomorrow. So with that, let me know if you have any questions.