So welcome everyone. So glad that you could you know take a break from your busy schedules and join us for the webinar. Uh just a small reminder that this is a recorded session and the recording will be posted in the community both alumni and learners and uh we have a Q&A session at the end of the webinar. So as you have questions during the webinar, put them in the Q&A section at the bottom of your screen and Our guest speaker will be answering all your questions at the end of the webinar. And also this is
an interactive session. So our guest will be asking you questions for your uh you know answers, polls, all sorts of things. So put your answers in the chat box and uh so let's begin. I'm Shani your community manager. you all know me and I'm thrilled to uh you know co-hosting this webinar and can't wait for us to dive into this very interesting topic today. Uh today's Topic is product management in the world of generative AI and we have with us Vanit Greybal senior product manager of Udacity and she's a fantastic colleague and an excellent mentor
and I'm so excited for you to join us today. I extend a warm welcome on behalf of our learners and alumni community. Welcome Vernit. Cloone is all yours. Thanks Shiovani. Um and hi everyone. This is I think my first webinar with the community team. So super excited for This opportunity. Um and super excited to chat with you guys, answer a bunch of questions that you have. Um hopefully the topic that we're covering today is of interest to folks on the call. I know we have a broad audience. Um, so I'm hoping I can cover a
bunch of your questions, things that you're interested in, and share what's going on at Udacity. One second. So, starting off with just a little bit about myself. I joined Udacity two years ago. I used to be the content platform product manager, but now I own everything to do with AI strategy and road mapping across the company. Um, previously to Udacity, I was a product manager at Salesforce, Tableau, and Forward, which is a medical device startup in San Francisco, but now I currently live in Manhattan in New York. I'm originally from Toronto, Canada, and I studied
systems design engineering at the University of Wateroo. Um, and outside of work, I do a lot of martial arts. I travel a bunch, and I'm also very much into painting. And so that's a bit about myself. With all that said, we can dive right in. So just an agenda for today so I can set expectations on what we'll be covering. And again, hopefully these are topics that are of interest to everybody. Um we'll start with just a brief overview of what is product management just so that we have a Unified understanding. Um I'll do very
quickly again uh an introduction to generative AI. I know that this has been a hot topic for a year and a half, two years now. So, I don't want to spend too much time defining things that folks on the call might already know, but we do have a broad audience of uh junior practitioners to seniors. So, just want to make sure we're all on the same page. Then, we'll talk about challenges of building AI products, what it means to Build AI products, and how Udacity has been building AI products across our platform. Um, we'll talk
about something called evals, which will make more sense after the building AI products conversation, and then I'll end it with Q&A. I know that folks on this call did submit um questions in advance, and I'm going to do my best to try and answer some of those throughout the presentation, but if you want to post those questions in Q&A um or when I have Uh opportunities to chat, feel free to do so and Shiovani will help filter those in for me. So jumping right in, what is product management? It's a bit of a squishy title.
And so different companies tend to define it differently, but across the board, product management generally means the organization, the team, the people that are in charge of identifying problems, defining solutions, and ensuring you execute against those Solutions for different companies. It differs from project management which is a lot more focused on timelines and execution and program management which is a lot more focused on execution across projects. And so product managers tend to focus on what and why. I realize there's a typo in my slide. It should say project product managers at the beginning. Um but
they focus on the what and why versus project managers and program managers that focus on the how And when. As product managers, our job is typically to bridge the business, tech, and design and ensure we're building the right thing for the company's customers or for the business. Who are product managers? So, um, product management is a pretty diverse field. it it draws talent from a lot of different buckets, but generally speaking, a lot of product managers that I've encountered or that you'll see um in in the industry have a computer Science or engineering background. Um,
another really common way into product management is business experience and typically MBAs. Um again because product management is at the center of business uh engineering and design a lot of the people come from those specialties and then eventually make their way into product management. There's many different types of product managers. Uh there's a myth that product managers have to be very Outgoing and um extroverts. That's not true. I've met really amazing product managers who are introverts. But the biggest characteristics that I think define product managers is that you're extremely curious. You're the type of person
that's always asking why and you don't really stick to a status quo. You're really empathetic naturally. Um, and you're able to relate with users really easily and put yourself in your user shoes. You're very good at Communication. That's super important to make sure that you're able to align people from different backgrounds and different stakeholders. and you're you're very anal about execution. You're all about getting things done. Um, people often refer to product managers as the mini CEOs. They define product vision, strategy, and roadmap. But I I like to think of product managers more as like
the train conductors. We're the ones that decide where the train's Going. And it's our job to make sure everybody works together to make the train successful or the journey successful. But we're no more important than any other member of the train crew. we're just the ones figuring out where we're going. Product managers tend own the product development life cycle. This again doesn't really change company to company. It's a pretty generic um life cycle that we manage and it's the Different phases of it are ideulating and prototyping. So this is where you identify the problem and
then prototype and try to figure out how you're going to solve that problem. You move on to designing and building. This is where you bring your designers and engineers together to build the right solution. You're in charge of testing and debugging. So once the solution's built, it's your job to make sure it's actually solving the problem the way that you Expect it to and that it works consistently. And then you're in charge of launching and iterating. So you launch the product. You make sure it's hitting your metrics. It it's doing what it needs to for
the business and users. And then you iterate and decide what comes after. Product managers are in every part of this life cycle. And this life cycle doesn't change with AI product development. Um the only thing that kind of changes is where you end up Spending a lot more time now. And in traditional product development, the life cycle is very predictable. Whereas in AI product development where we're all kind of discovering this um new technology together, the different buckets start to become a lot more unpredictable. But we'll jump into that in a minute. So my first
kind of question for the audience is um where are you guys coming from? So I know again we're we're talking to I I Think alumni and current learners and potentially people that maybe have never interacted with Udasi before. So I'm curious to see who's in the audience. For myself when I mean background I started in engineering but I moved into product because I was a lot more drawn to big picture thinking and problem solving. Um, so yeah, I'm I'm very curious to see where the audience is coming from. Shiovani, if you don't mind helping me.
Yes. Yes. I think we have a great mix here. Uh, we have people from data, marketing and PR, software engineering, product and data. I think data programming and product uh is dominating the chat right now. So yeah, from what from what I'm able to see, I'm actually quite surprised. It seems like there aren't too many product managers on the call. Um, a lot of software engineers, a lot of engineers in general in data. So hopefully again my presentation is still relevant to you All. Um, I am not an engineer by trade anymore even though I
studied engineering. And so this this presentation is less in the weeds of the technology which we can do in different webinars if that's of interest but a lot more focused on the big picture thinking and how we um think about product now in the world of AI. So um thank you for participating. I will move on to the next slide. Um so let's jump into generative AI. Again I don't want to Bore too many people on the call. Genai has been around for a a few months and it's been really hot. So hopefully everybody on
the call has some familiarity with it. But just to level set, generative AI differs from or is a branch of AI that's focused on creating new content versus just predicting um outputs from existing data and models. And so this can refer to text image code generation. Um and why this has impacted product is because one, it's expanded Our product capabilities. we can now build very different types of products that weren't possible before. Typically products that focus a lot more on creativity and personalization which is a lot more difficult to do with traditional um software development
that's really founded in like routine business logic. Um it changes how we build not only in terms of like AI tools that engineers can use but also in terms of managing expectations when you're Building AI products. It's very different from a product manager's perspective when you're trying to set expectations of what you're delivering to your stakeholders and it also shifts a little bit of decision-making. We now have AIdriven insights and automation. Um you can give a little bit of control to AI to manage um user experiences. So overall generative AI um is making the entire
world rethink how they approach problem solving and product managers at Least in the world of tech are often the ones leading the charge. So my next slide's a little bit more in the weeds of technicalities. The reason I wanted to include it and I kind of went back and forth. It's not critical for you to understand model types and foundation models for you to like be able to build AI products. But the reason I wanted to include it is when I first started driving the AI strategy at Udacity, there was a lot of terminology Coming
at me and I was just like, what does this all mean and how does it fit together? And so I'm trying to create maybe like help you create a mental model of where these different pieces fit in and how that impacts your world especially if you are a product manager. So model types is kind of like the gen generic term like transformerbased models, diffusion models, GANs and VAEs. These are different algorithms, different techniques for building um AI Models that can then generate something. Um transformer-based models are the backbone of the large language models like chat
GBT, Claude and Gemini and these are very um good for text generation whereas diffusion models are really good for image and video generation. So this is the backbone for um foundation models like Dolly and Sora. The other two types of models I personally have very little interaction with and is not super common right now In like just a standard tech company or where you're building just standard products. They're more focus specialized for like ultra realistic images, audios and like deep fakes um or for music and audio generation. And then when you hear the term foundation
models, um these are basically uh the models on the market that are trained on large data sets and then fine-tuned for specific tasks. So GBT is a foundation model. Claude is a foundation model. Gemini is a foundation Model. And then if you ask like okay GPT and then what's chat GBT? ChatGBT is just like the application built on top of the GPT foundation model. So chat GBT is your UI to be able to interact with this foundation model that's running underneath. Um the next term that you might hear a lot is reasoning models. And how
that fits in is just reasoning models incorporate additional mechanisms to improve the structured reasoning of these foundation models. And so the Different techniques you might hear of is like chain of thought prompting, train of thought reasoning, or reinforcement learning. So, chain of thought prompting forces the AI to break down its reasoning step by step. And so, it will tell you how it came to the output that it did. Um, tree of thought forces the AI to go really broad. So, if you ask it a question, it'll come up with multiple different ways to respond and
then pick the best one. And then Reinforcement learning is where you can give the model AI feedback or human feedback to help improve its accuracy. Um, and how that all then eventually comes together. So kind of getting out of the weeds again is when you're building AI products, you can kind of think of it as a pyramid. At the very bottom layer is the training data. Above it is where you build the foundation model. Above that sits your company's infrastructure, your personal Infrastructure. And then above that is the application. I don't work with training data.
I also do not build foundation models. the if you want to do that if that's what you assume as an AI product manager then you're working at companies like OpenAI Anthropic or Google um I also as a product manager and in the product org don't really deal with the infrastructure that's typically handled by your company or my or Udacity's engineering team and so they Set up the infrastructure to make sure that you're able to access thirdparty APIs from open AI and anthropic and Google um in a safe and secure manner where we live as product
managers and product engineers and designers is at the application layer. We're building applications um which are features or products user experiences that leverage these AI uh APIs from these thirdparty companies um and with those APIs you're able to pull in model Outputs. So hopefully that wasn't a bit of too much rambling. Um, so I'll get to my next question, which is, um, I'm curious to see from the audience which generative AI tools that you're using the most or which ones you've interacted with. Uh, some common ones to just get you thinking is chat GBT. I
think a lot of people have experience with it. Um, it's competitor is Claude. Um, there's also Dolly, GitHub, Copilot, Notion AI, Grammarly. So um super Curious what people have experience with. Yes, the popular answers are chat GPT Claude uh somebody mentioned Gemini Perplexity Dali Adobe Grock. Grock is starting to come up a little bit more. Um but yeah, Chad GPT definitely had a first player advantage. I think it's become the common name that people refer to. But um super cool to see that people are also using DeepSeek. Uh if if you're not caught up on
what DeepSseek is, it's a foundation model that was made open source, meaning you can see exactly under the hood how it works. Um which was a bit of a exciting um also nerve-wracking launch in the AI world. So um awesome, super cool. Thank you for responding. Uh great to see that a lot of folks are starting to play with AI tools. Um, so let's jump into building AI products. We'll start with challenges. For the folks that have a lot of familiarity or experience using AI tools, you've probably already encountered a lot of these challenges. Aloo,
um, just from the jump, the difference between AI or traditional software development and AI products is that traditional software development is very deterministic. If you put in input X, you almost always get output Y unless there's a bug in your system. With AI Products, they're probabilistic and non-deterministic. So if I in a traditional software world, if I built a support bot, I would code in business logic saying if user says this, respond like this. Whereas with an AI powered support bot, if I it's it's more along the lines of if user says this, 90% of
the time we're going to respond like why? And so all the challenge becomes how do you con constrain and how do you control the model outputs? So Hallucinations and inaccuracy is probably one of the biggest challenges that people encountered early. Um, AI will not only sometimes it can just give you the wrong answer. Sometimes it will hallucinate in the sense that it will make up an answer. You'll find that these models almost always want to give you an answer. If you ask it a question, it's never going to say I don't know or I don't
have the information. It just is determined to give you something. Um, And so that can lead to hallucinations. There's context limitations. So these models have a fixed memory window. Um, they can forget earlier messages. Um, and there is a sweet spot of how much context you can give it before it just like starts to forget. Um, cost and latency is obviously a common problem, especially with the more advanced models or with image generation, video generation for sure. You're now starting to spend a lot of money for these API Calls. Measuring performance becomes a challenge again
because we're not in a rulebased system anymore. it's hard to say did your feature pass or fail um is not as simple anymore but we'll talk a little bit more about that later. Um ethics and bias issues and then data privacy and security. This is a huge topic especially if you're building enterprise products. Um consumer product uh consumer users also will ask questions about this and care a lot but Enterprises a lot of the time are concerned about data privacy and security. how their clients information is being saved by these thirdparty companies and potentially being
used to train their models. Oftent time it's not actually a um high concern. It's more around the perception um that you as a product manager end up having to manage. So going to the next slide, there's a bunch of techniques um to improve model outputs specifically when We're talking about inaccuracy and hallucinations. um prompt engineering uh good prompt engineering goes a very long way and there are many I should have linked them I may add them to the slide deck after um there's many guides and best practices developed by companies like OpenAI in terms of
how to do good prompt engineering things like using structured tags um giving very clear instructions and rules to the agent um sometimes even giving the agent a Personality can go a really long way in terms of improving the model output Put few shot prompting is another technique where you give the model very specific examples in its context and try and try to en encourage the model to formulate its responses based around those examples. Rag is super common retrieval augmented generation. So basically this is taking external information and put augmenting the input to the model with
that information. And so let's say the Model, let's say your base prompt um has certain information about the user and then you pull in what enrollments or what programs that user has completed in order to respond to the user in a more personalized and customized way. Um feedback loops and human in the loop just having this is this is nothing fancy. I don't mean automated feedback loops. we haven't implemented that and there isn't a really good way to do that but more just uh very typical product Management experiences where you look for feedback and then
you try to improve your model output and adjust the the prompt accordingly and then fine-tuning we'll talk I have a slide dedicated to fine-tuning and uh we'll talk more about that when we get there so I'm going to jump into how you this is the part I was super excited to get to so hopefully I haven't been speaking too fast but I do want to talk about the different products I've built At Udacity and the different products that we're scaling and trying to improve. So, we'll start with Udacity AI, which internally is affectionately known as
Marvin. Um, but Udacity AI for learners, for mentors, for alumni on the call is an in-classroom chatbot that you might have experienced. And so, our chatbot is AI powered using um chat or open AI. And this is a really great example of where we use retrieval augmented generation. So all of our Program content or at least the program that the learner is currently consuming is converted into embeddings and stored in a vector database like pine code. And then when a user submits a query or a question to our chatbot that is also converted into an
embedding and then the database searches for similar embeddings and whatever it finds in forms of documents it puts into the context and then when Marvin or Udacity AI responds it has that additional program context To be able to respond to the learner. And this is how we've created an experience where Marvin can respond to very specific program questions like um can you summarize this lesson or could you go deeper on this specific topic or even can you summarize this video? Um yeah sorry next slide. So with Udas with Marvin this is also another example where
we're using feedback loops. Again this isn't groundbreaking it's not revolutionary. This is where we merge um traditional product management with AI product management which is we've developed simple feedback surveys like the one that you see on the left so that we can hear from users what challenges they're experiencing and this is really valuable because for example we launched a feature in classroom called summarize video and enables the user to like quickly review what was covered in our video. Um and whenever I click the Button it works. So, I was like, "Oh, the feature works." But
I'm noticing now that our number one driver of one star ratings is people saying that the summarized video button isn't working. And so, again, we're in a very probabilistic world. And it's hard to know for sure that your features working 100% of the time. So, this type of feed, real user feedback becomes really valuable, not only to improve your model output, but also in terms of catching Bugs. Um and then we've also done a similar UI um feedback survey the way that chat GBT does in terms of a thumbs up thumbs down on the message
level. The next Udacity product I want to talk about is Studio. So this was actually my first baby when I joined Udacity. This is a product I've been working on since day one. And it's kind of crazy to know or see how far Studio has come because we were building Studio when AI was super new and we had no idea The limitations of AI at the time. And the vision for Studio was to be able to generate programs immediately on the fly um with very minimal human interaction. But we quickly learned through prototyping and experimenting
that AI does not generate as high quality of output as humans do. and oftent times it sounds AI generated. So now Studio has morphed into a more co-pilot experience where it's a creative partner for our instructors. Um what's also really Interesting about Studio and a good case study when we talk about AI products is it started with multiple routine-based agents and task specific prompts and now it's going a lot more into an agent-based interactive experience. And so when we talk about routine-based or task specific, I'm talking about things like quiz generation and video summaries, it's
very specific, narrowly defined scope where it says take this page content and now generate a quiz That follows this specific JSON structure. Um, and that's a lot easier to get right versus something like Studio Copilot, which is now very similar to um, Chat GBD Canvas for anybody who has experience using that tool where it's a creative partner that can generate quizzes, generate video summaries, it can make changes on the page for you. And so now we're trying to orchestrate many different types of experiences and we're running up against Really um significant problems and challenges when
it comes to this. But essentially what we're trying to do with Studio is an evolution from AI as a helper to AI as a creative partner. And I'll talk through some of the challenges that come with that next. But before I get there, I'm curious to hear um what have you guys built with generative AI? So, um, whether it's at your companies or whether it's personally, I I'm very curious to see how people are Implementing Genai in their personal lives. I'll share for myself outside of work or still with work, but outside of our products,
um, I built an AI agent uh, that could help be my personal data analyst. I named him Isaac um, after Isaac Newton. And my agent has a very deep understanding of Udacity's data tables. And instead of me now be having to become a qu a SQL expert whenever I have a metro question or I need a specific query or I want to run some Data analytics I go to Isaac um first and so that's been really cool to see. So I'm curious to see what other people have done. Okay. So I see uh learning material
courses and um uh some graphics um cartoons presentations. What else? AIPM coach chatbot business plans. I'm super curious to see what this AIPM coach chatbot looks like because that's one of the products that we're always talking about building is a Coach that can help our learners um helping on coding. Yeah, that's something that AI is really good at. Um, so cool. I'll let those keep coming in. For the sake of time, I will keep going. But yeah, super curious to see what you guys are building out there. Um, and the more you build with AI,
the more you're going to really relate with the different challenges I'm going to talk about next. So, this differs a little bit from My previous slide because this is now more in the weeds of when you actually start building. you start hitting these like aha moments where you're like, "Oh, that's not working the way I thought it would." So, some emerging problems, uh, models really struggle to make tool calls, which makes it really complicated, which makes it really difficult to build complex user experiences. So, what are tool calls? Tool calls are basically Um you can
think of them as traditional function calls or other agents. It is one master agent calling a bunch of different things to help fulfill a task. So, an example of where a tool call would be really beneficial is with Udacity AI or Marvin, our in-classroom chatbot. We're expanding its capability to um to accomplish support requests. And we want it to be able to file support tickets or technical support tickets um on behalf of the learner when They're running into a problem that requires a human to come fix a program. And that experience would be Udacity AI
making a tool call or a function call to be able to submit a support ticket. But you'll see from this benchmarking provided by Enthropic tool calls don't work consistently. Even at the highest level, if you say it works 80% of the time with uh Anthropic's latest model, that still means 20% of the time the mo the bot is not going to do what you Think it's going to do and 20% of the time it's going to fail the user. And that's typically speaking the world of product management an unacceptable failure rate when we're talking you
know products working 99% of the time in the traditional world. And so without models being able to perform tool calls consistently it's really difficult to use them. And that means it's really difficult to now start building very complicated user experiences. Um, two Other things when you get into actual product development that's really hard is that there's emerging UI and UX standards and outside of chat bots and co-pilots, people in the industry haven't really iterated past that. So, an analogy to use is kind of like when Apple first came out with the app store. I'm I
bet if we all took a time machine and we looked at those apps, they were atrocious. Um, the same way if we were to take a time machine and look at early 2000s websites, they were not optimized for like great user experiences because nobody had set the standard at the time. And so we're kind of in the same world with AI products is nobody's really built these standards of like how do you build how do you even build an experience around making good tool calls? Like my Udacity AI calls a support agent. How do I
indicate to the user that's what's happening and and make it so the user is not confused and Why did a support ticket just suddenly pop up and why did it get filed and how there's no real UI and UX standards and so we're trying to establish those and learn right now and it requires a higher emphasis on prototyping than I think what most product managers and designers have been used to up until now. Um and then for engineers it's really hard to predict this work. It's not again it's not traditional software development. It's a lot
of engineers having to become Prompt prompt engineers and prompting experts. And from a product management standpoint, it makes estimating engineering time and level of effort really difficult. But the industry still expects you to say these are my quarterly goals and this is the feature I'm going to launch for this client by this time. And so it's become a large struggle to try and balance expectations with your stakeholders when you've entered a little bit of an ambiguous State. Um, so I'm going to go through one last case study of a feature I built at Udacity, which
is our AI translations. So we have an entire now new infrastructure that leverages that was built with AI in mind and and an AI first approach. I don't usually advocate for this like not every time a new technology comes out do you need to rethink things from ground up but AI was so significant and is translations is such a significant um use case for AI That this was one case where at Udacity we took a huge step back and said yeah there's a status quo in how we used to do things but maybe we really
need to start from square one and do things totally differently. So once upon a time we used to extract our nanoderee and courses from our platform, send them off to like thirdparty vendors and humans to translate and then we would reimpport them and provide them to our learners and that was extremely time consuming um Really really really expensive and it was really difficult to control the quality of what we were getting from humans. And so we took a bet to try and use AI to do this. And that bet paid off in in spades because
it is much faster in the magnitudes faster where translations take a couple hours versus human translators take weeks um if not months to get us our translations. It is way cheaper. It's not even in the same ballpark. We're talking hundreds of Dollars versus tens of thousands of dollars. And we The biggest concern was quality. if AI could generate as good of quality as humans. And right now, we're actually finding that our AI translations are generating sometimes better quality results than what sometimes what our human vendors used to. And that's because the AI is given all
the context of our program where human vendors were only really given little snippets and they weren't looking At our programs holistically the way our AI agents are. But this whole system was extremely tedious and really really difficult to build. And I'm just going to share um like a overview of all the different agents that are working together. And this doesn't even include like the business logic that also wraps these agents. But you can see that what we did is build very very specific agents that do very specific tasks. So you know when we start translating
a Course or a nanoderee the very first thing we do is we have an agent or like an AI agent that goes through and pulls together a list of a glossery. The reason we do this is for example we don't want to translate the word generative AI differently on every single page of a nanoderee program. So this agent pulls out key terms and says okay for the rest of the program this is how we're translating it consistently. Then we have agents like a classroom Translation agent and a fluency agent that work together. So the translation
agent is in charge of translating. The fluency agent is then in charge of pulling it all together and making sure it makes sense across a page. Video transcripts are the biggest pain in the butt. Um they are the most tedious. So like we have six agents alone dedicated to video transcript translation. There's an agent that translates, an agent that breaks the transcripts into different Cues, an agent that then runs a grammar, the agent that then comments on the translations, an agent that then refineses, revises the agent uh translations, an agent that's then in charge of
stitching the transcripts back together um to fit certain to match the English cues. And so this is a really great the one the reason I wanted to talk through this is one it's a really great example of how you just bypass this whole tool call problem. It this Does not every product innovation leans itself to this but we have a fixed orchestration here. They are agents that are passing the baton to each other. And so this whole idea of like messing up tool calls we just don't have to deal with because there is no one
agent that's trying to call these different agents. they're just passing baton sequentially. But I recognize that a lot of product um innovations don't lean Themselves to fixed orchestration. Um and another reason I want to bring this up is our translation agents were a great example of where we were using fot prompting where we were giving these agents very specific examples. We were working with mentors who uh spoke these different languages. So for example, for Japanese, we worked with a Japanese mentor who gave us examples of really high quality translations and then we would feed it
to these agents in order To help the agents understand how they were meant to translate the content. But what's funny is we spent a lot of time doing that and then found that it didn't have that much of an impact on the quality. And so now we've stopped doing that um and we're instead trying to go back to a world where we can use um traditional like software logic or business logic to try and confine the outputs from the agents. So it's an example of fuchsia prompting but also an Anti- example where you know just
because these techniques exist does not mean that they're always going to be worth the effort. Um which leads me to the final one of the final slides which is all around fine-tuning. So this came up a lot when I first um started t getting into AI product development. Um at the time people I don't think had a good understanding of what fine-tuning was. Um and especially for our translation use case we thought we would Have to fine-tune to support specific languages. But for the most part as a AI product manager you will almost never really
need fine-tuning. It only really makes sense for extremely narrow use cases and it works well over large data sets, but it doesn't have generalizable results. Um, it's hard to maintain. So, anytime the underlying foundation model or your policies change, you'll need to retrain it. And most applications like just simply don't need it. So, one Example where we could use fine-tuning is um our support use case. So again, we're trying to make Udacity AI the support experience, the first concierge support experience in the classroom. So we could fine-tune the underlying model and feed it a bunch
of data of how our human support team is supporting um learners and try to confine the output so that it mimics that human interaction. But the second the underlying model changes, you're going To have to retrain it. And if we ever change our support policies, we're going to have to retrain the model. So, for the most part, the juice just isn't worth the squeeze when it comes to fine-tuning. But I'm sure that others might disagree and there may be use cases where you have had to use fine-tuning. Um, and then my final final slide is
around evaluations. Uh, this is a really important topic that I've spent dedicated very little real estate to. So, don't be don't be misguided just because I'm speaking about it the least. Evaluations are pretty much your your your number one way to drive high quality outputs and consistency of your product experiences. Um, eval basically you're evaluating model responses. And so you can do this a bunch of different ways. The ways that we've done it is we've come we've curated I've hand curated a list of about 40 questions um that are somewhat some of them are very
Deterministic such as like what is the release date of those cores or what is this program's difficulty level that's very deterministic those are metadata tags that we have that the model should be pulling and responding um knowing that data in mind and then there's some questions such as did this answer the user uh question that is a lot more subjective and so we do LLM driven evals and human evals where we just give a Where a smarter model evaluates the responses. So let's say we have Marvin um answering certain questions and we are experimenting by
changing the prompt or changing the underlying model and we want to see did this improve the output um we can compare those responses so the default to the candidate and use a smarter model to evaluate if it still passes or fails and then sometimes a human i.e. me goes in and I mark it past fail and oftentimes you're trying to Drive this to 100% where it's like if I run this question a h 100 times I should get it should answer what is the release date of this course 100% of the time. Um so that's
how you try to start evaluating where your model outputs are failing or succeeding. Um, one thing that we did learn is early on we try to do evals that are a grade like oh this is 80% or 70% and that's really really hard and it's really tedious and our um even the smarter models just like didn't Really do a good job at it. So, if you're ever going to build evals, um I would highly recommend that you just do pass fail criteria and you just be really ruthless with it where it's like I kind of
got the right answer, you just fail that. Um and if it unless it gets 100% the correct answer that you were expecting, you don't give it a pass. And that makes it really easy for you to start setting benchmarks of performance. Um sorry, so that was a lot Of talking. Um uh my last slide is just around and I think I saw a bunch of questions coming up in terms of what uh courses and programs Udacity offers. We have a little bit of a bug on our website. So these programs do have reviews. Just ignore
the fact that the reviews currently aren't showing. Um but these are I just curated a couple uh programs, but there's definitely a lot more Genai content on our platform. So feel free to browse our catalog. But Relevant to this talk, we have an AI product manager nanoderee. This nanoderee actually launched maybe many years ago, but it has been revamped to accommodate the world of Gen AI. Um, the content developers on this program are wonderful, extremely smart, um, very well-versed in AI. So, there's a a lot of really good content in here. Highly recommend taking it.
We also have our generative AI nano degree program u Which is two months long will give you all of the information about genai that you could possibly ever desire. And then if you're just looking for fluency if you're trying to like get your hands around just terminology or just like a baseline understanding you can take some of our quicker programs which are only three weeks in one week which is fun uh genai fundamentals and genai fluency. So with that I think I'm almost exactly on my 45 minute time that I was given and So I
will open it up for Q&A. Okay. Thank you so much for this was such a great session. I think we all got like a view of what product management at Udacity means and all the behind the scenes. Thank you so much and uh okay to quickly remind you at the end of this session you will be prompted to uh fill in a feedback form for us. So do uh give us your feedback because we want to know whether you like the topic, whether you need like more topics like these, more Webinars like these. It really
helps us and uh I we have received uh questions live and also from the registration form. So I'll be popcorning between these two. Um so first question for you Wanit uh is what should I study to get into the field of product management? Um great question. So I don't want to answer this as if it's like a definite answer and there's only one way to product management. I we have a product leader at Udacity that I think his Origin story is that he used to be a magician. And so you could literally get into product
management taking any path. I want to heavily emphasize that. But um if you want a more direct answer, the typical ways that people get into project management is either studying something in tech, math um or the sciences or studying business. Um I would personally always recommend having some sort of technical background. That does not mean you need to study Engineering. But if you know how to build uh software and launch software even if you take a boot camp or a course that enables you to have the same language to speak with engineers and then if
you have some baseline business understanding that will enable you to set success metrics to understand how to drive business metrics. Um and so yeah the typical ways to get in engineering, computer science, MBA are pretty common. I hope that answers the question. Okay. So the next question is I think uh I'll just get this uh out of the way. A lot of you are asking like whether the recording will be available. It will be available in the community our alumni and learners community. And for the deck we'll make slight changes to the deck and then
uh one do you think uh you would be able to share this with our community members? Yeah for sure. Awesome. Awesome. Awesome. So okay so the next question is uh what is the Difference between product manager and product owner if you want to quickly touch upon that. Oh, that's an interesting question. Again, a lot of squishy topics. Um, product owners. So, it it depends on like the industry. Typically in tech, we don't really have like product owners, but I know more in banking, more in like hardware, you'll have product owners. And product owners, from my
experience, can be a mixture of product managers. They could be product Managers or they're a mixture of product and project managers. But depending on the industry you're in, a product owner fulfills pretty much the same duty. You own this product and it's your job to make sure that it's successful. Whether it means defining what you're going to build about that product next or whether that means like executing against um timelines. So in the world of tech, product owners, product managers tend to be the same thing. I've noticed though In the world of hardware and like
uh banking that they can be slightly different and that they fulfill a little bit more of like a program management role. But that's just that's my experience. I don't call myself a product owner. So again, squishy squishy titles and the the definitions can definitely change. Okay. Okay. And the next question we have is how is the feedback from users incorporated to the model? Is it Automated or reviewed by humans? It's always reviewed by humans. So um there are certain types not so much feedback but certain interactions that in the future we're going to look at
automating. So, for example, for our translation service, we have built up a program around it where our we're hiring contractors that are specific to those loces to review the quality of our AI translations and we're automatically we're tracking what edits they're making And we are discussing if there is an automatic pipeline to take those edit histories and refine our model output using that. But in the meantime, it is much safer for you to manually review it. And this might be the case forever. Model outputs are very very sensitive to what you input. And so building
a pipeline that automatically takes like any random user feedback and feeding it directly into the to the model without a human curating it can be Really difficult and lead to bad results. So for now, we yeah, we just it's me. I read all of the feedback and then I decide what problems we have to solve and then we work together engineering and I to decide how we're gonna fix those problems. Okay. Now taking some questions from the registration form. Uh our first question is how do we make sure that we keep ourselves updated relevant with
Genai products and how do we efficiently use It? Yeah, keeping up to date with Gen products is one of the biggest challenges um in tech like right now the best way to keep up is first build yourself an a baseline understanding like part of the reason in my presentation I wanted to cover some of this terminology is like you kind of need to know where you're mapping all the new information coming out. Um, so taking something like a fluency course Or a fundamentals course is really helpful if you're not there yet to just like have
that baseline foundation to build on top. And then after that there is no secret right now because this indust because Gen AI is innovating so quickly. There's updates every single day. Um and it's so new. There is no university program a boot camp or anything that you can really take. The best things that you can do is pay attention to the big devel the big Companies like OpenAI, Anthropic, Google. See what they're launching every week, every month. Read their blog posts. really try to understand what is changing and then the biggest suggestion would be just
use it. Use it every day in your life. If you are a product manager, use it to develop your slide scripts. I'm not saying I did that, but I might have done that, you know. Um, use it to do your data analytics for you. Use it to create your emails for You. The more you use it, the more you'll understand it naturally and you'll notice the changes happening. Um, that answered the first part of the question. I've totally lost track of what the second part of the question was, but I don't know if I answered
all of it. I think that was all of it. And uh I think when you shared about your translation agents, we have a question from one of our attendees. Are your Language translation agents built off of other agents such as OpenAI, open source resources? Yep. So these are still just when we talk agents, when we talk um uh tools, they're all built on those foundation models. And so we what what you do is you build an application that calls an API. And that API what it does is it sends the foundation model your question and
the foundation model sends you the response. And then in your application, you can make it look all Pretty and you can decide what to do with it. But everything's built on top of these foundation models. So our translation agent is built on top of GPT which is OpenAI's foundation model. Thank you. Our next question is can you talk about some challenges you face with developing new AI tools and initiatives? Um yeah the biggest challenges one is again like testing and controlling the outputs. So, we built a summarized video button which when I test it, it
works Exactly like I want it to, but then I put it in front of a 100 users and it's failing for 40% of them. So, getting visibility into the performance of your products um is really challenging and really important because you live in a probabilistic world. You can't guarantee that it's working every time. And sometimes it's not even a factor of the model. Like sometimes the systems that are the APIs that are trying to call it aren't set up properly. So, you're Debugging at multiple layers. You're not sure if you're just not calling the right
uh you're not pulling the right information from your database or it's because your prompt isn't set up correctly. Um, another challenge is prompting is extremely tedious. It's it's extremely tedious work. Um, even within Udacity, we have some engineers now that are becoming kind of our prompt experts and their job is very tedious. It's not it's not an easy job to try and Craft why your output is not um the what you expect it to be. Um with our translation service some of the things that we encountered is like the use case can really impact the
output. So, for example, um our translation service worked wonderfully for Spanish because these models have a lot of Spanish. There's just a lot of Spanish content online for them to pull from. Um we've really struggled to deliver Japanese, which is a lot trickier of a language, a Lot more nuance. And so, there's like a broad broad um bucket of challenges we hit whenever we're building AI products. Thank you. And uh next question we have like are the courses that you showed Vanit is are some of the courses free also uh at Udacity or do you
have any other external sources like some blogs or uh you know sources that you recommend to people who want to learn more about generative AI especially like for a product manager. Unfortunately I'm not the best uh resource for external resources and courses. um obviously because I just consume Udacity courses when I need courses and then when it comes to just like having a good understanding of um generative AI it's a lot of reading anthropics blog post open AAI's blog posts um I spend a lot of time talking to our engineers as well they're really in
the weeds of documentation so they've become a huge source of my knowledge uh That's my kind of cheating way to stay on top of things so unfortunately I'm not the as uh person to talk about other like resources off of Udacity, but I'm sure that you could probably ask chat GBT what resources it recommends and it'll give you a bunch of good ones. Awesome. Awesome. And uh one of our learners is asking, is there a significant planning and delivery timeline changes with Gen AI and normal product launches? Yes. Um that's a fantastic question. Yeah, huge
timeline changes. So, first, like I mentioned, um, because we don't have UIUX standard set up, anytime you're trying to build these exper experiences, you're almost always now in a a world where there's nothing you can copy from, unless you're building a chatbot, there's nothing you can copy from. And so we are spending a lot more time doing designs and actual functional prototypes which in my experience in in Product management just hasn't been as necessary because most of the time you can say oh I'm going to build this feature and I don't really need to prototype
it and test it heavily before I build it because I can see that Apple and Google and Microsoft have all built the same feature in a very similar way. Whereas now I'm trying to build a content creation co-pilot experience and I have very little things to copy from. And so how do I know that the users are Actually going to like it and it solves their problem? We have to build real functional prototypes, put it in front of users, see what's happening and learn for ourselves. And so there's a lot more time spent in testing
and iterating and ideulating or sorry in uh prototyping and ideulating. And then also a lot more time spent in testing and like perfecting. So we can put out experiences that work 60% of the time and now it's a tedious process to try And get it to 70% and 80% and 90%. So it it definitely does impact a lot of the timelines. You can shorten those timelines by being very crisp and very specific with what the feature is that you're trying to build. Like a quiz AI quiz generation feature we built really fast. Our studio co-pilot
took more like a a quarter and it's still not perfect. So I hope that I hope that helps answer the question. It's a little bit generic but the answer is yes the timelines are Definitely impacted. Thank you. And the next question we have uh I think a lot of learners have sent this similar questions that how do you stay ahead in AI product development as everyone around you is doing something similar. How are the expertise requirements changing? How do you stay ahead? Yeah. I everybody's in the race together. So I can speak for edtech. You
know, we have our Udacity has its own competitors um The Plurals sites, the Corseras, the UDME of the world, the other edtech companies, and we are paying attention to what they're building. They're paying attention to what we're building. But it's kind of a cool experience because it's been a really long time where I feel like everybody's been trying to figure out how to make the maximal impact of this technology together. Um and so just staying ahead again like there is no silver bullet here. Um it is One paying attention to the market and seeing um
what capabilities are are being developed. Two it's having the courage to actually challenge yourself to break the status quo. Um I'm really fortunate to work at Udacity where there is a huge desire and um support to break the status quo. So we have a lot of freedom to say this is how we're going to rethink this entire thing using AI. But it's it's important if you want to stay ahead, you have to be really good At first principal thinking. You have to look at what your existing tools and product experiences are today and ask yourself
like if I were to start again, if I had built this company assuming Gen AI was always here, how would I have done things differently? And then you build a road map with that. that is like the only way to really stay ahead in this world. And so one of the questions we asked I asked myself when I was building studio was how would Udacity Have built content differently if we always had Gen AI in our back pocket? And that starts to create a vision for how your tools need to change and then you look
at what you have today and you slowly walk your way towards this future. Or you do what we did with localization and we just throw away the old solution and we start from scratch. So um yeah, staying ahead is just it's challenging yourself to start from square one again and finding like Crafting these visions, being creative, pulling in different experts. Um in terms of maintaining expertise, again, no silver bullet. It's just about you, you have to be in in the game. So if you're not at in a position like mine again where I'm really fortunate
to be leading these type of innovations, you need to be challenging yourself in your what your current job or schooling is to be using AI in every chance that you get. Um instead of doing Google Searches, you should be doing chat GBT searches. And so that's the only real way to like keep your expertise up to date. Thank you. And the next question There was one question, Connie, I don't know if you're going to get to it, but I actually think it's a really interesting question that I saw come in on the forums. Um, one
that like I have my personal perspective on, which is how to break into product management. And I Know that's like a really big one. Um, because product management is a competitive field. It it can be tricky to get into. So, I have a little bit of a multi-pronged answer. Number one is obviously if you have like computer science or business background or the specific um education plus internships that's kind of a no. I'm not saying anything new that will help you get into it. But another the path I took was product Management when you're new
it's like how do I get this job if I don't have the experience that they're looking for because they're not giving me the job to gain the experience. Product managers are it's a lot of soft skills. Communication, timeline management, execution. It's a lot of soft skills and you can get those skills in different roles like program managers, project managers, um in a ve a variety of different industries like in banking, in Um hardware where the the competition is a little bit lower and you're still gaining relevant experience. The thing that you're not gaining is like
that product vision, strategy, um like really defining a feature type of experience, but that's usually required in more senior PM roles anyways. So, when you're trying to break into an entry-level product position, you can focus on those other skills, project execution, timeline, and you can go look for jobs That let you gain those skills. So for me when I was trying to break into product management I actually worked as a program manager first and I did that intentionally because it's a lot of transferable skills and also program managers tend to work with product managers so
you have good networking opportunities. You can learn from them. So I started at Salesforce as a program manager. I reached out to a director of product management that I worked closely With. I asked her to give me product management type of projects that I could do along with my program management job. and then slowly I was able to transition into her team. So I wanted to share that because I know that's a lot of anxiety for a lot of folks. So there's a lot of creative ways that you can get into product management is not
just that you have to follow one linear path. Thank you so much. That was one of my questions as well. And uh the next one We have, how can product manuh managers balance the need for AIdriven automation with maintaining a human centric user experience? Yeah, that's a really really good question. Um I think it all comes down to you need to let your users tell you what they need. And so on one hand, you as a product manager should be really mindful about what types of experiences you're trying to deliver to the user. You need
to understand your user. What Are they trying to achieve? What are they feeling? For example, a support bot that just follows routine business logic might solve the user's problem, but you're talking to a user who's really frustrated that their product's not working the way that they need to. Maybe they were trying to reach a deadline and now you block them. And so having a bot that just gives you the answer but doesn't make you feel better about your experience is probably not as good as One that does. And so it starts with first understand your
user. Where are they at emotionally? What is the task they're trying to accomplish? Then you define the experience that really solves that problem in a very like nicely packaged way. And then you put it back in front of your users and say, "Are you happy? Did this work well?" You survey them. And then you'll learn from that feedback where is AI automation totally reasonable we don't need a human-driven Approach or where do we need a more human touch and a lot of um the products that we're building at Udacity we are assuming that humans need
to stay in the loop like studio for example is far away from that vision where it started from which is it's going to be a one-click program generator and now it's all about how do we enable our authors to build highquality programs together. And so, one analogy I always like to use, one company that I think does a really good Job at this is Apple. Apple's always looking at ways of how do we merge technology in humans. Um, if you've ever had to call Apple support, for example, I had to call them to do an
Apple Care replacement of my iPhone. At first, it starts with like a support uh phone experience, and it's just me pushing a bunch of buttons and telling them what I need. And then immediately they put me in touch with a human and the human's like,"I got you the entire step of the Way. I'm going to track your return package. I'm going to track you." And so Apple understands that this was a really frustrating experience for me. I'm going to put a human in, but I'm going to have AI do a lot of the tedious stuff
up front. And so really learning how to balance that experience. It all comes from like understanding your user like what do they need and what's the best way for you to deliver that. Oh my calendar is being shared. Um I'll Stop. I stopped your screen share. Okay. So yeah. So um one of our learners is asking can they access uh products like studio that you showed in the presentation? No not today. These are all internal products aside from Udacity AI which is in our classroom. Okay. The next question is very interesting. I know the internet
is divided. What according to you is a Better is the best uh AI product chat GBT or any other [Music] what is the best AI product? It's hard to say because so many of them do different tasks. Um I think generally if you use Claude it has better outputs than chat GBT. If you're just talking from like me using it for my personal day-to-day um yeah I don't have a straightforward Answer because like you know you can't comp compare Dolly to chat GBT because they're doing different things. One's doing image generation um one's doing text
generation. Um I think all of them have really good API documentation. Um and then when you look at benchmark scoring like just very like technically speaking they all like some do better in certain benchmarks and worse in others. So um yeah the deepse comment around data privacy being zero Is an interesting you know deepseek's pretty good but you decide what data you feel like feeding it. Um so yeah I don't have a straightforward answer. I think cloud's better than chat GBT but that's that's probably pretty much it in terms of my opinion on that. Yes,
that's a that you know that's an interesting question. Actually, a couple a couple ones I could if you haven't used VO, definitely use VO. Um that's a great tool on the market in terms of Learning how to code. when I was using VO to build some prototypes for myself, I was I I was thinking to myself that I don't understand what my intro to programming um university instructor must be doing because this what they took four months to teach me I learned in like an hour with VO. Um and then cursor is also really amazing.
If you're a software developer and you haven't started using cursor, you must you should. Um so yeah, I can add those. Thank you Anit. We have another question on certifications for becoming a product manager. And there was also a question regarding is PMP certification important uh to become a product manager or any other recommend uh recommended certifications uh you know in your experience? Okay. Uh, so I I don't have too much experience with the certifications and I don't personally know too many people in my network that got into product management with Certifications. From my understanding
like PMP and those different like product management certifications become really beneficial again in different industries like banking. I know a few product owners that did PMP and became product owners in the banking sector. Um, and also in like hardware development. to like companies like Tesla um or Apple where you're working like hardware lines in terms of like my type of function. If you're trying to Get like a product management role at a big tech company at like Salesforce or Facebook or Google, the certifications become less important than you being able to demonstrate that you can
really build product. And so you don't need again like one way to do that is to get project or program management or similar types of job roles at these companies. You can also go into product marketing roles and take that avenue. You can go into business analytics and take that Avenue. Um, and also just personally, are you building apps? Are you launching websites? Like anywhere where you can show that you're able to drive something that doesn't exist and make it exist like that is a skill set that people are looking for more so than a
certification. Okay. And we have time for I think one more question. and then we'll wrap it up. Uh Okay. I think one that we can take is is there a difference in product manager dealing with Gen AI products and from the ones dealing with traditional products. Um difference in terms of like are you a different type of person or did they hire me specifically because I studied AI product management? No. Um, every product manager, every engineer, every data scientist, every marketer, every every single person is ha going to have To transition themselves eventually into an
AI, whatever you are. Um, I just happen to be doing it early, but people are across the board, everybody's going to have to really master AI skills. There's a really famous quote um going around that says, "AI is not going to replace your job, but somebody using AI will." And so, no, there isn't a difference between an a product manager and AI product manager. The AI product manager is just a product manager who's Using AI and then building AI products. Um, but everybody's going to have to go into that realm eventually. Hey, I think that's
it for today. Uh, we have covered most of the questions. For the rest of the questions, we Vanit will answer and we'll post it in the community in a follow-up post. And uh, okay. So what a great session Wii. This was so so interesting and so you know good to learn all the Udacity uh you know products that you've been working On. Uh any parting thoughts for our audience? Uh good luck. I hope a lot of you get to uh interface with AI products. If your dream is to get into product management, I wish you
a lot of success. Um and yeah, thank you for attending. I hope this was helpful and hopefully see you guys in future webinars. Thank you so much everyone and again like at the end of this webinar there will be a survey feedback uh link and Please share feedback with us. It really helps us bring these awesome webinars to you. Thank you so much for joining. Thank you so much VN. No problem. [Music]
