Welcome and hello to everyone I'm Jeff Easley I'm the leader of the responsible AI Institute we're so excited to have everyone join us um what a great response terms of registrations and I I know we'll continue to see more and more folks join but in the first few minutes I wanted to just introduce myself and talk a little bit about the Institute and before I hand it off to our amazing panelists that we have assembled today So again I'm Jeff Easley I've been with the institute for about six months now um you know as it
is the time of year where we come to the calendar end and we look back on what we've done this year it's been an amazing year for us we've added a number of folks to the team a new head of growth several amazing talented policy analysts as well um and we've also brought in a number of of big corporate members that are already active with us we're already partnering On initiatives and doing amazing things so just wanted to say welcome to KPMG Dow k gpack Ally uh one trust and further welcome to the family and
we so look forward to the partnership and working with you guys as we move out uh looking back in the year as well we saw you know huge growth in our activity with members just this meeting alone is already double what we saw last year so we're excited that the engagement has increased over the year and more and More people are getting involved with responsible AI you know again that's our mission that's what we we do we we we're a nonprofit that helps the community unlock the power of AI um and in the most responsible
way possible and so that's what our team is dedicated to doing we're very passionate about it and as someone who's come from and you know 30 years in very large corporations it's it's so fun and fulfilling to be part of a very Mission oriented team very Passionate about responsible Ai and so couple of things that we also did this year to mention we launched our Hub so responsible. if you look for the Hub there you can join the Hub you can join our um Institution for free as an individual member and get access through the
Hub to all kinds of incredible resources that the team produces we've also launched a monthly series um we've done a virtual event series we we just started yesterday with our Roundtable Series um which is much smaller but you know in the chatam house rules type of environment to really drive that Frank Earnest Open Candid conversation to help members learn from from each other on how they're doing in their REI Journey um you know several of the things we've been doing lots of very impactful guides and research has been recently put out there one of our
most popular is our AI policy template we've also just done something recently with independent Reviews and and REI governance and co-published with our members uh such as BCG and and um and one trust so we're we're very active uh in the space and and we're very happy to see the engagement and all the folks that have joined us for this absolutely fantastic panel um we we've seen you know tens of thousands of downloads in terms of our subscriptions and our our uh great digital assets you know we we we just really enjoyed growing and having
a Bigger impact year-over-year with our mission and just enabling you and the community and and organizations to do the right thing unlock the power of it but obviously with an eye towards responsible AI so thank you all for your membership and your engagement with us us um and you know I'm very happy to be here look forward to talking to you next year about all the other great things we're going to be doing so it's fantastic and without further Ado I Wanted to um turn things over to our head of of policy and delivery uh
hadasa Duart she has assembled an amazing cast of luminaries uh in the in the AI space and with a you know key focus on responsible Ai and how that plays into governance so um without further Ado I'll hand that over to you hadasa to take us away yes thank you so much Jeff um good morning good afternoon and good evening to everyone attending I we have a pretty International audience today so that's really great I hope they get a bunch of really interesting perspectives from every uh one um welcome to the race 2024 event I
Echo all of the words that Jeff uh just spoke uh and want to welcome everyone to the panel session that we'll be holding um today the final panel session um or virtual panel session that Reign be holding this year uh we'll be focusing on lessons learned over the past year uh in responsible Ai and AI Governance trying to understand what has worked what has not and what organization should be looking out for as uh we head into 2025 all of that especially this last virtual event series of the year has made me realize that effectively
we only have a couple of weeks left until 2024 is over which is pretty crazy it feels like yesterday that 2024 took a start and we were talking about the executive order and the EU AI act Proposal and now we are right here I think we can safely say that the market has matured uh in many different ways both at the policy level at the implementation level we're looking at very different questions than we looked at about a year ago that's why I'm really excited to have um or to be leading hosting this panel uh
session um today without further Ado I could go on and on talking about everything that we've learned over the past year but I Will spare everyone from those insights right now and leaving for this session um a brief intro on my end I realize I haven't done that yet Jeff just did it but my name isar director of policy and Delivery at the responsible AI Institute uh and I am joined by an incredible lineup uh of panelists today Courtney a crumby founder and CEO at AI truth um Asha Saka who is the founder and CEO
of women leaders in data and Ai and the AI Factor Institute and finally we have um Lama s a senior policy adviser at the Canadian AI safety Institute want to make a brief note that questions can be asked throughout this session you can add them in the Q&A um section I'd be happy to address them either in the Q&A at the end of this session or throughout where the questions may fit in nicely so I will leave it of that and I really want to hand over the microphone to our panelists today maybe starting with
you Lama since you're right at the top of my Uh screen right next to me then heading to um Courtney and finally Asha if that's okay I'll hand the mic to you thank you uh yeah so my name is Lama s I'm currently a senior policy adviser at the AI safety Institute here in Canada um so my main role now what keeps me busy is trying to identify and uh set the priorities for the in Institute but also support the development of the international Network work of AI safety Institutes and briefly about my background I
spent the past decade working across industry Academia and the public sector on um on various technical research and policy approaches uh to advance responsi AI even before the field uh existed um thank you again for uh for inviting us and um looking forward for the panel I'll pass it back to youa yes thank you so much much Courtney to you um Courtney ABI here a founder of AI truth and nonprofit dedicated to AI ethics as well and responsible creation and use of AI um I'm also on the um on the editorial board for the first
academic journal for the AI and ethics for from uh Springer nature just want to give them a quick shout out because they're doing such great work with white papers um and also on the board of the College of information for University of North Texas um all um focused on responsible Ai and both from the student Uh level on up through Executives right so I acquired my I AI experience when I led a shark tank style AI incubation um uh effort at IBM as an executive there I got to travel the globe and meet with the
Fortune 500 and their data science teams and get a good look at how the operations were going on behind the scenes for AI and honestly at that time in 2016 2017 2018 it was a hot mess and it kind of still I mean I hate to say it But we're still not quite there yet you know or otherwise we wouldn't be here talking about what we can do better right we can always be improving on this um prior to that I led one of the largest uh Chief data and analytics officer communities for IBM but
but the AI incubation is where my pass passion for AI ethics uh came in so much so that I had to go write a book about how it affects us in our literally our everyday lives and and the things that you do as As AI folk um you know affect everybody everywhere in their everyday lives I just can't impact that I mean I can't say that enough about the impact that we all have here absolutely all the more reason why it's so important to talk about these topics thanks Courtney Asha uh thank you for first
I want to thank hadasa for responsible AI for putting this panel together uh every time I see Courtney I have a smile on my Face because I love seeing you I love to be on the panel with you and I love the book you wrote and the work you're doing so huge shout out for Courtney llama nice to meet you and I'm so honored to be on this panel with all of you um I'm gonna have a great time discussing this important topic but I'm really happy for the audience who's here and who will be
uh not only listening to us but participating through Q&A so I'm really excited for that I'm Asha sakena founder And CEO of uh wilda women leaders in data and AI we are incorporating world leaders uh within our organization to make sure that uh the the male leaders are equally responsible and participating in creating this movement because we can't do it alone we have to make sure that everyone comes together to build a responsible AI uh framework bring par gain equity in our world and make sure we have equal opportunities uh I also wrote two books
uh the AI factor And the AI Factor uh work workbook uh which now led to building the AI Factor Institute uh where we do AI strategy workshops and U make sure that the organizations are on the right path of building the framework and the picking The Right Use cases for implementing AI uh and I also teach at Columbia University I'm a professor at The Graduate School where I teach entrepreneurship and AI so I'm really happy to be here discussing this topic Thank you thank you so much I'm really excited to have all three of you
uh on board and maybe we can kick off this session by going a little bit back through the basics and kind of understanding the lay of the land I think that will provide us with some good context to understand where some of the gaps exist and what as I said at the start you know what works what doesn't and and what organizations may need to look out for so importantly I'm heading back to you um you mentioned that there's still a lot of questions that are open right we're not quite there yet um I think
some of the questions that we get you know at the Institute starting from awareness um so kind of like that why like why should I care about responsible AI to the controls what what do I need to put in place in order to make sure that we're doing the right thing and then finally how do we actually Implement that so That's the the how so from the W to the what to the how um in that AI governance and responsibility tend to overlap a lot it's I think that there is a more traditional divide between
the two but now they seem to be more or be be used in a more synonymous way used interchangeably so I wanted to start with you and get a better understanding of how these two um fields or essentially how responsible Ai and AI goverance merge uh how they Diverge uh and kind of like where they originate from yeah and we could I mean we can talk about terms but I kind of want to really just focus on what does this actually mean to us in our jobs as people who I assume us who are on
this um listening and also you know just amongst ourselves that we are in some way shape or form either influencing the development of or actively involved in the development of AI so just to kind of Set that context um I would say responsible AI should really be the result I mean if we looked at the ideal world the responsible AI is the result right of really great AI governance and the AI governance would be the mechanism to get there but I think in today's version of AI governance because we're so nent just in AI development
as a whole I think still I think there's a ton of stuff that we know but there's a ton of stuff we still Don't know about how we're working on AI together and how our norms and our cultures and everything's blending around how we're developing so I I want to back us up and say well I think that the the now version of AI governance has more to do with when and what and that when is when we start considering the governance process which is usually something we only govern things after they're developed right which
I think if you're going to have a True responsible AI product at the end of the day you're going to have to back up to the very beginning like is this even a Humane use case like is this something we should even be doing could this cause more harm than good if it's released out there or or or do we truly think that it could bring more good to the world than than than anything else and it's worth doing sometimes we don't even ask ourselves that and and I'll give you just an example um you
know and It all kind it all depends on where we're starting in in in our thinking about and when we think about designing some of these and creating some of these we talk about um the process of actually creating getting the data getting our hands into the the mlops parts of these things and then we we we talk about governing it you know once once we release it and then past the release even so there's like this part of this middle part and that gets governed so From the moment that the data scientists take it
through the through to the to the end moment where they're kind of releasing it but there's no like what's happening with the executives when they're thinking about the the the thing that they want to do there's not a whole lot towards the not not a whole lot of meat towards the end like once we release it in the wild do we have the ability to rectify it if it goes wrong or remediate some situations if it turns Into a crisis situation like you know hey we just gave up whole bunch of people a bad credit
score or something like that what do we do what do what is our what is our crisis management around this AI product I don't see a lot of that whole end to end process from the very beginning of the consideration all the way through to the very end past the wild past the point that we release it we just kind of go oh it's released and now we'll just keep maintaining our part Of it but we don't go so far as to like even train users sometimes on how to use these things what's the appropriate
use of these um AI systems and what's not an appropriate use of these AI systems because we all know these AI systems are very expensive to produce users are going to be just as Thrifty as we are about trying to reuse them for things that you know maybe they're not meant to be used for and that suddenly changes the whole dynamic but let's just quickly Define what is AI governance as we know it today it's a set of policies procedures software tools sometimes it's as simple as checklist if you're a startup you're you're just like
well give me the checklist of things I should be thinking about principles um that kind of aim at reducing all of these big risks risks like human safety issues privacy security fairness um robustness correctness things like that surrounding the AI development and use that's what AI governance actually is today um and that's kind of a little bit vague you know um and and also there's varying degrees to which companies are carrying out these governances right now and a lot of the lawyer folks that I work with are like well I hate to say it but
you know AI governance is so complex that until we get to the point where we've had an O crap moment you know um we don't know if people will actually you know fall in line with doing enough About this and I hope that most companies ought not to have the O crap moment when they're facing down illegal struggle or something more horrendous quite frankly I hope that we can start having these conversations and start talking about how do we map this out from the very moment that we're even thinking about a use case all the
way through to how do we remediate so that's a long that's my long um and I'll just say one last thing so the responsible AI Part to me is also not just about the development and the use but it's also about the norms and the culture um that go into it um from the beginning to the end and how responsible we actually feel as people who are developing this towards the people that um we may be releasing this um AI system on into there's so many points that are extremely interesting and I I do acknowledge
didn't give you an easy question there to summarize in two words So no worries at all sorry yeah that was definitely not words not not a at all I think there are a number of really interesting elements especially the last point that you mentioned on you know the critical thinking about what are the values you really want to embed in the technology that you put out there um can be blurred a little bit where you focus only on the checklist AI governance process would love to get that uh into that a little bit more but
I I want to Turn to Asha and ask kind of follow up on what you were mentioning um before courting about how you know these o crap moments may happen I like how you phrase that offal that's an official term oh crap moment I think the audience is picking up on it as well so um sometimes you need those moments uh even though you yeah you know there's like this balancing act like there there is like a part of this work that's a lot of Testing and you're going to feel doing that and I think
it's okay to acknowledge that obviously there need to be guard rules in place not not okay to say like oh we're you know growth is all that we're looking for Speed we need to get out there we need to do this otherwise we you know we won't be able to survive in this uh the this AI Market um there is I think a distinction across levels of maturity um different sizes of the organizations different types of Organizations the industries that they work in um so I wanted to move over to you Asha and kind of
get a better understanding of what you've seen out there in the in this field in terms of maturity um what what considerations have you uh come across for organizations of different sizes and their nature of work yeah great great question because really if you think about as you said rightfully uh soas that uh in a lot of organizations are at Different level you know they could be based upon which industry they are at or where they are located geographically right so it depends upon the industry or the location uh and I think Courtney mentioned that
2016 right he said a lot of the organizations were not there and guess what they're still not there yet we are still on the early and midstage of adoption we cannot say that we are sophisticated yet we are still very early stages we are recognizing the Importance of responsible AI people recognize it for sure but they're struggling to figure out how you connect the dots how do you take these principles and align them to strategic and operational framework so there are truly a lot of questions and I think that's why these conversations are so important
and sharing real world examples so listeners here or the organizations can take some nuggets away with them and say all right this this Makes sense to me maybe everything we discussed today will not make sense but even if you walked away with couple of bullet points that you can walk away and actually Implement I think that would be a win for us here and I would say that that I personally see that this as four stages organizations could be you know the first stage could be where awareness you're just learning to understand what is the
importance of fairness or transparency or accountability or Ethics In AI systems right so you're just being aware of it learning about it and the second stage I say that the organizations really go through from awareness to experimentation now they say okay now we know what we need to do and now we're going to Pilot this project using the responsible a principles to align our business goals we always talk about what's the outcome so you can't run away from the outcome or the business goals you want to make Sure they are aligned and you can create
that differentiation or still focus on the competitive market markets but you want to make sure that you're putting in the framework putting the fun Foundation as you think about experimentation even in your pilot stages and then comes the policy development stage where you're saying all right now we ready we now don't need to look at our internal ethics policies and governance structures and again I agree with you Courtne completely the AI when you talk about governance it can be a complicated world because there are so many cooks in the kitchen you know when it comes
to governance but really internally you can start crafting what your AI ethic policies and governance structure would look like uh and then comes understanding your operational challenges okay now we ready we're trying to oops or as you quot you said you know oh crap moment these are The gaps you know we are not ready we can't be you know we don't have policies implemented for fairness or transparency or accountability or ethics now you start seeing what the gaps are you know so you can start figuring out where to fix and how to implement but I
would also say that when you want walk away and you really are thinking about practical steps I would say it's really important to establish AI governance you know even though there are going to be Hiccups they're going to be you know issues you want to make sure that the governance Journey starts when you're thinking about responsible AI you want to invest in explainability and transparency tools you know there are tools available now Microsoft Google there a lot of large tech companies which are providing you tools so using those tool investing on in those tools so
you can explain and make your data transp transparent you know for the Stakeholders is critical and one one thing I would say and I would walk away by saying diversity in teams is really important you know I talk about having women but minorities is important it's important to make sure that you have not only having women at lower level but it's important at the decision level we have women representation and we as I said earlier we can't we can't do it without men so we need the men to be in this movement to make sure
that we focus On diversity in teams I talk about gender but there's also talk about we need to think about thoughts diversity in data sets diversity in culture you know so it's diversity will bring you robot especially now as you're building AI Solutions it brings you a lot more uh wealth in your ai ai tools and you know uh applications and then you can talk about Frameworks or education or training or Community or ecosystem which is all Important you know but really thinking about what are the steps I can take to make sure that I'm
moving forward from one stage to the next stage and moving systematically yeah diversity is absolutely key I think and I've seen these questions in the chat I just also want to make a note I know you guys have a lot of questions thanks for all of those feel free to add them I'll try to address them as we go and of course panelists also J of you if you want to Address any while we having a chat feel free to um but there are have been a couple of comments around um AI governance is a
team sports who does ownership for these efforts live with so before I head to ulama and uh did open up the Pandora's box of AI governance framework standards and regulations which I'd love to hear more about just want to follow up on one thing with you Asha which is that ownership piece right like diverse teams are good but you're Going to have to have some decision making power when you want to move forward so where does that piece fit in and which you know you know what roles are are best fitted to um to take
that that role on you know uh ownership is really critical you know and and as I said the too many cooks in the kitchen can you know make create a lot of confusion uh a lot of organizations I work with uh are creating AI Center of Excellence so they are creating uh or AI Ethics uh Center of Excellence so they are division they're creating functional groups which own uh and have the responsibility and I also the AI teams are now getting uh directed directly to the revenue Center or the Strategic centers so I think that's
the right direction because when you're thinking about AI it's not just a tech functional area AI is truly can be incorporated into business into strategy into operations into and and ethic is The foundation of that so I would say the ownership has to lie into the and for organization to really think who the person would be who would own it and be responsible for it you can't have 10 owners you need to make sure that there is a and and based upon your organization if you have a center of excellence can you have somebody there
or if you do have a AI head of AI uh applications can that person be directed to a strategic or operational or Revenue Focused role which would actually make things happen thank you so much I think Asha if you're uh or I mean sorry elama if you're okay with um talking us a little bit more through I guess like there's a big component in um in in this whole discussion around what we just mentioned Usha with you as well the maturity of organizations um there is a big component that I guess relates to what Organizations
can actually do in terms of implementing um different steps at the like at this AI governance level across these different levels of maturity now we're seeing for instance regulatory Frameworks being pushed out there and I know that you work at the Canadian AI safety Institute uh obviously will uh will play a big role in this but I know you've been involved in standardization efforts before this as well um so I want to kind of get your Take on how these efforts need to be contextualized to a what the level of maturity is of an organization
what industry they work in and then also when we're talking about this topic of ownership um who has ownership within an organization for their AI governance effort right now most of the standardization and regulation efforts that we're seeing are holistic I would say in their approach right they're not Tailored to a specific domain they're not as much tailored to um spe specific use cases trying to take like this more broad approach I would love to hear more about how you see standardization efforts play a role in um I guess digging deeper into the the guess
what you need to do the house of AI governance great thanks um hadessa so first I want to start by maybe demystifying the standardization work I think uh the word standards is being Used to mean a lot of different things so could be like ethical standards or technical standards or Professional Standards Etc but also like um so what I would like to use this term for is more to refer to the various types of efforts that are trying to bring consensus on uh practices and like synthesize um what's what's maturing what what kind of approaches
are actually maturing and we can agree on so I want to include um both the you know sdo standard development um organization uh standards so things like um ISO uh or Cenac in Europe Etc but also looking at other pre what I like to call like more pre-standard isation effort that are happening in the field and that would include uh um uh products that are being developed by for example the safety institutes um the the code of practice currently being developed um in Europe uh for general purpose AI system Um or um other community and
Industry uh efforts such as uh you know building uh benchmarks uh and eval like developing standardized evaluation tools for the community like like ml common work so all all of these are really needed um to to to to start finding consensus and conversions um so in terms of um custo like you mentioned whether like the the level of contextualization I think we still need two different types of uh standards or Tools there we we need um a general process processes General practices that could given that you know AI is a general purpose technology and um
some of the approaches are similar for example like uh developing processes for risk management like the work that is being done in the 4201 ISO standard or or for example developing General process for red teaming or um or standardized evaluation approach not not necessarily standardizing the metrics or Standardizing uh risk specific um measures but like the whole approach landing on common approaches but also we need um and that could ALS that would include also um efforts at the model level at like um what what should be done uh for large model developers what is feasible
to be done at the level of the model and what is needed versus versus what is more needed to be done Downstream on a context level on uh for a product for a Specific product so this second layer is really important as well which is like context specific product specific um and this is mainly because the risks are very like a lot of the risks that we will see emerging and we see applied are very context specific and for example developing the the RIS uh risk thres old or like what is the tolerance for uh
for the risk that um is accepted in a specific industry and that would really um uh need the input from that specific Industry and like understanding better the use case and I see like even on broadly more on the regulatory approaches that are being um developed yes we have like efforts like the eui act that is a bit more um I would say um horizontal uh even though even within the AI act they identify different types of fuse cases as well that are allowed or not allowed uh but a lot of um other jurisdictions like
whether in the US or in the UK they are trying to Le Nonexisting industry um uh Regulators so you know there is something common there is some kind of General uh um a code of practice or um a level of commonality to um to to to adopt but also like how can we um work with specific in Industry Regulators to F to um to identify those context specific um um requirements um and I want to just also maybe mention another point that you Said like about the ownership uh at the different level uh of development
uh which is a great Point by Asha um I think the a big conversation that we had over the past two years across the field is how can we ensure accountability across the whole value chain so AI development now is very you know Global and uh and has like multiple layers and we are seeing um Upstream models like the big models that are being developed Um basically empowering a whole ecosystem of new startups developing you know products based on sometimes just Implement using an API from these big models and so to be able to like
even like regulate this industry we really need to understand um at every level of this value chain what is feasible to be implemented and what is needed and when I say um those are two important aspects what is feasible like you can't really expect Upstream to identify all the use Cases that Downstream could be used but they need to do the thinking to start um to start maybe um predicting or like like um understanding how it might be used but then uh the downstream developers would need to really do a better job at contextualizing the
risks but but um and but for example the Upstream requirements might be more on improving transparency and like what what uh what what had gone into the development or the evaluation of these Models so that the downstream deployers can actually test their own models and understand the limitations of their products um so that's another thing uh ownership and accountability for for various actors in this development is is really important to uh to the whole industry thank you so much Lama I see we have a lot of questions coming in so I want to give you
Courtney and Asha an opportunity to respond if there's anything you want to add to this um Especially on that accountability uh and enforcement side of the the discussion happy to move on but want to give you an opportunity to to respond the accountability that's to me that is the Crux of everything um and I think that um when you have and I'll give an example because it's so complex because it depends this answer is complex depending on um how big your company is how big is your just your jurisdiction within that company so I Know
sometimes um the purview of of uh leaders who are in charge of AI products don't have the so to say the jurisdiction within the company to actually take over areas that they need to um in order to have the endtoend responsible AI product at the end of the day sometimes they have to go through legal Hoops with legal they have to go through the risk teams they have to go through decision management teams marketing teams I mean like there's a Whole gamut of people that even when you want to do the right things it's just
virtually impossible sometimes to get a hold of how can I at least you know for my AI product that I'm trying to produce cuz I I'm kind of um sort of uh feeling out um I see a lot of questions from uh Melanie I'm just gonna I'm just going to say your your name because I see a lot of the questions and they're exactly the same things that everybody deals with and I see one from Ry Dash Pandy that uh Is also you know this same type of question and at the end of the day
it's all about like what can you do in your purview and so so this is a really complex about who owns what and who has responsibility here's what I'm going to say everybody has purview over themselves you can do things and you can insist on doing things that are within um I'm going to call it the proverbial five feet of you right so so even if you can't control what the CEO is doing you Can probably control some aspects of what your direct leadership team sees and understands right and you can kind of Point them
towards the right things by educating them in the right ways to do these things and um I did see when I would go out to all the clients there's a lot of efforts where the leadership is just totally disconnected what I mean is like the chief marketing officer who's asking for an AI social media Command Center For example may say yeah you got like six weeks and you've got like um six weeks to develop the product I want to see a proof of concept they're used to using these words right but they may not understand
the details that go behind and the Very intricate processes that go behind that if that person comes to you that leader comes to you or the executive says I've got money I got like $5 million I got $1 million I got like $100,000 I don't know what the amount is But whatever it is they say I got this much money I want to build this and you know right off the top of your head that is never going to work I can never do that ethically and give enough time to look at the data quality
aspects I won't have enough time to to analyze um break it apart in your head put down a list of all the things that you need them to understand immediately when they say something like that don't just feel frustrated and go God they don't Understand anything I'm just going to go do it my way don't do that please don't do that pull in yourself and try to figure out how how you can be brave and communicate back to them no I have to push a little back on you Mr or Miss's CEO CMO whoever you're
dealing with and say look that's not enough budget to get the right kind of good data that we actually need to do this project that's not enough time to do all the investigative research that we need to Do behind the data to get it massage the right way that we need it for this algorithm it's not enough time for us to test it it's not enough time for us to put things in policies and things in place before we release it this is what I need in order to do that if we don't start educating
them we're just going to continue in this cycle of well leadership told me I have this much time I'm frustrated I might look for a different job as opposed to just like Pushing and educating but a lot of what we have to do is is educate up I hate to say we have to manage up right now because there's a lot of leadership who just hear the word Ai and they salivate they get excited and they don't know what goes behind it and as practitioners in the actual field we need to get the Hooda to
kind of push back and say wait a minute and we don't necessarily always have the right people on board right to do those things and to communicate like That and even if you have to go out and find your Advocate within the company that can help you communicate and push on Executives even sometimes look you'll find friends in the weirdest of places I have seen risk risk management people who who get what you're going through and will help Advocate I have seen um people in legal departments who are like yeah yeah yeah tell me whatever
I will promise to bring it up at the next you know board level discussion if you just Tell me an educate me on how to so go find your Champion that will help you get the right budget the right timing cuz you know we know those are the two big things and we know like you guys half of what's going wrong in my opinion this is Courtney's personal statistic but every time I see I saw something going very very wrong in a in an initiative it's usually because you don't have it's data related it's like
I don't have enough data I don't Have enough time to investigate the data um I'm just filling in the blanks in the data sometimes because I'm having to go so quickly and then on the end I'm just cutting this out there and I don't even know how the user's going to I don't have enough you know budget to train the user or do anything like that so I mean part of this is we're going to have to develop some you know guts and get out there and be brave and kind of tell people this is
our process people we Have to have you on board with this before we actually accept producing this thing out into the wild and if you can't be on board with that then you know maybe it's time to have different discussions with different departments within this company about what risks are about to happen and things like that so I mean you know I what I that's not what people want to hear though right I I want to bring it back to something really what uh Raymond said in the chat Raymond said whether you use AI or
not your technology must solve real real problem for the business and I really truly believe that doing good B good AI doing good business will bring you prosp cl ity bring you growth so I think we can't forget that good behaviors will bring customers back to you when you talk about responsible AI ethical AI what we are saying is having these practices will bring trust within your customer having transparency having Accountability will bring good business practices so ran I couldn't agree more because ultimately it's about business growth everybody's looking at at the bottom line are
we creating an impact or not doing responsible AI or ethical AI or doing the right thing when you use AI or technology can help you grow your business and there's no doubt about that so I think even though we look at uh AI ethics and and these practices as a cost Center right now these best practices will bring you Revenue bring you growth and so we can't forget that and I think it is so important that we bring and tie it back to business strategy business growth and business outcomes I think that's a really strong
point I'm aware of the fact that when you're having fun time flies so we're nearing the end of this panel session I think there are so many more things that we could possibly discuss we're really I Think we're only touching the tip of the iceberg there's so much that we could dig deeper um in so I I want to give everyone on the panel starting with llama an opportunity to um talk about prioritization right Courtney you just mentioned um a lot of different things that we like organizations will need to focus on it all starts
with what you just mentioned Asha you know the the business objective the business strategy but there's so Much that needs to be done to embed that accountability from the outset and be able to push it through the entire life cycle right from start all the way to finish and beond and on and on so in terms of prioritizing um that could be at that's you know organizational government governance level but I know that you want to take um Lama you may want to talk about this as governance more broadly what does prioritization look Like moving
into the next year yes um I think uh so I'm gonna just cover maybe more the international governance aspect because I know maybe Courtney and Asha can go into more um industry related priorities um I think one of the things that we need to figure out better in the past in the in the next year is is what is the role of the this different um International efforts that are emerging or like converging now over the past few years we had like explosion of different Initiatives and some of them are like Are Meant To Die
eventually and some of them are meant to stay um and being able to differentiate and focus on different aspects of governance over the p over the next few years would help build this more accountability framework that we're looking looking for and I'm I'm mentioning this specifically because I I work now at the safety Institute and there's an effort of solidifying the Network of um AI safety Institute but also distinguishing this effort from other kind of efforts like um oecd and Global partnership on ai's role um other efforts like um these various um industry um and
standardization efforts that are happening so identifying where the role of each starts and where the role of the other um um sorry where the role of one ends and the the role of others start um and how would that affect um industry in particular um so I'll I'll pass it now to courtne Asha to give their final remarks gu I can I can go next if you want um so I would say the five steps really think about the five things you can you know prioritize and start your journey on the first thing is remember
that you need to embed your responsible AI principle into strategic cultural and operational fra fabric what does that mean it means that when you starting your journey embedding human Centric design into your as you Implement your application so so think about or ask question like how does my system serve the your customer human in the loop are we considering all stakeholders that are going to be impacted you know uh Courtney you gave an example of the credit score are we taking you know the majority and minority uh or the marginal groups you know to make
sure that we are taking the feedback creating the feedback loop uh to adapt the real world use cases and And also the consequences you know again all these practice will bring you the business outcome so aligning corporate values with AI ethics you know asking questions like are the values we are promoting in our organization are they consistent with what we are building as AI systems you know so really truly maybe formalizing AI impact Audits and say going back to what you're building and saying is this application fair and transparent or privacy you know so Really
looking out and saying what am I doing to create this or aligning our corporate values within our AI applications and the third thing I would say is integrating ethics and knowing that this is a dynamic principle it's not something which you can static or you you have to make sure you keep going and revisiting it so you know again asking questions like you know are ethical Frameworks are you know uh giving us that diversity the cultural Expectations that we need from this uh applications or these AI um Solutions we are building and then I think
the uh one of the chats they said accountability and we spoke about accountability yes maybe your organization does need a chief AI ethics officer you know or Center of Excellence who will be responsible and accountable for you know the outcomes so really really asking these questions that who is accountable for these decisions for our AI systems And are we transparent about these decisions and the last one I would say is proactively preparing for these regulation because the regulations are ever evolving and so making sure that we are compliance with the standards and we're benchmarking working
our responsible AI practices so I think if we keep these five things in mind and we align our outcomes to our business outcomes and knowing that doing good will get us good results you know is the Best practice of today's business yeah exactly because the minute that you make those headlines further to asha's point the minute that you make this the newsworthy headlines of hey you just you know what what happened Experian produced 150 million wrong you know and what was the wrong um you know credit scores to banking institutions and it affected the the
people who needed it the most you know and I think wow we have to be Thinking about people and how we are coming off to the world if we can't even secure their data and we have data breach after data breach why on Earth should they ever trust us to report on them about anything or to to fix something some major huge problem for them um as big companies in the world and even as smaller companies trying to tackle big things um so I would say this um I have a list of questions that I
that I like to ask AI creators and maybe Some of those are going to be outside your perview but if you are creating AI right now go look at that list on AI truth.org and just see if you can answer any of them in as yes today if you can't then I would say start making plans and strategies and processes start looking at your existing processes your roles that you have developed for accountability do you even have anybody in the organization that actually is responsible should some should someone Fall asleep at the wheel and be
seen going down the street in their car on the highway actually in what was it Massachusetts um fast asleep is there a way that you at that company could stop your car and have it safely pull over um for example if you're doing automated um AI inside of cars is there is there for example a way that you could stop those credit scores from going out if you know can you detect that do you have a plan for Remediation if something goes wrong with what you're going to produce out there because you know what it's
going to happen and don't wait until you have the oh crap moment for that to happen get ahead of it in advance so that people will trust you will use it and I'll say this for sure if you're in the health care space and you want doctors using um doctors and patients using what you're putting out there especially like if it's a diagnostic Tool which is so helpful things that we all need in this world then you're definitely going to have to be like a CSI agent these days you're going to have to get out
there and understand the context going to have to treat it like it's crime scene investigation you know get out there understand the context in which this um algorithm will be deployed and how people are going to use it so that you can fully understand how to work within an environment with people Um so that you include all the things and all the types of people that might be affected as well to to asha's um points about diversity understand what you're doing and how you're doing it and then Benchmark your own processes your own roles your
own tools internally and see if you've got the things in place to handle that and I mean everything people I mean look at your hiring contracts do you have it in there that the data scientists have to even document what They're doing do you know how many times I've run into the Simplicity of just data scientists don't come in they say that's not my job and they won't document what they're doing like sometimes it's as simple as like make sure in they're hiring when you hire them they have to actually document everything they're doing and
putting it down so that there can be transparency and explainability and governance don't forget the last mile is training people And getting the corporate policies in place and communicating I know as data scientists and as product people sometimes we forget to actually communicate with people about what we're doing and why we're doing it and that kind of thing but we have to absolutely challenge ourselves to do that so go answer those questions and if you can't put some processes and and roles and tools in place to do that thank you so much Gan Thanks to
all three of you for your incredible insights um lesson learned reflection for 2025 we need a longer raise panel there's so much more that we need to discuss so on that note I thank you once again please feel free to reach out to all three panelists and to us as well if you have any questions I'm sure they're plenty we haven't answered but on that note I want to hand it over to Patrick again for the final part of today's Ray Event thank you hadassa all right so uh for those of you who do not
know me I am Patrick mcandrew I'm the member engagement manager with thei Institute working alongside all of our members to Spotlight their leadership in responsible AI as well as spotlighting uh other leaders in respon responsble AI across individuals organizations and initiatives which brings us to our leadership in responsible AI Award winners were very excited we had many Many votes and nominations come in over the course of the last several weeks uh to hear your thoughts on who you believe is uh should win our leadership in responsible AI Awards we had three different uh categories including
outstanding initiative outstanding or organization and outstanding individual so with our remaining minutes I'm just going to go through those real quick for outstanding initiative our top nominees included ts's AI report the power of Perspectives RBC Borealis respect ai kpmg's ai impact initiative One Trust co-pilot the standards Council of Canada ISO 4201 pilot with ATB Financial Verizon responsible AI initiative and the gender biases in AI course and the winner for the outstanding initiative is one trust co-pilot uh the on trust co-pilot powered by the data guidance platform and enhanced by AI driven advancements helps privacy and
AI professionals Around the world accelerate their responsible AI commitments it's built upon the trusted research of over 1,700 regulatory experts across 300 plus jurisdictions and it delivers realtime insights personalized guidance and advanced tools to help organizations reduce compliance cost and minimize risks One Trust congratulations for our outstanding organization we have our nominees including the institute for experiential AI responsible AI practice at nor Eastern University Tellis KPMG shell Brown Foreman ATB Financial and Boston Consulting Group and the winner this year is brown foran brown foran is considered a beacon for responsible AI development in this distilled
Spirits industry they are actively partnering with the national Alcohol Beverage Control Association in providing thought leadership and research in AI Brown Foreman congratulations on winning Outstanding organization and finally our outstanding individual category these are uh the list of nominees including Michael Brent the director of responsible AI at Boston Consulting Group yuken Jung director of AI governance and responsible AI at ATB Financial Amy chalen the global head of AI at Shell Brian macallen the principal Global trusted AI team at KPMG Lana Lo Piccolo responsible AI governance and law expert JN Diamond director of AI Governance and
data ethics at telis and jansu janja director of responsible AI practice at The Institute for experiential AI at nor Eastern University uh quite a great list of nominees here and the winner for this year is Brian maowen of KPMG Brian is a principal in the KPMG advisory practice and leader of us trusted AI for Consulting in this role Brian continues to expand his passion of leveraging technology to drive Efficiency enhance insights and improve results Brian congratulations on your win for this last moment I'm just going to hand things off to Jeff uh to close us
out for today so what an amazing session thank you so much to to llama Christine and Asha for joining us today in your amazing insights I love that we have a new very simple definition of AI that's using a responsible AI that is using AI without having an O crap moment I think I think that's the best Definition I've heard and I've been in this space for a while so thank you so much for the engagement in in in Courtney we we'll give you full reference uh anytime we use the phrase but uh super grateful
for all that that you guys brought to the table I I would say I'm also very grateful for all the amazing insights and comments and points that that the attendees made in in the chat thread we're definitely going to take a look at that after the after this Session and and learn from it and even reach out to folks to engage so I'll close this out thank you so much uh again to everyone who involved um I hope that you all have a fantastic uh you know end of year fantastic holiday and yes I agree
with hadasa we need to find more time to have more conversations because there's just so much passion and energy here that we'd love to tap into so thank you all and hope you have a good rest of your year