Webinar: Impact of Ai on Assessment in Education

Welcome everybody to our webinar sorry but it's just a couple of minutes uh late starting but we normally allow a couple of minutes anyway for some late joiners but great to see uh the waiting room filling up with loads of loads of people joining probably from all around the world um it's really great to have you here I know how busy it is for everyone in Academia at the moment so yeah we really appreciate you giving up Your time and uh we're really pleased of course that this is an interesting topic to you all well

of course it's interest uh interesting topic Ai and education and assessment all over the world at the moment so really great please feel free um to yeah say hi in the chat uh hi we told and uh yeah like let us know where you're calling in from it's always nice to see uh people joining from from all over the world which is really great so yeah well just to confirm this Webinar is all about the impact of AI uh in the context of assessment and education and we'll be talking about uh simulations and uh well

AI resistance including detection software and and and and things like that but also more generally you know AI in higher education obviously chat GT gbt and uh all other kind of yeah generative AI tools that have been filling the news over the past 12 months and and before as well everyone's trying to kind of Make sense of things we'll be talking a little bit about impact on teaching methods uh assessment challenges uh and uh Solutions as well so really really uh excited to get into the conversation and uh supported by uh my colleague Peter who'll

be managing uh the chat and answering questions but also leading on the Q&A session a bit later feel free to say hi Pete good uh good afternoon everybody thanks for taking the time out For what's uh yeah lined up to be a really exciting engaging conversation thanks again yeah no worries and um well perhaps before I just do a little bit of an introduction into uh simulations as well before we get into the conversation just to introduce our two guest speakers which were really privileged and uh pleased and and yeah well it's very nice to

have you both here um Emma Connor and Gard as well um Emma well both from from Grenwich business school at University of grenwich and uh well Emma I know you've been uh receiving all sorts of nominations and receiving all sorts of awards at Greenwich for Innovative teaching uh practices but also I think rising star of last year for example if I'm not mistaken um please feel free to introduce yourself as well Emma hello everybody uh can you hear me well is my first question always in a Webinar right um yes good so I'm em Conor

I'm a teaching fellow in digital marketing in the School of Management and marketing at grinage business school but I'm also the incoming program leader for the new msse digital marketting marketing management which is very exciting I've T across different subjects uh but I've returned to my love of marketing now um I'm also a PhD student in consumer behavior and I've taught across different I think three Different simulations so it's a pleasure to see you today and I look forward to talking about my experiences with you well really glad to have you Emma and um yeah

thanks for the introduction and geard over to you as well to introduce yourself and well actually just to say that um we we've we've met both of you some years ago as you were both embarking on a a journey to evaluate our simulation software and then it was subsequently deployed and and embedded At Grenich and I think uh since then gard it's been great to see kind of like how much you've kind of positioned yourself as a bit of an authority on the subject of AI and I know you have like all these uh videos

on your YouTube channel for example on AI so yeah it sort of also made real sense to have you here as well but yeah please introduce yourself as well yeah so hello everyone thank you for having me I'm Gad Christ and I am asso professor in technology Enhanced learning and accounting as well and uh yeah what you just gave as a long description is basically just show for tech nerd because that's what I am and uh uh I'm really invested in in this new technology I find it very exciting interesting and hopefully this will come

across in connection with uh well simulations and assessment today so we were ahead of our time actually getting it in to grenwich a couple of years back yeah well absolutely yeah and uh it Was really great that now like you're both able to kind of also yeah Be Our Guests today know talking not only about the impact of AI more generally uh but also then specifically about bringing that back to simulations as well um and since then we're having countless uh conversations also with other colleagues at Greenwich Business School uh about similar approaches as well

which is which is really nice too um okay well before we get into the kind of topics um Just for a bit of context for everyone uh we're just going to summarize a little bit about yeah what we mean by business simulations and how those are typically used so I just take a few slides to do that uh so for example well edim Mundo as you probably know we've been delivering Management simulations in higher education for over 20 years and they're all web-based platforms um so we have we have platforms which can cover most business

school subjects but They're all web based just no software to install they just work on the browser and we're talking about something where you know students manage a company in teams of maybe four or five it's flexible and they'll then manage that company over a number of virtual years or rounds where one round equates to one year of operations um highly competitive of course it's gamification and uh it's a dynamic algorithm so student teams competing against each other uh to win Ultimately although it doesn't matter who wins or loses in terms of the learning you

know you can often learn uh from failure as well of course we can for all student levels so right from Foundation right through to uh yeah MSC level and MBA level and a number of our simulations are very flexible as well uh so there's certain tweaks that we can make but we can cover most most subjects within the business school typically what we find is that these simulations Are often used strategically to improve and enhance uh student engagement uh and study success of course so typically we see much better moduled feedback scores and linking to

satisfaction uh as a whole across the business school and across the university more generally uh but also yeah linking to cont continuation and progression uh of course it's about boosting the skills the employability skills which come through regardless of the subject and Come through really strongly so clear links to graduate outcomes there as well um and often we find that you know the business schools that adopt our simulations will often start to use uh that as part of their marketing proposition to to recruitment as well uh to entice students it's what students want these days

of course uh and just to say like very quickly so here's a range of a snapshot of the different simulations that we have uh actually It's it's it's there's many more than just just these but these are kind of like the core ones we have and there's some examples on the right there of the sorts of subjects that typically these simulations are used in so I think I'll stop sharing at that point as I say just just thought it was helpful to give a a brief introduction and a bit of Contex text into what we're

talking about today um but perhaps it's also useful to now yeah I'll start the Conversation off a little bit I know uh that yeah through conversations we've had already with our guest speakers with gard and Emma you know it's important that like yeah we acknowledge at least that like everyone I'm sure everyone here is really busy and uh you know dealing with everything that's involved in uh the world of Academia and to be expect expected to also be an expert on artificial intelligent and generative AI is uh well we just don't have the time For

it isn't it but yet somehow like everyone is being asked to somehow consider this and the impact of this on their teaching on assessment and everything else so hopefully today will be really valuable for everyone um but we're also you know Keen to keep the conversation going afterwards as well um so please uh stay in touch afterwards um for now I think uh was there a couple of things you you wanted to mention as well about I think the disc report and some Of the latest news uh yes thanks Le just just um to put

it into context we're going to introduce a couple of polls through through the course of the uh the webinar looking at the J report which is um well there's been a number of them actually but the one that we've been looking at is is looking around the types of concerns that are being voiced uh by by students um and they've also quite helpfully made some suggestions and had Comments on policies guidelines initiatives that institutions could be taking up to to help address some of those so we I'll drop them in at you know the appropriate

moment we'll just like to get an insight into uh whether these concerns are mirrored at your institutions and then just you know very delicate question about types of policies and guidelines that you might be putting in place to support your students that's It yeah great okay well um I think uh perhaps let's just uh yeah over to um to our guest speakers then um perhaps some initial thoughts on on on just yeah like how how big this subject has become all of a sudden in in over the last 12 months yeah um I could start

by sharing my screen if that's all right yeah feel free so I hope you can see it all yeah yeah so again welcome to this webinar we um m& been working with Adom Mundo for a while mostly Emma I was just the enabler to get it in U but Emma was the one really using it in teaching and to put it in in context a little bit uh I'd like to set a kind of a a common understanding in the beginning because we were we talking about academic integrity and how AI gen AI so when

I when I say AI I mean generative AI because there's other type of AI of course uh the record are a little bit straight because we we we should Consider in general our students as people that don't want to cheat yeah that want to know um how to use AI in an ethical Manner and that we guide them that's that means we also need to understand how e use AI ethically because that was the first reaction in early 2023 we have a oneye anniversary if you think about it a year ago jet GPT burst onto

the stage right so almost to the day I don't know the exact day but it was November 2022 and it has been quite a wild right since then and the first reaction by educational institutions at least those publicly voiced was all right this this new thing that's GNA this is able to write similar to a human being they're going to use it for cheating yeah so called cheat GPT I think in general we should say that our students do not want to cheat outright okay youve always have those that that that of course have this

intent but in General our students have quite a nuanced approach as we know by now yeah so Peter mentioned the JIS report I'm assuming you're talking about the same J report uh and I'm saying this is where they basically surveyed students so what do you think about this a gen thing and they show a very nuanced understanding at this point by and large right so how they using it they're using it for writing for understanding concepts you know explain to me like I'm five years Old it's a classic uh for Maths for coding for facilitating

research uh for generating images for fun and you know some also for some of my students even for their personal lives get them more organized so they advocate in that report for an integration of AI into the education to prepare themselves for the future because it'll be with us it'll stick with us it'll be part of society at one point we're going to be quite used to it Like you're going to be used to having an Alexa at home and talking to Alexa as if it was a human being and at the same time they

do voice these concerns about you know digital literacy uh data security transparency of how we are using it as Educators how institutions are using generative AI uh in marking the assessments maybe in creating assessment questions and so on and so forth and I think with all of that the uh academic Integrity question Mov moves to a slightly different stance not to say that everybody's cheating and do we need to prevent it at any cost but it's it's it makes us question um our own approaches more uh asking okay is my is my teaching still up

to speed am I doing the right thing and of course with the plethora of information coming our way about AI certain AI fatigue has settled in in some Educators uh there's so much out there that it's it's almost impossible to keep up Uh so I want to say that academic Integrity is a crucial topic yeah and it remains a crucial topic and it has given another boost to the so-called authentic assessment type you know these types of assessment that doesn't test just the knowledge but how students apply knowledge in a real world context and simulations

are a type of authentic assessment so to um really understand there's some scribble there I don't know where that Came from um so to really understand uh why authentic Assessments in particular simulations are AI resistant or more AI resistant because I'm a bit Lo about talking about AI risk so AI resistance against misuse of AI in use for passing an assessment Without Really performing um let's put ourselves also here on the same level uh traditional assessment model so I'm talking about the you know the typical exam uh even pen and paper you know fall Back

to the olden days um pen and paper exams that are still prevalent in in institutions they they they prioritize knowledge testing over competence testing and application testing and performance evaluation they they also struggle to test the ability of students to apply knowledge in a real world context to new and unfamiliar situations which is real life right and there is an incentive incentivization don't know if that's the Right word uh but it is incentive for students in these types of assessment to focus on passing the assessment that's their goal but not the learning per se that

kind of gets sidelined it becomes even to gets to the point that if if it's not assessed students won't do the work don't know if uh Educators in in chat um have that situation or I had this as a student myself that an educator has limited the topics that they are assessing us on to like out of 20 give me a list of 10 topics we assess to you on so what's my incentive to learn about the other topics and how it all fits together right so the problem of this traditional assessments clearly is they

don't go be beyond the know what the knowledge testing they don't test the know how and the show how absolutely Gad sorry to interrupt I was just kind of thinking it's it's that age-old uh thing of of students just focusing yeah on what they they have to do isn't it to Get the credit and and not doing things that are that are assessed ultimately so so Savvy students essentially so ultimately this is where the simulations come in right as as because the question about authentic assessments that's not new that's that has been giving a boost

over the pandemic with online teaching and so on and now we're more so with AI because they are more resistant and why are they because they are immersive you know it's Not just there's an output that I need to generate uh they need to immerse themselves in a decision-driven learning environment it engages them they get much more agency in what they do the process is as important as the output because the output might differ between students student teams if they use simulations such as what aded Mundo provides um and there's much more emphasis on the

critical thinking part yeah is when decisions are being made Within a simulation something unfamiliar comes around it it's does much more emphasis on the evidence-based analysis and the reasoning and if we're talking about AI we mean to talk about hi as well right human intelligence that needs exactly these skills honed and simulations can provide an incredibly important uh factor in in developing that in our students U developing competencies uh not just finding the Easiest solution but the solution that work best that is the most efficient and effective and there might be several Solutions coming out

of a simulation case study that that fulfill that Criterion um H when they work in groups that's much more group dynamic it's much more resistance to an direct AI input in terms of we can use thei to do the work for us entirely and then submit our our our results and pass right which is um it's it's not as easy to do in a Simulation and then ultimately this the you mention the competitive aspect I think that's really give them even more immersion and agency in what they do and what they produce and gives them

a sense of Pride as well uh in in in what they uh provide at the end and uh it's real time you mentioned uh in in an earlier chat Leon it's playing the game right and the question here is why do simulations Al from a technological basis make it so difficult For AI to be misused and even Embrace AI as a facilitating technology instead um the typical features you know edmundo simulations are endtoend encrypted they're closed systems it's um unless they're really Nifty and technologically very Advanced then I would ask myself if they could do

that they could figure that out they should work in Tech development and earn money straight away um endtoend encryptions closed systems preventing any direct Intervention unless there a a rail guarded or guard rail system um an AI that the provider admits actively realtime interaction makes it really difficult forp yeah I was just gonna add G like to expand I suppose I think I think what you're saying is is essentially like by end to ending ion so so generative AI won't be able to yeah access what's behind closed doors essentially isn't it that's right it's it's

it's it's Difficult to interfere with these closed systems so there's a couple of closed systems now coming to the rise with the expressed purpose uh of of you know preventing any direct AI intervention um and I think that's that's massive Advantage from the technological side uh and gives peace of mind to to the educator uh to do that and you the use of unique data sets that are changed according to the decisions that the students team are making uh it it it it Makes it really difficult for an AI to to mimic human behavior in

that way and in extension for students who have ill intent to abuse and misuse AI in that context so um I I think sorry yeah just to jump in as well a little bit I think um the unique data sets is really important isn't it because actually if you've got a whole load of different students that are working on the same simulation but actually the data that they're looking At is different to what the other students are looking at then when it comes to for example if we're to come back to your point about uh

yeah student coming at this from the perspective of maybe students want to cheat maybe they want to ask chat GPT to do their assignment for them well at actually they're going to have to get so involved in the simulation anyway to know what to ask chat GPT that like that that's probably a skill in itself right to be Able to uh come up with the correct prompt really really tell the generative AI all of the things it would need to know um yeah like so I think that that's really something to stress I would say

yeah it's it's a it's ultimately it's a cost it's a cost effort benefits balance right so if they have to put a lot of effort into cheats and then realize it's actually no point because if I do my work as intended uh within the simulation alongside the process of Learning and not just you know to get to an output I can submit it it it's a realization that there's actually no point and putting the effort in to cheat uh is less getting me to my to my hoped for result than actually immersing my myself in

the case and really be be in the learning and not just ogal the the output at the end of the of the of the simulation of the speak yeah yeah exactly I even go as far as to to add that like um you know even If you don't have what we might regard as an authentic assessment even if it's quite a traditional uh assessment perhaps in the form of kind of like a a reflective piece that you know even if that's the case it's still going to be very difficult for uh yeah for for you

know in in in correlation with what's happened in the simulation it's still going to be very difficult if you match the two up sort of you know let's check What you've done in the simulation what your strategy was everything else then marrying that up with your assessment even yeah as I say if it's not authentic uh then it's yeah it's still kind of going to be hard isn't it for students I would say yeah know it requires much more reflection and critical thinking that's something that a I can can try to mimic but it can't

do it yeah and that that that will be the big difference between how to approach This uh so in a in a conversation before the webinar uh Leon wanted to know how's say eye detection looking at the moment and is it is it Def and S sorry to interrupt G just just before you you lead on this slide I just had a I can see a question from witold in the um the chat about the images images generated by Dar these ones yeah di 3 yeah yeah there you go yeah well spotted with Old sorry

Gad sorry to interrupt no problem uh yeah we know in the context of AI why not generate the images with AI uh so yes so we had a conversation right is AI detection software effective because you see a lot of claims on LinkedIn by various vendors of well well-known vendors I won't name them um that oh they found the an AI detection software way and tool that can detect AI reliably uh unfortunately I need to burst the bubble because the short Answer there is no such tool out there that is reliable enough that we can

with peace of mind again use in order to detect AI generated content the longer answer is that there's there's independent tests again if you do your Google research there you find plenty of those um that uh that none of those tools at the moment can provide and that's the premium tools provide something better than 70% accuracy and that is by a long shot not enough Because we would be opening as institutions a can of worms uh if we starting to accuse students that are um that with with without enough evidence without enough um uh meat

to the Bone when we accuse them and uh it's it's not trustworthy enough and some vendors might claim that their TOS are reliable but they are not unfortunately that is because how AI generates text because how is similarity picked up similarity is picked up by Previous text and comparing to previous text but what does AI generate new text yeah based on of course a lot of training data so and there's a lot of ways to circumvent this very easy uh throw in a couple of typos change a couple of put a couple of grammatical errors

in run it through a series of chatbots it's not just ch GPT out there and it it's impossible to detect reliably um so and that's also the Problem of false accusations and in particular was highlighted that uh non-native English writers are very often picked up uh incorrectly as that they provided AI generated text so I'm a non-native English speaker and writer and how do we learn English in school we learn standard phrases a limited vocabulary and then we use that and that's what these detectors tend to pick up on yeah the limited range of vocabulary

and and expressions and that Makes this very problematic so yeah I think yeah I yeah I I also heard that uh like a lot of yeah non-native uh English speakers if they're required to do their assessment in English they will maybe start in their in their home language first and then put it into a translation software um which you could argue well that's not necessarily plagiarism is it but uh then because of the way that that's been translated the detection Software will will pick that up as you've cheated exactly that's I me it's it's it's

it's just not accurate enough it causes way too many false errors and accusations and then that this is a problem on an Institutional level uh goes without saying and at the same time it deters students yeah if they're afraid of using the the the software maybe just for you know explain a concept to then they use similar wording Um to use that or or you know it it it holds them away from something that they should be exposed to because it will be part of their personal future and professional careers so we would do them

a disservice and also with a critical discourse of the students with the Gen that we should be facilitating in a way to to critically evaluate is thei output useful in what way or how is it not useful and the problem is that adds a burden to the Educators now because if There is no tool that Educators can rely on that detects it reliably then they need to learn um and build their AI critical literacy and the problem is we all very busy Educators not everybody has the time to do that um to learn about the

potential of of gen the limitations that it has the pedagogies that need to be involved maybe even new pedagog is entirely and the ethical considerations and how to build that in and assess their own Assessments are they AI resistant yes or no that's important these pre- skills and also what follows after what do I do if I find my assessment is not as a IR resistant as I hoped for and simulations again come in here because they take a lot of that off because simulations per se as authentic assessment and authentic learning Tool uh facilitate

this much better because of the reasons that we just discussed earlier I think gard it's probably an Appropriate point to um also yeah like talk a little bit about the University's response to a I as well like we're talking about detection software and things and I I know when we've spoken in the past you've raised this point that like when we go back 12 months uh when this kind of really came to the scene in a big way that a lot of universities were really reacting in very different ways and and some completely outright Banning

uh Ai and I Think yeah well since then as you pointed out like a lot of oh you're already coming on to this but okay but before you do perhap has the hand raised yeah yeah beautifully managed there Le just before we move on to that could um I just want to launch the first poll which is just a real snapshot um and it's just um looking to get a a little insight into the the student concerns and perspectives all of the attendees Relevant institution so it's a multi-choice question just tick all any that apply

to you that should now be launched you should all see that um and just a little bit of housekeeping we've had some conversation about the definition of digital literacy I think that's been understood but there was a couple of points about this idea around cheating and and a couple of questions came up about whether there is any thoughts or research On propensity to use tools like this shall we say to teach in terms of different profiles study levels regions of the world I I don't know if there's any thoughts or any sort of initial feedback

or comment on that particular Point yeah something that that universities and educators are finding their way into because uh it's it's not entirely clear often it depends on what a university sees as acceptable yeah and how to define it it was one of the Reactions see it here uh on my screen uh policies yeah that they develop typically based on already existing policies makes sense you know as a short ter term shortterm measure to deal with that and then what is acceptable use and if you look around again a Google search um about academic Integrity

policies in UK universities for example if you focus on the UK or any other part of the World shows you there's a lot already out There but if you compare them then it really depends on what is too much from the perspective of the institution and what is still acceptable and that might differ so it will depend ultimately on what your institution regulates in absence of a broader regulation yeah EU EU has is is is looking at the AI Act the US has just initiated legal regulation considerations about you know watermarking AI output and so

on to really to really Inform institutions uh like universities what is acceptable use there was a big question across the entire year what is it what is acceptable what is ethical and that was not 100% clear should I wait the end okay well should I continue I think uh yeah let's let's close the poll now I yeah I'll end the poll uh I think it was supposed to be a multi-choice uh poll but uh as a couple of people noticed it was only possible To select one um but I think perhaps we'll have a look

at those results and yeah maybe see if we can kind of build that into some of the discussion at the end as well if there's time yeah please continue Gad yeah so as as you as you nicely transitioned my segue over to the next slide so to speak how did universities react if you think back to especially early 2023 there was a lot of banhammer going on yeah Banning restricting we're Not allowing our students to use it uh very prominent examples were Oxford and Cambridge for example yeah I said we we restrict it we don't

allow it and there was the other side of of the coin uh universities Greenwich was one of those uh University one of those universities said we're not going to restrict it because you know if we say no and ban it students will find ways of using it and if then then we have not no chance of telling the students how to use it in an Acceptable manner so we were more like on the embracing side of things and started working very early on on policies and adding to our academic Integrity policies uh how what is

acceptable use informed by the office for students Department of Education outputs at the time um and a lot very lengthy discussions were this acceptable use which goes to show that it was not an easy question to answer uh now these institutions that have since that have Banned very publicly banned have uturn yeah because they realized it's unbannable it's it's going to be part of our daily per personal and professional lives I'm sure to a point that it's normal for us we don't even realize anymore that with that there's CH involved at one point that's that

point is not now but it will be I'm pretty sure like talking to Alexa is normal at home um also other reactions early initiatives in staff Training to tell staff all right this is what it what that geni thing is and this is how you can use it and how can you prompt so I've run a series of prompt engineering webinars for example uh as a door opener to the understanding what can gen do and what can it not do where are the limitations of it what does the output look like um and that goes

further users of AI in the classroom with the students for the students teaching about AI teaching with AI so a Myriad of training rolled out by UK institutions and uh worldwide um to to educate the staff to tell them what is that geni thing and these were understandably short-term measures to be ready for 23 24 but we only beginning not just as institutions and higher education but also as a society what may be the long-term implications job displacements new new jobs are being created that we don't even know about yet longevity of that technology and

What would it would it impact and we only beginning to understand that and um I'm sure that universities will react to that and hopefully even guide the discussion and direct it more instead of being reactive to it I'm muted uh yeah gard and uh well perhaps Emma as well feel free to CH in I'm interested to know whether you think your opinion of course but uh whether you think it's likely that any University at all now will not have yet devised some sort of uh at least an interim policy around AI to some degree opinion

wise do you want I don't think so I don't think there'll be any institution whether it's public private big or small I think every University will have to have some kind of policy regarding this ge hards what's your thoughts yeah exactly I mean again if you do Google search just just pop in the search term here on on the screen AC Academic Integrity policies in UK universities and Ai and you'll find a lot of links and a lot of long lists yeah and if if you put the UK away and look globally or in your

country of origin you'll see there is something so clearly they need to deal with it and they are dealing with it and these policies will develop because this is the first Academic Year full Academic Year with Gen fully in the picture and it's moving very fast the policies will Need to adapt as we go along um to maybe entirely new ways of AI uh use U new ideas um new research findings um so expect these policies to change because they will have to they will need to to adapt to what's going on yeah absolutely yeah

and and and Emma do do you think um is it the case I think you you were saying as well that like at grenwich as well every module leader has been asked to kind of score their their basically to Review their assessment approach for every module of course and must kind of score it with a uh with a with a waiting essentially to say you know kind of how how a AI proof essentially is is my assessment structure something like that yes Leon that's correct so every program that we have uh you may in your

institution call those courses we have modules or you might call those units and within those across each levels whether it's postgrad or undergrad we Got undergrad level uh four five and six and then postgrad being level seven um in any of those all of the modules should complement one another and should be an overall learning outcome for the year and within those module leaders work very hard to make sure that there is a limited possibility for misuse so of course we cannot entirely prevent this kind of ill intent or malpractice but yes every module leader

has uh been tasked to do that that's Correct Yeah well yeah as as you say as well it'll be interesting to kind of see how this evolves over the coming uh months and the next years as well how how things change how things shift um yeah so okay well um shall we talk a little bit about uh teaching methods as well and um and then I'm also here Keen of course to hear um from you on uh yeah around like the coming back to this authenticity and assessment as well um But G was there anything

else you wanted to say on on the slides or is that yeah where are you at the SL yeah so so the in terms of so don't worry I'm not going to go through that entire thing this is um because this is recorded and uh slides will be sent to participants PDF afterwards uh have a read through that because that this is just a a a small selection of possible potential impacts uh but ultimately it will impact what we do and that can start from that you know One once we have ai resistance achieved to

a certain degree that it can actually enhance the learning uh of our students be a facilitator being you know study body is is for instance a nice application of of jet GPT um learning about you know challenging as we just done so far challenging traditional assessment are they up for to the task not only assessment for learning but assessment of learning both uh perspectives uh and and a view on Authentic assessment and a view on that you know if you look at your your assessment structure in your individual modules courses programs are they doing what

they're supposed to do and that is a discussion that maybe AI has pushed even further up the agenda but that discussion has been around for quite some time you know our assessment types models are they up to task and this is is definitely something that will impact also the teaching methods and not Teaching to the exam but teaching for the learning and the retention and the competency and the skills ultimately leads up us back to um considering simulations in that context and a very strong focus on ethical and responsible use not just that we are

as Educators or institutions as a whole use it responsibly and ethically but also teach and educate our students how do you use this ethically and responsibly because they take that to with them into the Into their professional and personal lives later on um skill development will be very strong focus in the teaching methods so again leads us back to simulations um to Foster AI literacy skills in everyone not just the students uh supporting inclusive uh education uh making Concepts much more accessible by having Concepts explained differently and even triggering ideas how to explain how to

teach in an educator so as an ideation tool gener generative AI is Wonderful I'm using it for ideation yeah if I'm stumped with a complex concept that's coming up in my next session I need to explain maybe I I'm asking Chad GPT can you give me five different analogies for that particular concept that I describe as the subject matter expert comes up with a couple of ideas and said that that's actually really good I'm going to adapt that I'm going to use this yeah so it it this will impact us and uh it will impact

us on so Many different levels that I can only say this list here can only be non-exhaustive yeah there will be more impacts than I can put on one single slide there will be impacts that I don't even can I can't even speculate at this point yet but it will impact great thanks Gard um I'm Keen I'm watching the time of course the clock and I'm Keen to hear from Emma on uh authentic assessment as well um yeah Like it's a term that's being used a lot more and more these days as well um would

it be okay uh Gad if we pass things over to Emma a little bit to uh talk a little bit about I keep sharing my screen but these are her her slides okay cool that's correct hello everybody it's lovely to see you today um I'm going to talk about my direct experiences with using simulations but particularly for this uh webinar I'm going to be talking About the trainers startup game which I used in one of my level four modules last year now what's particularly unique about this experience is that all of the materials were designed

ahead of much of the kind of implementation and use of AI tools such as chat gbt so when I originally designed the module and I had in mind uh I wanted to do something immersive something interactive with my students immediately a simulation came to mind now I've been very lucky that I've used both Tech and non-tech uh simulations in my career so I've used uh drama role plays as well as uh technological simulations but with this particular one I wanted to really move away from company case studies now now there are a number of different

reasons for that number one is obviously the potential for students to purchase assignments based on case studies now we've all had experiences I'm sure where we have had to report plagiarism we've Had issues of uh company case studies becoming outdated for example so what I wanted to do with this is to create a scenario based environment so right in the designing stages of the module and this was a marketing principles and Sales Management uh module I wanted to allow students to have that immersive experience but also be able to prove what they've learned in the

immersive experience now bringing it back to the theme of the webinar it's all about the Impact of AI how can we make sure that students are actually learning and can prove that they can now in the subsequent slides I'll show you a small snippet of an assessment rubric that I use to assess my students in their written work so to demystify that I'm sure many of you are aware of rubrics but really it's just a set of criteria with different percentage weightings that we're able to use to assess students in written or in presentation Work

now what I found with the simulation that I used for this is that I set up um two block teaching sessions where I had my students come in for six hours and play the game and I also set them uh homeworking tasks that they could do within their teams I made it flexible in that if team members needed to change that that was also okay because over a period of time as such sometimes we accept that there are mitigating and exter um Exterminating That's not the word extenuating circumstances where people can't attend so I think

for my personal recommendation allowing room for flexibility for team members is a very uh important factor there but I think it adds that kind of real time uh element to it as well because adding a little bit of time pressure um I think as Leon mentioned and geard mentioned earlier is extremely important when it comes to um designing any kind of assessment Particular if it is authentic and emulates the real world so what I found was with by having rounds allowed students to become very visibly uh competitive towards one another There's A really lovely element

in the trainers simulation that allows you to create news pieces or PR as part of the promotional mix um among the uh seven PS and it was very delightful to see students get very passionate about competing with the other teams and Because that was in real time it really added an element of fun and interactivity that I hadn't quite anticipated in there which was very enjoyable but bringing it back to algorithms and data what's interesting about um using one of these simulations is that the company data which students set targets upon to be able to

see if they're going to be a cost leader whether they're going to lead in quality for example example in different Customer segments because they're frequently changing it doesn't allow AI tools um in this end to endend kind of viability to be able to go in and predict what might happen in the simulation so I think we're focusing very much on the game play but also oned the assessment so next slide please geart here's a quick case study of what I used and how you can immediately implement this into your own assessments well I found that

as I me mentioned at The beginning um of my slides that I wanted to from the very offset embed the simulation into the entirety of the module so this is a 30 credit module it's quite a large uh module that runs um very intensively with lots of topics but I was able to really tailor the topics that were suitable for what I wanted students to be assessed on so instead of me coming up with a list of I think you mentioned earlier if you've got 20 topics and there are 10 that are In an assessment

those are 10 they're going to get lost and they're not really going to be retained by students so what I found is that I matched the game rounds with all of the topics that I had in my module and then match that with the different elements of the assignment brief that I asked students to do so one might be asking oneself well how do we reward students for gameplay how do we really know what they did do I have to watch them for the entire time no Absolutely not the way that we do that is

through the use of reflection so I found that in my modu I added a 5% uh assessment Criterion on exactly how students comment on that now I've been involved in a number of Assessments and sometimes we typically have a company case study and then we have group work and then we maybe have a section on um a reflective cycle or a group development module I don't know tuckman's or Gibbs or something like That and we allow them to talk about their experiences future action plan whereas what I did this time is that I actually very

clearly guided students that in every answer to their question in the application for example of the 7ps or 5 c's for example what they needed to do was to use evidence to support their assertions and their claims not just from academic texts and from industry-led uh sources but also to use and reflect on how this was actually Applied in the game which students really enjoyed and it really paid off for one very important reason what I found is there was no detection at all of any kind of um plagiarism for example the sources that students

use were very high quality because they needed to support the gameplay that they did um and that was really good for me as an educator not only because I was very pleased that my students had not committed any plagiarism um and we have Very robust tools to be able to check in terms of um academic and Industry sources if that's the case but what I also found on there is that it was a sense of Rel belief that it made my job marking a lot easier because it would have been very easy to see superficial

um Reflections and it's very small text on here so it's hard for you to see but you can see in the failure category that's in the kind of third box there little or weak reference to engage in With the simulation game yes of course some students are going to very weakly comment on that or they might simply put some uh comment on there that I participated in the round but don't mention anything whatsoever so this is a way that you can kind of mitigate that against really strong cases that not only connect to Theory not

only connect to game rounds but also support the discussion in an excellent manner Gard next slide Please so finally supporting student outcomes now something that I'm very passionate as well as learning Technologies I'm also very passionate about human uh interaction with our students as well and what I like to kind of draw upon something called the pedagogy of care so students very typically come to uh institutions because they want to learn because they want peer interaction they want to learn something and use Technologies but we Also need a human element to that as well so

what I find is and according to many studies that I've read in the literature and my own experiences is that that motivation of the tutor guiding them through the rounds and a mixture and kind of combination of those factors of peer interaction actually create a really nice relaxed class environment yes there are periods where students highly competitive but I also find that this Kind of relaxed atmosphere but also highly supportive and human atmosphere mixed with the technology was very um successful in my module I think there are a lot of misconceptions um from students about

how they will enjoy using the simulations and exactly how they're going to be able to use that in their assignments with any new technology or any new software students can find it really difficult um and can you know at Times see that as a burden to their learning you know oh we're gonna have to learn how to use this simulation now I need to get used to the navigation um I've used a variety of complexity of different um simulations some where you have to spend two weeks on simply learning the navigation of sap for example

but when it comes to um web-based simulations such as edim Mundo and others I think it's it's really good because it gives students that kind of Takeaway that they can use it anywhere um because it's web based use it at home they can learn how to actually use it very quickly um and they can have a lot of motivation from their tutor in which to use it which I found really helped and really increased my engagement so I hope that was helpful to you I think we now have some time for questions but thank you

very much for listening to my Reflections really helpful em thank you So much and well both of you as well I think uh it's really important to remember that we're all yeah we're all finding our feet isn't it with this subject and uh it's an evolving landscape and let's all support each other with this as we move forward in uh well it's an interesting landscape as well of course um yeah I I think it's been really interesting having this conversation it feels like a Conversation we could probably allocate three hours to uh rather than one

hour uh We've really had to try and condense I think a lot of the conversation here and I know in some of our prep Preparatory conversations yeah there was uh many more things we could have covered today as well but really really appreciate you putting the effort into to join us today and yeah thanks again um I'm just gonna bring my colleague Pete in uh and say yeah I Think probably Pete we don't have enough time to uh maybe open the mics for people to ask more questions but maybe you you can summarize perhaps a

couple of the questions that have already been asked in the chat is that a good uh way yeah there's there been a healthy debate on on particularly how you know simulations um play Play into this um you know how how much of a safe space are there you know what sort of learnings what sort of techniques and Tricks might there be to to make better use of those and of course we're very happy to to follow up with those but there's been quite a few comments as well because every as you say Leon we're we're

all learning and we're all adjusting and adapting almost on a daily basis so there's been quite a few comments about if I'm learning this if I'm trying to get my head around everything that's going on at the moment how can I be best placed to really Support my students so I I I I don't know if in you know a matter of seconds either Gart or or Emma have any any wise words or thoughts on on how to support their colleagues in institutions with with what must be a real you know a real challenge yeah

Emma please or or or get hard please you know feel free to decide who wants to take that one yeah it's it's it it it does start with us doesn't it we need to find a way to To hone our own AI literacy and I know that is easier said than done with everything that an educator has on their plates and more so uh an approach that that that I'm trying to push a little bit at at grenwich is that uh prompt engineering um so the purposeful construction of prompts to for a desired output is

not only just that but it opens the the the it unlocks the the mind for understanding okay what is that geni Actually what can it do and what can it not do and then um um try to apply exercises with the students uh in class and also know to to educate the students in what AI can do and cannot do I found I have basically I have two sets of students those that have already played around with it and are quite have a good understanding of it and then those who did not go near it

yet you know so C your institution cpds hopefully There are plenty of those will Al will also be a Val valuable resource to just expose yourself to the topic that's important don't be afraid of it yeah I know it's a lot it's massively much coming our way but engage with it as much as you can so you get a better understanding of it and demystify it and finally I think I would say don't be afraid of using simulations um with your students and don't be afraid to actually put that into your assessment you will Get

engagement and I think students really really enjoy it and it's probably one of the more safer options if you're terrified of the impact of AI I think it can really mitigate that and give you a safe and supportive but also very Interactive Learning environment yeah well completely agree yeah well thanks both for your responses there pet do we have time for maybe one more question or are we gonna gonna call it Leave it there I think we're at the top of the hour oops yeah I'm okay I think we're at the top the hourly on

it's it's two o'clock on the dot now so can see well yeah yeah and I'm not sure Perhaps Perhaps you have to rush off as well um but um and and all I guess are attendees of course but I'm just going to bring up this last slide here um that's a QR code which you can scan as a attendees today if you'd like to have A well in any way continue the conversation with us that will take you through to a uh a meeting booking form to meet with Peter or I and we can yeah

continue the conversation and explain more about how simulations can help you uh AI proof your assessments uh we're really to kind of yeah to talk more about this subject in the coming weeks and months um Emma and Gard would you like to like yeah as a followup if any of the attendees would Like to reach out to you or anything if you're open to that how can people get in touch with you LinkedIn I think is probably the best would you agree Gad yeah LinkedIn is good good good good place to get in touch so

yeah it's I'm I'm I'm quite unique in my name at least so I'm easy to find yeah okay my hair from the photo most likely I was gonna say yeah yeah but uh same with Pete and I as well um if you Want to connect with us we're quite active on LinkedIn as well um well thanks again to all the attendees we're going to wrap up now uh and maybe well Gad and Emma I think um if we were to carry on this conversation for another hour I think it would be great to also uh

yeah like talk more generally about yeah like the kind of Technology quotient part of it which I think I mentioned last time as well like let's let's recognize isn't it that students These days will have this technology at their disposal when they go into the real world of graduate employment so in that sense you know the more equipped they are to kind of utilize this technology uh in their real world of work and add value and maybe that's that also comes back to kind of this thing about AI literacy as well isn't it yeah yeah

okay well thanks again everyone uh for attending we'll follow up with a Recording uh and a summary uh via email but feel free to get in touch with us if you'd like more information about the subject and we'll also share that J report that we've referred to as well um yeah we'll be in touch thank you thank you you have a good day thank you for having us