good afternoon everybody um Ken was K's gone Broad and big um and I'm going to go much much narrower I'm going to talk about um Ai and education specifically large language models um chat gbt co-pilot Claude whichever one we're particularly using and I'm going to focus on higher education um and the impact that AI this form of AI might have on the way in which we spe and the way in which we do scholarship in our universities um AI is not the first um technology this pictures of AI if you Google search AI this is
what comes up I won't get into what those pictures mean but I'll leave them up there while we think of other things that have disrupted education um the book the the brro um the internet all of these things disrupted education and we we're in a time where there's a new technology and that technology is going to disrupt the way in which we work now we can either step away from that or we can engage with it and see how it moves how it changes the way that we work and how it changes the way our
students are assessed principally on their learning how do we measure their learning how do we certify their learning what is it that we're actually certifying and measuring in the credentials we give as a higher education institution um I'm looking at it from the perspective Ive narrowly of academic writing I'm the director of the academic Writing Center at the IE not just because of that I'm also Aur on various postgraduate talk courses and we assess our students through their writing whether we carry out other fortive forms of assessment whether we incorporate developmental ideas within our our
writing at the end of the day they produce an assignment and we assess it based on how well they argue in the written form now part of the the problem I'll get to this later is that academic writing the product itself is highly valued and yet it's the process of writing of producing that text that encapsulates the learning that guides the learning that is the learning by producing text rewriting text critically engaging with the ideas in it and rewriting again that text that's where the Learning Happens not through the final polished product um I also
want to briefly say I don't want to get into side issues of uh broader than than this but it's called artificial intelligence and I think we need to step back intelligence is a very dangerous word in in in the academic world um and it's led us into places that we probably shouldn't have gone um and the dichotomy artificial and human again is quite worrying why why do we think what do we think is artificial and what do we think is is human and and if human is just caring and we've heard a lot about care
today if human is just caring then I think that's got quite big implications that I won't have time to go into today so I also don't want to focus on this slide because I think it's been very well um explained by other people um the idea that AI is going to become it's going to end Humanity it's Armageddon um I think is is wrong because actually as Kean says the the impact's already here it's already out there it's already changing things it's changing us it's changing the way we work um and the way we relate
to other people so the catastrophe maybe is already here and we need to address it now um Lorna Campbell is very interesting writer on this she's pointed out it's there it's out there our students are using it um and they're being used by students in really interesting ways so it's probably time to engage with that and by engaging with that we also allow ourselves the space to think um this could be a starting point to try and rethink the way in which we ask questions about knowledge creation and the way in which we encourage the
creation of certain knowledges and again number of speakers have pointed out that our knowledge is quite narrow I'm going to draw a couple of times on we run a a series of seminars on on academic writing um and a couple of them in the last year have been really helpful for my my thinking in this area and I want to share some some with you first of there's two colleagues here is clous M and Mike Groves clous from University of Nottingham Mike R from lingan University in Hong Kong and they're experts on machine translation they
worked for a number of years looking at the impact of machine translation how do students use it and what what impact does that have on their their learning um and they've now started to look at chat gbt or they did a year and a half two years ago and they point out that again chat gbt is just another tool and we already had Gramm we already had spell check in word we already had the the blue line and the red line on everything we write that guided us we didn't question that we didn't suggest that
actually students shouldn't use spell check we accepted it we also accept final texts that probably have been proofread by somebody else by a colleague by a mother by a a paid proofer and we didn't we did worry about that but we ignored the fact that that was happening because we valued that final text we didn't worry so much about the process we worried about the product so clouse and and and and Mike suggest there the important thing to remember is the tool isn't the craft okay the important thing is the craft how do we produce
that text how do we engage with the ideas that form that text how do we put them together how do we justify them the form of the sentence isn't the most important thing um and K you talked about complimentarity and and here there's a technology that possibly could be very helpful for students who struggle to form that text because perhaps it's not their first language perhaps their education hasn't been formal and therefore they haven't been trained in rhetoric or in forming perfect sentences things that we really value in Academia and yet they're not ideas they're
not knowledge they're just a form an acceptance form of expression of that knowledge and a valued form and we we place the value on the writing rather than the ideas and I think they they're very helpful very helpful the other one I wanted to focus on was a woman called Helen beam who also gave a talk to us about academic writing as passing and and using the the the the expression passing as in pretending to be I and when you write academic you pretend to be an academic you pretend to be like like us because
you're mimicking our writing you're mimicking our way of working um so what does it mean for a student to pass as an academic writer when AI can do that for them AI can produce the text that they can hold they can present and they can pass as writers but actually they didn't write that so that and Helen suggests that we need to renegotiate what we mean by writing what do we want people what's the purpose of that writing um and in making this argument she she doesn't I think we've got to be very careful to
to sit between this fatalist idea that that's it that's the end the university can't do this anymore because AI is doing it for us but also the fantasy that this will incl will lead to Greater inclusion that this will lead to Greater Equity automatically the future is not inevitable the future of AI is not inevitable in hii she makes a number of points I think are really worth us thinking carefully about the first one and this is the big conversation in my university is assessment how do we reframe traditional Notions of assessment so that we
can think more carefully about what it means to pass what is a compet writer and there's one aspect which is to focus on originality to focus on human experience and a lot of the assignments that have been reshaped in the last couple of months at The Institute we had a window of reassessment we we were allowed to change the assessments on our course normally it takes at least two years this was done in a month such was the urgency to rethink how are we incorporate or how are we allowing students to use the tools that
they using using while not losing the craft not losing the process of learning that we we all value so much um you need to change the assessment criteria and one way of doing that is to place reflection within your assignments to change the rubrics um to think about how can we emphasize criticality how can we get students to look critically at text whether they're produced by humans or produced by by machines and particularly how can we get them to become critical about H about machine created text engage with them what are the assumptions behind that
text what's missing from that text why does that text change when you put in a different prompt that sort of Engagement that critical literacy seems to me to be a highly valuable thing outside the university and will become increasingly important if we're surrounded by AI generated knowledge AI generated TCH to be able to question them and see beyond the surface she also encourages collaboration again complimentarity that there are there are ways in which it this will be really helpful we shouldn't close them down at the Institute we and I think it's across UCL there are
three classes of AI in courses you can either go for level one which is AI is banned you're not allowed to use it now that seems a bit silly because we can't detect it so if we can't detect it how can we ban it but that's one not many people have gone for that level two says you can use it but tell us you've used it acknowledge it yeah now that seems sensible but quite limited level three is you have to engage with AI our tasks require you to use AI evaluate its outputs be critical
about it and produce your own reflection on that and level three seems to me to be the logical Direct action we should go in but obviously requires a lot of change Helen also talks about the ethical considerations and there's a lot written you'll find huge amounts about the ethics of this really we talk about academic Integrity how can we ethically use AI how can we take a text that's been produced by Ai and actually say that student has learned and this is evidence of their learning how can we do that when we don't know who
produced the text who there are big questions to ask some of them are around guidelines a lot of them I think are around transparency just talking to students finding out what they do I spoke to a student yesterday about her work and it was C it clearly been produced or edited or helped by by chat gbt um it was that I didn't have a problem with it but as soon as I mentioned it the student went extremely red she she was concerned that I was picking her up on on something she'd done wrong that she
transgressed by using this now all of my students over the last year and a half have been using it I teach predominantly Chinese students at the moment previously their texts were difficult to understand you had to work hard to get beyond the syntax to find the ideas and the arguments I the last year and a half I haven't had that problem the texts are easy to read some of the ideas are good some of the ideas are bad just like as they were for I don't rate those texts higher just because I don't have to
work so hard to mark them but I can mark them more quickly and I can see the ideas and the arguments more more clearly so students are using it but how do we maintain that academic Integrity um yeah that just in a side really um we as I say we can't detect turn it in the the system we use for plagiarism doesn't detect protect AI um I'm sure there's lots of work going on of people trying to create systems that can do that but there are some markers that are still there delve is a lovely
one um this is a graph of I don't know how they do this but somebody measures the number of words used on the internet um and this is the use of the word delve in in the internet I think this was articles and abstracts um and you see the spike there and you know what happened just before that spike it was the introduction of chat gbt chat gbt loves delve don't know why no idea why I don't think it's a particularly appropriate word for research it's too General it's too fluffy you don't you delve into
something it's quite imprecise it's not I I wouldn't want to use it so and I say this quietly but I read quite a few abstracts of this conference there's quite a lot of use of the word delve in the abstracts of this conference again that's no criticism but it is the reality that this is just one example of ways in which we're led to use words for the reason that AI likes them not because our tradition likes them or that they they're they're correct or they're the most appropriate words just because it's normative it decides
that's the useful word these are other words that um chat gbt apparently likes a lot and I like quite a lot of those words as well so I don't think they're very useful markers the point is we can't really tell Excuse Me So within um a university we we start start think well how do we do this now what is AI literacy um the last talk I gave was on digital literacy and I said at the time this use of the word literacy has got nothing to do with literacy my background is in adult literacy
have literacy research and literacy here is used in terms of competence competence in doing something nothing to do literacy I don't know why they had to use that particular word but they did so we've got lots of forms of literacy they're actually competent so how can we become competent users of AI um so I asked chat gbt and obviously chat gbt very quickly came up with a very comprehensive list now we won't go through this I I'll give you some other Frameworks for this but this is just scraped from the internet there's no criticality in
here there's no questioning of any of these values there's no juxtaposition of possible Alternatives there's no accountability no accountability all I have no idea where this comes from I have no idea whose ideas were stolen by these data processing processes that large language models use um and also there's no creativity in there chat gbt didn't think carefully about this it just spewed it out okay and I think this is where critical literacy comes in because that seems to me to be a really useful exercise to do with students ask chat gbt something and then question
it and maybe give chat gbt a better question next time ask it not to do that or ask it to to do something different if you ask chat gbt to tell you for example what is interaction hypothesis in language learning what's interaction hypothesis it tells you you then ask it well could you give me some academic references for that it gives you some now if you know anything about interaction hypothesis you immediately know that those are really old references you also re once you click you realize those references quite often don't even exist and yet
a student would present me a paper on interaction hypothesis with a series of references that I know are out of dat that I know don't exist um and the student a process of of engaging with the student and getting them to think about that seems to me to be a really really positive way forward um other people talk about AI literacy um a set of competencies compet which is here enable individuals to critically evaluate AI Technologies so the first starting point here and I like this this this description is to critically evaluate the use not
accept it not use it not have it as one of your as the main tool you use but to critically evaluate what it produces um communicate and collaborate with it work with it complimentarity again use it in order to achieve the things that you want to do but do that critically um and it's a tool it's an online tool at home in the workplace now there lots and lots and lots and lots of AI Frameworks for AI literacy already on the internet and I picked out a couple here that the general ones have these five
categories the things that we need to do you know we do need to and I think they're they're they're they're fine there's nothing wrong with them they're useful um we do need to know what it is we do need to understand where it comes from what it represents um we need to be able to use it and apply it to different things it using chat gbt to generate ideas for an assignment that seems to quite a good use of it using chat gbt to change the referencing style from Harvard to apa that's a I did
that two to two weeks ago saved me masses of time and was accurate whereas I would have missed a couple of commas and a full stop that certain reviewers would be unhappy about so that that's seems to be how do I use it how do I apply that in my life but then evaluate it evaluate and recreate what it's produced for you don't just accept it um cooperation and creativity creativity chat gbt is not creative humans are creative so if they just accept what comes from chat gbt we'll lose that creativity will no longer produce
ingenious new things because chat gbt large language models they look back they're based on what's already known they're not based on what we could know they're not based on new ways of looking at what we know which is where academic thought knowledge creation has to come from we have to know what we know but we have to think about how what we know applies in the future world in the current world rather than just simply accepting it um and then ethics ethics is very important in this that abis Smith University um the lady's name was
um I can't remember now sorry but they produced I think a really simple guide for their staff that I like again choose you know make make informed choices about what you're going to do with it don't just accept it know the strengths and weaknesses of it critically evaluate it know about data protection in UCL we've been given co-pilot rather than chat gbt and co-pilot doesn't take what you put in and add it to the language model okay so it's a closed system and I think it was Helen Bean who was suggesting well perhaps what we
need to do is create our own large language models based on our own databases which is all of us all of the things in our libraries all of the things in our moodles and our courses our materials a database or a large language model that had learned from that would probably be quite different to chap gbt which has scraped everything from everywhere without really looking and thinking why are we looking at that why is that good knowledge why did I choose it's just taken everything in order to learn um the second one is to critique
things that that and and again every every framework you look at puts at the center critique criticality question don't accept rework ask new questions and I think that's Central and really really important and you critique it you as I say you've got to ask is this true chat GPT is not always true and what's true in an academics environment we need to ask that question whoever Kean stood up and talked now we have to question whether what he says is true we can't just accept what he says we have to compare what he says with
other things we've read other things we've heard with what the person next to us commented that's how we develop knowledge not by accepting other people's knowledge as truth we accept that it could be the truth We can question it we can use our own personal experience our own context in order to better understand it and produce something new um and then yeah how do we use it AC you know the academic Integrity questions say is really really um out there so there are lots and lots of these Frameworks that are being developed mechanisms for teaching
it less so I've seen very little thought about how do we teach people to use AI what do we do in our classes um there's a lot of um I'm not sure we can call it denial but there's a lot of denial out there in our round that we were allowed to change assessment I don't know the figures but I'm pretty sure it was maybe less than 20% of the courses changed that means in the vast majority we education and the social sciences we assess through the 5,000-word essay and some of that's changed there's more
reflection there's more personal experience but essentially we assess through the 5,000-word essay so the vast majority of the assessment over there TRS over there is still done through something that chat gbt can actually do I I shouldn't say this AOW I put my own assignment for a module I teach which is a module on practical language teaching what you do in the classroom as a teacher but I assess it by getting people to describe the classroom and theorize about why why did the teacher do that that's related to a theory interaction hypothesis pairwork put people
together get them talking so chat gbt can do that too in fact did it quite well took me three or four prompts to to construct at least half of an assignment that I and I have to be honest I would have passed it it wasn't great it really wasn't great I maybe a c and I think at postgraduate level most chat gbt outputs have about a c it's not an a undergraduate level I'm not sure I don't teach much undergraduate level but I think you could get an A with a chat gbt assignment so if
that's changing our assessment needs to change and our way we teach it and the incorporation of chat gbt into what we do is really really important um now this is the core thing I think for me when when you write you the process of writing is where the learning comes a writer's Changed by writing this this is all text not just academic text but I think particularly with academic text because of its representation as what we know and its relation to other texts that we all aspire to we all know fabulous writers who we've loved
reading their stuff and it's influenced us and sometimes that because they've got really good ideas but quite often let's be honest it's because they express those ideas really effectively they're eloquent they're you know they're insightful but the Insight comes from the placement of those particular units of knowledge units of language in order to express a complex idea now that's writing that's not thinking by writing the writer changes the writer develops their ideas now if chat gbt does the writing where's the learning how are we learning we're producing fabulous correct texts my students don't make grammatical
mistakes anymore that's good but did they go through the hard work of writing that down reading and thinking no that's not what I meant or why did I say that that doesn't match with what I said over here chat gbt can make those connections and make sure you don't make that mistake you read through a text and you realize you quoted somebody up here and quoted somebody down here who says completely the opposite and then you understand how those two writers work together and you can work to try and integrate their ideas but you have
to do the writing in order to do the learning and we don't value the process enough I think currently in the universities we value the products we absolutely venerate those products we love those products published an article didn't just publish an article publish an article in a good Journal everybody loves it everybody's quoting my piy clever sentence that I formed if I didn't produce that if I didn't go through the process of thinking and learning there's no value in that piy sentence there's no value in the quote that you've taken from my text and you
put in your text somebody else will then quote your text with my text inside it there's no value if I didn't learn by producing that text so the the the there's a re reciprocity between thinking and writing writing and thinking if we can find ways in which our university courses are assessed are are are driven by that reciprocity by taking text and rewriting text taking text and evaluating text taking ideas and questioning them and if we can find ways to build assessment processes that allow us to do that that Force us to do that then
students will leave our universities able to do the same thing in the rest of the world in the workplace in the home in their Leisure they'll be able to question and to engage in some of the discussions that Ken is encouraging about the place of AI in in our societ soety um just to finish you know to engage or ignore as I say level one of our courses level one that says no you can't use it banned just seems to me to be a bit silly and I think we have to engage with this thing
just as we had to engage with the internet you couldn't google Wikipedia Wikipedia was the you know we question lots of our assignments were called Wikipedia assignments because students could go and basically get quite a lot of the information from Wikipedia and we all said we knew we knew when they' done that we didn't we really didn't okay and the internet's the same and AI is the same so I think we have to engage by engaging we can foster critical literacies goes back to Paul freri that we've forgotten literacy is not literacy critical literacy is
the important thing literacy reading and writing is not the important thing that just leads to an understanding of the world an engagement with the world and the ability to transform the world and if critical literac is not the heart of what we do as a university then I think we're probably doing it wrong um and what does it mean to be a competent writer I'm a competent writer I should be my position I have to be really I'm a competent writer why is that so valued why aren't my ideas valued rather than my ability to
express those ideas in a very narrow and let's be honest not particularly creative form of writing so the conclusion i' Dr i' come to is yeah we have to engage we have to engage with AI we have to engage with AI teachers very very frequently things are done to teachers as I'm learning they're also done to care workers very frequently we're asked to adapt to technology adapt to the things that are given to us in the classroom and perhaps what we really need to do is to say we want it to adapt to us and
our needs in the same way as as K and Pinda have explained involve people understand how they work before you launch this system now chat gbt is already there but we've got space and we've got time and and we can influence the development of the technology rather than just working reactively to it thank you all very much I'm going to allow one question very quick question there's one there you've asked a question before you haven't asked a question before I don't think there quick question quick answer thank you very much uh my question stems from
your statement that uh students are falling trap into copying from chat GPT very often without realizing what they are doing and you are catching them in their assignments so do you think we need to create awareness about this by uh treating AI uh literacy as one of these subjects just like uh mathematical literacy or language literacy or any other subject yeah thank you yeah I mean I won't be that quick but yes we definitely do we we're planning it for next term we're going to run workshops with all of our students in the first term
we've developed at The Writing Center a webinar on use of of of of chat gbt within academic writing ucl's got a compulsory um short course that students have to do but I think it'll really be at program level it's me sitting with my 20 personal Jes and same to them you can use trat GPT in fact we're going to do it now I'm going to ask it what interaction hypothesis is and you're going to critique what comes out because as soon as they see that yeah it's it's visible it's transparent you're not going to be
marked down for using it it's not transgression it's you can write perfectly well you can use this as a tool but as long they understand that it's a tool like Google translate my students were worried about translating using Google translate why why spend so much time translating a you know a a 30 40 minute interview from Chinese to English put it into Google get Google to do the first sweep for you it saves so much time but tell us that's what you've done I analyzed in Chinese and I TR I translated it through Google into
English and then I rewrote the text because Google's good at translation but it's not very good so that's accepted so with chat gbt if you place it out there hold it up show students as I say all UCL students are given access free access to co-pilot if UCL is doing that then the student should be empowered to use it and we then have to think how do we want to help them use it and how can we produce people who can shape the future using these tools and only we can only do that I think
if they become critical um readers of the output of of AI please join me in thanking professor
