Prompt Engineering Techniques (extended version)

great so uh welcome everybody it's great to see almost 300 people in attendance on the webinar my name as some of you might know is Jonathan yonei I come from a background in software engineering around 15 years in the industry previously I spent six years at Google as an AI andl specialist and recently in the past four months I founded an AI consulting company called shujin where we've been taking projects across the Spectrum from classical ml to gen AI to mlops and we've done quite a few projects around prompt engineering and we thought it'd be

a very interesting topic to come and talk in the community a bit of the agenda demos demos demos we got quite a a lot of a lot of demos I hope to be able to cover them all we'll focus on techniques going from basic to Advanced to Concepts third will be discussing the prompt Engineering Process H throughout Hands-On examples that I'll show throughout uh and we'll also discuss how the process itself looks step by step and we'll finish up by discussing papers and resources what you can read where you should find it and how to

learn more just to level the playing field we're discussing prompt engineering which is the skill of effectively crafting prompts if you look at the left you can see that we have a prompt a and a prompt when we hit enter in chat GPT can give us several different responses and if any of you have tried this in the past you know that the responses are not necessarily consistent the models are not deterministic um and we won't go into the theory but technically a prompt can end up in many different places it's your guide through prompt

engineering to guide the model into giving you the correct response or the response that you're looking for if I should give you just a mental model about how to be thinking about this I think that you're speaking to a toddler who is rather fluent in English but still has very unexpected behavior and it's your job to slowly guide the toddler by giving him all kinds of things that we'll discuss and sparking his his memory okay prompt engineering seems to have this illusion of e this is what I want to teach you today how to go

from what's on the left to what's on the right if to give you just a preview of the demo will be going up through at the end it looks something like this on the left you can see a legend of colorcoded things which um I will explain them as we go through the lecture on the right you can see the prompt that created the very good mail that you saw just previous L and you can see the different things such as examples and tones and titles all colorcoded you don't have to understand what everything means

right now we'll go over this H together so the basics we have task a task is what you want the model to do it can be anything from summarize analyze blog post write me an essay or a Cod email that I showed you before the second thing is the context this is really a crucial part most of the guidance of the model happens uh through additional context context sparking the memory of of a model is also done through the context and normally the more context we give the better uh the output will perform next we

have a role role is when you tell the model to act as something and it can be a mathematician it can be a philosopher it can be a historian we don't do this just for the purpose of styling if to give you an example which I use in one of the demos at the end there I tell the model act as a mathematician now I'm not doing it in order for it to run write fancy formulas and stuff like that I'm trying to trigger a more analytical approach ER in the generation of the output if

I would tell it act as a philosopher it might use more um more Concepts from philosophy when trying to interpret H The Prompt next we have formatting this can be anything from bullet points essay style to a poem style a tone pretty self-explanatory I can tell which tone you use it can be formal informal it can be domain specific such as legal and it can also mimic the style of a famous person such as JK Rowling or Steve Jobs last one is constraints and this is basically setting the boundaries for the model great we've reached

the advanced there are a lot of papers on the topic of prompt engineering I have tried to curate a few which I think are more interesting you can see the list of them here going from Chain of Thought all the way to reverse prompting since we don't have a lot of time together today I've selected a few which I'll go over now Chain of Thought may be one of the most famous and influential papers of prompt engineering as some of you might be familiar and anybody who's joined the Meetup we had two weeks ago um

the whole progress of generative AI is constantly viewed through the length of benchmarking we get a new model or a new version of the model somebody maybe the company itself or people from the open source Community take that model and they Benchmark it every new paper that is released is is being judged according to those benchmarks Chain of Thought is a paper that display the following capability when you ask the model and you can see an example here on the left Roger has five tennis balls he buy two more cans of tennis ball each can

have three how many tennis balls does he have and the answer is 11 you ask another very similar question cafeteria so on and so forth and it gives the answer is 27 which is the wrong answer okay um when you use Chain of Thought You do one of two things on the right you can see the same exact prompt only instead of saying the answer is 11 the answer Still Remains 11 but only only I give him the steps in order to reach that solution I say Roger started with five balls two cans of three

tennis balls each is six tennis balls 5 plus six is 11 now when I ask the exact same question of the same format he actually gets the answer right why because he's trying to emulate my process and I think some of the Benchmark improve 34% when they use Chain of Thought now this is the robust way to use Chain of Thought when you actually detail the entire entirety of the Chain of Thought but there's also a trick to use it where you simply say let's think step by step and sometimes it's also able to do

that this is a relatively easy example but when your prompts get really complicated it's not always easy to go and give an example of the step by step and you might not always know the actual steps required for Solutions so that's where the second option really comes into hand okay here are three different papers all which are post query model enhancement starting from the top right we have knowledge generation let's say I have a question I want to ask the model asking the model the question straight up doesn't always yield the best result what this

paper showed is that if you before asking the actual question ask the model to generate some knowledge based on the topics from the question and then append those topics with the question itself you get better performance the second paper on the bottom right is similar here you ask the model prior to querying it to give you a hint based on the topic this is very akin to what I said before the mental model where you're trying to spark the toddler's imagination so you're giving them a hint or you're giving rather the model a hint of

what information he should try and tap into the third similar and most recent paper is about step step back and abstraction this is St two-step process I'll give you I'll give you the example that's written here we asked the original question is some kind of physics question which requires you to understand a few Concepts in physics so the step back question is what are the physics principles behind this question And now when we actually ask the question with the ER abstraction it's able to understand that it should probably go figure out those Concepts and make

use of them when answering the question H the full paper details exactly how much better it is over Chain of Thought which is one of the best methods so far so you can see Chain of Thought in yellow and you can see step back prompting in green both of them I think are papers by Google stepback prompting has really um excelled great um now I'm showing all these examples not necessarily that every time you write a prompt you'll go and you'll have a checklist and you'll go over the checklist and you'll go oh I put

in Chain of Thought and I've addressed the knowledge generation and I've implemented emotional stimuli no I go over this these so that you will get a sense of what works and what doesn't work and then you'll be able to have a better intuitive feeling as you're going through the process of prompt engineering most of the time you won't be using all of the concepts you'll see when I show you uh some of the Demos in a moment I won't be using all of these all the time it's rather cumbersome and not all of them are

always effective I'm continuing emotional stimuli a very interesting paper which kind of drives home the point of maybe talking to a toddler emotional stimuli showed that if you simply append at the end of a prompt something which appeals to the emotional side of the model granted that you think it has an emotional side it it performs better they've shown that prompts and I think they have almost 15 prompts in the paper itself pumps like my job depends on it or take pride in your work and give it your best which appeal to like the self-monitoring

part of that I see that we have a a poll here so um if you could just take a moment and kind of fill in the poll discussing briefly like what is your experience with prompt engineering if you're a non-tech function with experience or an expert or if you are a tech function with experience that would be great so give it just I will give it just like 10 15 more seconds fill in the the uh The Prompt itself and then I'll continue on reviewing the following papers great great almost 300 people take some time

to to answer all of these uh polls great okay let me continue so just to recap emotional stimuli very interesting paper they found 13 prompts which appeal to the emotional side of a model such as self-monitoring and social cognitive um um side it's again been shown to improve and similar to the second option of a Chain of Thought it's really easy to implement you'll see this when I show the cold email example where at the end I tell him you know my career really depends on you getting this right self-consistency self-consistency is very intuitive it

simply means that you'll be querying the model several times as you can see the model here on the right has been queried three times two out of those three times it says $18 so uh when pulling this together the answer that was most common is the $18 and it just so happens to be the right answer we're using the greedy simply flat out asking the question got it wrong okay another one is um recursive criticism and Improvement now we're stepping into methods that you should use always okay at least until you reach a higher level

of expertise in prompt engineering recursive criticism and Improvement despite its uh daunting name is simply using the model asking it to critique your prompt and then taking that critique into account you can either tell the model to put the critique into the new prompt or you can read what it thinks and uh do it yourself reverse prompt engineering this is as the name implies starting from the end result you may have an example of a very good mail that you or a colleague or taking my example have written but you don't exactly know what prompt

could generate such an email because maybe you want to generate many more emails like that or you want to have a template of a cold email where you can just plug and play and change things like like the product catalog or the seasonality let's say when Christmas comes up as was just the case now you give chat GPT in this instance you give him a actual output of a prompt and you ask chat GPT to give you the prompt that can generate such an output this again is a very good method for anybody starting out

because it's really easy to miss or have a lot of block spots when trying to prompt engineer if you can recall what I showed in the beginning the legend of like different colorcoded things I put into the email that wasn't even an exhaustive list you'll see more when we talk about Concepts in a couple of slides and so this is hands down the first method that I recommend anybody start with prompt revision very intuitive you just give the prompt at the stage you're at ask chat GPT to make it better if anything it helps organize

when you're really going down a rabbit hole of prompt engineering and you're working on a prompt let's say I have an example here at the end guard allocation problem it's a prompt that's very very hard to solve I first of all recommend each of you try it and second of all it took me around an hour to get the model to give me an answer which is actually correct um I use this method why do I use this method because you keep on subtracting and adding stuff and changing things and so on and so forth

and it gets messy it gets messy if even if you're an organized person and at the least chbt will help make things more organized and if you've been following along the models appreciate organization okay we'll see this at the end where I even put line delimiters between different parts of like the task and the context and so on and so forth great let us jump into a demo now I'm just going to run all of these cells let's start from a basic example okay I just want to show you how a very basic prompt would

look um I am just going to utilize task format roll and tone wanted to remind you here of the colorcoded Leen and this is the prompt that we'll be using I am telling the um model you are a scholar historian your task is to summarize any given text in five bullet points with a sense of urgency so here I'm using I'm using role I'm telling it you're an historian I'm telling it what task it is about to perform okay you can see that in kind of an orangish color I'm telling it the format in which

I want it to be I want it to be in five bullet points and I'm also giving a tone sense of urgency technically I don't have to really give anything other than the task if I don't give the task I'll just be having a a mute conversation with the model itself but here I've utilized four different concept to give you a very basic example and if I run this I'm using open AI the default model which I'm using is a gbt 3.5 Turbo and I've simply giving it here a very long text of World War

II great moving on to Advanced methods so I've only selected a few again we don't have the entirety of the day to to spend reviewing every every different H type of technique and great Chain of Thought you see here um two cells the first one is without Chain of Thought I'm simply giving it a math problem telling it which is the faster way to get to work there's option one and then there's option two and when I run this cell it will say option one option one is not the correct answer let's not do the

calculation now but anybody wants to do back of the envelope on their own and and if we actually give it um a Chain of Thought So here I'm saying the same way thing again which is the faster way to get home um and I'm saying option one will take 60 Minutes giving the calculation option two will give 100 since option one takes 60 minutes and the other so on and so forth and then I'm giving it another different question after having giving you know an example great and not another different question I'm giving the same

question again sorry this was a different question and this is obviously the same question I'm trying to compare and this time it gets it right it also details how it did the calculation trust me option two is right option one is not the correct here is an example of generated knowledge um here as you anybody who has a key eye will see that I switched the model these models work on very very large uh prompts if anybody has been following right now open AI GPT 4V is at 128,000 topic top tokens clae from anthropic is

at 200,000 tokens H and we're using maybe in the prompts that I've shown you the previous one I think is maybe 150 tokens not even scratching the limit you need to understand that 200,000 tokens is around 1,500 Pages um sorry around 500 pages the entirety of the Bible is around a thousand pages I think it's 1,543 or something according to my last Google search it's kind of hard to to demonstrate some of the easier Concepts using a very strong model but when you scale the prompts these behaviors start to re exhibit um for the sake

of uh of um readability I'm showing here at a much simpler model so here uh I use it part of gol I asked the following question sorry part of golf is trying to get a higher point total than others and I asked yes or no anybody who's familiar with golf and uh ironically I'm not very familiar with golf but it's not as B the same as basketball or football where you're just trying to get a higher score there's all these games with par and underpar and so forth and so on so when I ask the

model initially it says yes um and when I ask the same question again but I also give it some prior knowledge which I generated by asking a different model I ask it and it says no the objective of golf is not to get a higher point so this again demonstrating how when I straight up ask the model it doesn't know when I give it more information and I promise you the information was not a by the way the objective of golf is not to get a higher score no the the information that I added was

some information about how the game of golf actually works you play rounds and each time you strike the ball and so on and so forth and you get points and it's not it's not that I queued him into the answer I just gave him an understanding of how the game works next example is self-consistency self-consistency is a bit hard to Showcase so I just did an example here with chat GPT which I I also think is it shows two things first of all I'm not getting the same answer every time and I'm using the exact

same form with the same exact model it showcases the these models are not deterministic and to some extent even though they're not deterministic they do have um a good knowledge because they compress all of the data of the internet here you can see I only tried the same prompt three times and two out of three times it says yes and it just so happens to be that the actual answer to this question is yes rather than no a very basic example of how you would utilize self-consistency you can obviously create a small script of code

to run the same prompt several times and extract the yes or no and and and Pull It accordingly great let us go back into the deck okay now we will be going over some Concepts these are not specifically things that will be going into the prompt but there are definitely things that you have to think about when you're doing the process of prompt engineering so I'll just briefly review them and give a couple of examples on the different ones ER be specific don't use ambiguous language or concepts and if you do use something that is

left up for interpretation um explain it remove fluff we've talked about we've talked about the method of asking cheat GPT to revise your prompt H do it yourself let chat GPT do it if you let chat GPT do it definitely monitor the output that you got from it another cool trick which I I don't discuss but we do discuss in some of our workshops is you can ask tat gbt to put in bold any changes that is made it's not exactly Microsoft Word track changes but it's a very effective method to understand what might have

changed from the last time explain explain explain you'll see this in my later demo where I asked it to divide the workload equally between the different guards it doesn't work it doesn't work it doesn't understand in the context of the problem what equality means only after I explain dividing the shift equally means that no guard has more than one shift added on top of any other guard does it get it right prioritization you know prioritize the the tasks there's this whole concept of Lost in the middle which we'll see very soon when we discuss papers

don't be negative the models don't really handle instructions s such as don't do this or don't talk about that topic be positive they much rather be told what to do than being told what not to do a separators for clarity recency bias is a very big thing um as I said prompts can get very very long when we work with companies it's not normally on two three liner prompts it's very very long prompts utilizing all of the context window ER trying to figure out how we can make the prompt more effective so that we actually

have more examples so that it fits into one run one request before we have to go into you know pipelining or breaking it up into several requests it turns out that there's a thing called recency bias where a the model tends to pay more attention to things that come towards the end of The Prompt and there's another paper about lost in the Middle where they showed that the model actually pays attention more to stuff that comes either at the beginning or at the end but be be reminded this only applies to very very long prompts

another thing I want to mention some of the techniques that we've discussed here such as a Chain of Thought They don't work for the very small models if you're using Zephyr or mix 7B or Falcon 7B which is 7 billion parameters the the models which weigh around 4 gigabytes they won't necessarily exhibit all of these capabilities all of these papers have been tested on the very very large models some additional concepts are negating bias um a very interesting example is I don't know if it still works with chat GPT but if you ask certain models

the following question um Jacob and Daniela both work at a hospital what do each of them do unfortunately some models will still have bias in them and they'll say Jacob is a doctor and Daniela is a nurse so be aware of this whether it's bias that comes with the model whether it's bias that you might be bringing into the prompt be aware of hallucinations and this is a topic widely discussed I won't go into this more this is a problem how to get more examples examples are really important um when Gemini was released last week

ER it 30 out of 32 benchmarks that actually um beat GPT 4 Microsoft I think it was yesterday released a paper where they were able to use gp4 and beat Gemini how is this possible nothing has changed you know it was the same GP GPT 4 when Gemini released and it was the same GPT 4 that Microsoft used they both changed around the number of examples which is first of all to say that they weren't necessarily using Apples to Apples in those comparisons which is still being debated and second of all examples are really important

if you don't use any a lot of times you won't get the right answers if you use 10 or if you use 200 you'll also be getting different results so you too should be using examples one very good way to get those examples other than manually crafting them is to have a stronger model such as GPT 4 Create them for you very good way you say here's an example create me other similar yet different examples and then you can go use them either when you're using chat GPT with a different context or you can go

use them when you're using a weaker a cheaper easier to run on your own computer or somewhere else model chattiness versus being succinct a lot of the times you might want the model to just answer with a yes or no um maybe because it's a you know it's too chatty it's giving you a very long answer and you really just care if if if the answer is yes or no and there are trade-offs that this we discussed this more at length in our Workshop um yes you sometimes want yes or no because you just want

to be able to work with the answer let's say you're trying to do self-consistency so you might just want to either get a yes or a no and be able to to pull them up and see you know two times yes one time no but know that it can also sometimes hurt performance the model the way that they work I won't go into this now with probabilities predicting the next token looking at all of the tokens that has come have come prior if you don't let it kind of explore it might not reach the right

answer last uh concept which I use is telling the model how to engage you'll see this when I show the called email example here I am um telling it to ask me questions uh when it doesn't have all the information needed great so this is a demo of the Guard placement which I spoke about I show this because it's a very hard example I urge you to take the the problem that I'm showcasing here and try to solve it on your own as I mentioned before it took me nearly an hour to get this working

and I had to use a very specific model so the problem you get a list of people including their professions you need to assign them to guarding posts based on different constraints how does a bad solution look where where did I start out I started out writing the following prompt saying you have 10 people the last three have driving license they need to stand guard it is your job to build re schedule where the work is divided equally between people there are two posts which need to be guarded gate one should have one person from

6 to 10 and two people from 10 to 6 at night ham should have two people and one person at least should be should have a driving license ham is only guarded only during the day from 6:00 in the morning until 6: build a three-day schedule that did not work at all I got a very bad response it did not heed the hours it did not put the people with the license where they needed to be it did not get right who had licenses it put the same person on different posts at the same time

and it also made up a third post of its own how does a good solution look this was my final solution H here I'm using things I told you before such as telling it you're a mathematician why because I'm trying to trigger to elicit the model to use an analytical approach okay everything I do here is by Design when we work with people H we do a stepbystep process taking a couple of hours in order to showcase the process and make you understand that every everything you write in a prompt even if you're unaware of

it will affect the model it might affect it in a good way it might affect it in a bad way but it will affect it so I say you're a calculated mathematician you're you meticulously plan every step of the way and I start detailing the constraints and the guidelines I say you have 10 people person one through 10 should be allocated to Guardian post I explain why guarding uh posts are and what the direction of the shift notice that I'm I'm being very um you know organized here I'm I'm giving things in bullet points I'm

giving the name at the beginning I'm putting the shifts uh at the end I want the model to be able to to put these things together to take them into relation together I keep on detailing information about the personnel and how the shifts are happening and um the requirements for the different post and some restrictions that I have then I go and I give it the output format and this was just something that you know I typed up I said the final schedule something like this shift m and and then I got like sort of

like you know tired I just said you know person one one person two two person 3 three and hoping that the model would understand that I just want to you know I want it to in the output to give me how many shifts each person has and I finish up by saying create a schedule for the first day following the above guidelines perform this step by step while detailing each step what you did in the way which is sort of like a wink okay I told you I don't necessarily use all of the tricks that

you saw before or that we reviewed but here I'm I'm kind of tapping into this Chain of Thought let's think this step by step um great at the end this is the output it does actually do everything step by step and at the end the step which I care about is five and it gives me the final schedule it tells me how people need to stand guard and it also at the end ensures that there's a fair distribution of shifts a very good example I really urge you to try it at home a these are

some of the I won't go over them but these are some of the steps which I tried to write myself as I was correcting the prompt I went through more than 30 iterations I changed way more than 30 things but I wanted to give you a sense of how hard a problem could be and what like the different steps that you need to go through so let's talk a little bit about the process of creating a prompt again you will not be going into YouTube or medium and finding the 15 perfect prompts which will just

work for you I do recommend going reading some of these articles even though they do become a bit repetitive at some point H just to get a General sense but by no means will you be able to just take those prompts and apply them on a a scalable production robust use case it requires you to know how to do prompt engineering and to bring some domain knowledge and it's an iterative process so let's discuss this iterative process great prom engineering this is how it looks like er you state your problem you examine re relevant information

you think about it for a bit you start by proposing an initial solution I gave you the example with a guard problem it was a very bad initial initial solution but it's something to iterate upon and adjust adjusting the solution has three high level steps you test you examine the output and you do some research okay maybe some things that you know beforehand maybe some things that you you just didn't think of in the context of this prompt and then you rinse and repeat as you saw before rinsing and repeating can take you 30 plus

steps or maybe you'll get it right on the third or fourth try we've been doing this for nearly I've been doing generative AI for almost a year now previously at Google and now at my own company but we've been doing this for nearly four months and never have I just flat out written a prompt that worked didn't need any Improvement didn't need anything better it's not to say that some my prps aren't relatively good when I just write the initial version but there's always things that you can make better also I don't discuss this here

but the models sometimes they change over time okay there's this whole thing where open AI claim that they haven't changed the model and other people are accusing to have changed the model under the hood um fortunately unfortunately I subscribe to the one ones that say you know even if you think they don't you have to account for that once you're happy with the prompt and you've tested it out manually a couple of times to see that it gives you the uh result you're looking for your best option is to probably test it at a bit

larger scale again we don't go into how to do this because we don't have time in the 10 minutes that we have left but in the workshops we do actually explain how to scale this how to have your model be tested at scale using other models and some Automation and when you're finally happy you should launch your Solution on the right you can see an image where um I took from a guy named Nabil who has a very good um article about prompt engineering it's not always the more is better you have to be very

attentive it's a step-by-step process for a reason and you have to see that the things that you're adding are beneficial and are not taking away H it's a process and you have to see that you don't go overboard great we have reached the final demo and now I will show you how to actually write the code email that I showed you before step by step to the best of my ability in the five minutes that I want to spend on this demo a short reminder some of the color coding uh techniques that I will use

are role task engagement context uh so on and so forth you can see them here on my screen for yourself my first version of the email is as follows I simply say ER I assign a role I say I want you to act as an email marketing expert specializing in cold emails and your task is to create a Cod email that will attract the attention of my ideal customer Persona I translated this into the format which open AI expects to get on the API request and I'm using the most powerful model to date which is

gp4 I run and the result that I get I you know the models are still not deterministic but not very good okay and you can see for yourself it's rather generic with points one to five um it's a bit chatty it's saying sure I'd be glad to help you craft an email it's missing things like subject it's not actually taken into account what I do um it's not very appealing to the eye there's no visuals and really no call to action or no ask from the email I want to make this better okay so again

I go and I think and I say okay um let me add some more things if I'm working with a chat model such a chat GPT I could add some Rules of Engagement I say ask me questions until you have enough information to perform this task I give it more context I tell it that I'm actually an online Sports retailer and I sell stuff I say that we sell stuff at a high to a high competitive um level and not only do I say high competitive level I explain similar to what I said told you

before about the guard allocation where I said divide it equally and then I had to explain what equally means here I say at a high competitive level and then I explain a high competitive level implies that so on and so forth later I say we work with loyal customers then I go on to explain what loyal customers mean and in the task before I said my ideal customer persona but I did not specify what my ideal customer Persona is so here I specify I say they're athletes um and lastly I I remind the model that

uh we sell sport equipment online which is a wink to trying to you know make use of the relevancy bias there are other things which I use here which are not specifically color C is you know I start selling it at the end the last line you see make it flashy with Emojis H again I translated here into uh what chat GPT will understand and I run it let's run it again [Music] much better much better okay we're making progress and this looks much more appealing it already has a subject and not very chatty it's

not telling me Oh yes of course I can go and write this email for for you it would be my pleasure and honor no it's straight up giving me a subject it's using emojis it's a it's very appealing to the eye there's a call to action button at the lower and um generally I'm you know I'm I'm not too displeased with this mail it could work however I could make this better you could make this better and I go and I look at it and I start to think okay what's what's wrong first of all

it's it's not specific enough it's just Unleash Your Potential tailored for Champions er no discounts just value which maybe I I don't really wanted to say that um it's a bit lengthy okay I don't really feel like going and reading everything so why why would my customer really feel like reading this entirety of of a mail ER the subject maybe is is a bit too too lengthy it it kind of looks like an advert um and I go and I think about all these things and and when we work with customers we think about this

you know together and when we do the workshops like how do you take this step and you you move on to the next level Now by no means was version three the final version but since we we we are limited to an hour here I am jumping a few steps ahead to the final version and I've added a couple of things first of all I add ER more things to the task I tell it to keep the email under 10 lines and I also add a which is being more specific and then I tell it

use a tone of a positive encouraging coach again I'm appealing to athletes want to use a tone that they're familiar with okay this is a good style choice choice for me giving this specific problem then in red I give it the different steps that I want to follow in order to create a mail and here uh I think you should find this interesting the first step is I'm telling it to actually check my website check my website see see what I'm I'm selling become aware of my catalog so to speak then pick an item from

the page generate some information about this item to use in an email okay here I'm I'm doing steps in the red and I'm in blue I'm tapping into the concept that we saw before of the knowledge generation I did generate a photo of the item using delay 3 and then and only then write the mail and incorporate the photo into it I'm not done yet I want to give it an example of how it might look so my desired output is the following I have a subject line with a title hi and and all of

these things in um in in Brackets is basically a template for the model to H adhere but still be creative okay I don't want him to give me this as an output I want him to use it in order to to H keep it kind of a in in in in check okay you can think about as you know I'm keeping the model in in bounds of where I want the end result to be so here is an example um with all different things I'm I'm telling you to put some content and use emojis and

highlight certain specific key features and at the end I say my job depends on how good this email is don't be chatty give me the output and only the output in the format that I asked for some Concepts which I've used um but not especially that are the concept that we reviewed before such as you know length recency bias explaining separation recency bias and so on now when I run this mail ER by the way if anybody sees the max token and thinks to to themselves maybe that's what's actually making the mail not long that's

not the reason it turns out that if you don't specify with gp4 vision the max tokens it just so happened to only return the first line to me um max tokens 3000 is way more than the length of this mail um if anybody wants the notebook I'll share the notebook in the utility section on the top it actually has a token counting method so you could know exactly how much each answer each prompt that you're using has um great so this is the response way better great and I really feel that I could use this

I I deem this a success great back to the slides a few papers um we don't have time so I won't review them I've just I'll just briefly go over the list H we have lost in the middle which is a very interesting paper about how things that you put in the beginning or the end affect the model's output much more another reluctancy to retrieve injections is interesting article from anthropic where they showcase that they use the entirety of the 200,000 context window and yet they try to inject wrong information but the model knows the

context of the thing around the injected information and Exhibits a behavior where it doesn't retrieve the inject information which is very powerful very interesting take a deep breath for Microsoft directional stimuli and and hallucinations in a snowball all relevant papers I'm sorry but due to time sake we will not go over these I will however be happy to share them H we've already seen the guards demo I will talk about briefly about the future prompt engineering I think in the near future we'll have more and more prompts being discovered erh will get longer prompts which

are more akin to computer programs there's a paper on the left here which tries to already categorize different prompts into a analogous computer programs the prompts will gain more control will be able to manipulate better using prompt engineering and also the models will allow to more control we sort of figure out this prompt a bridge between a prompt to its output through the model and if we're going really far and that's my personal prediction I think that 10 years from now we're we're not going to have so many R&D people or software engineers and product

people are going to be able to do a lot of things just using prompt engineering and platforms like um juo how to get started and these are three very good resources I recommend Lear prompting dorg there's a master the perfect prompt YouTube video and write expert prompt on medium which is a three articles which I really liked and we also do a hands up Workshop where we go through similar examples to the one of the code emails that you see here on the right and that we reviewed before and we work step by step on

similar examples really helping you understand how to do prompt engineering we also go into deep deeper dive on things like agent active prompts what is memory how it works and so on and so forth thank you everybody for joining our next Meetup will be in three weeks from now on the 4th of January in this Meetup we'll be discussing Enterprise gen use cases I will make this Meetup available as we finish this session if we could just do the last poll now uh we'd really appreciate if you guys could vote we want to make this

community valuable we want to make these sessions valuable to you therefore if you could just take a moment to vote on the next topic we will try and do the ones that you most care about we guys are here for you and um I'd like you to thank you all for for joining my name is Jonathan if anybody wants to reach out for me to me my email appears on the slide it's been a pleasure on my part thank you all for for joining uh please like And subscribe as they say or rather join our

uh next Meetup in three weeks on the 4th of January thank you very much guys