thanks hello everyone so we will we will start the last but not the least session panel of the day I amre KFOR the co-founder and CPO of high AI a platform to hire and trade your AI agents we are using the infrastructure of scaleway training models involves compromises while inference um represents a million of diverse tasks a report by Bloomberg estimates that the inference Market should grow over 80 billion in 5 years this session will focus on the technical challenges and practical use cases of inference optimization will be the main keyword sustainability cost efficiency and of course we want performance inferences dear speakers starting with you Jeff please introduce yourself and explain us how your company relates to inference okay perfect so Jeff wck uh Chief product officer at uh at amp here um if you heard my talk a few minutes ago um I'll I'll go a little bit deeper uh so amp here we're about a six-year-old company uh we're building CPUs that are uh incredibly high performance uh but are also incredibly power efficient uh we have a couple of products in the market today we have our Amper Ultra processors we have our ampere one processors uh so you can span anywhere from 32 cores to 192 cores and that means you can deploy these uh really anywhere you could deploy them out at the edge you can deploy them in big uh hyperscale data centers uh and they're meant to be deployed for data center tasks of all types uh in Cloud environments and as it relates back to this inferencing is one of the major tasks that we built the processors for uh that we optimize them around and it's one of the major tasks that people are running on our processors today so our goal is to provide a more efficient uh inferencing Solution that's also more flexible and and that ends up resulting in power savings and cost savings for customer thank you hi everyone Thanks for attending this conference my name isri um French as you hear from my beautiful French accent F of gladia we are a company focusing on Speech to Text uh and focusing on very high quality speal text dedicated to multiple verticals uh we started the company a year and a half ago um and we have lot of customers that are very satisfied from uh the media industry to the call center industry um meeting recordings as well as medical um it seems that our model is also extremely good at Medical we didn't expect that um and that's pretty much it yeah thank you hey so I'm Mayan I'm CTO at ber uh so basically what we do is that we search Clips in how long videos um so to do that um we use a lot of different AI models and we have pretty specific challenges regarding inference as videos are so maybe that's it for me thank you thank you Maran uh so uh I think you have a big exclusive announcements for us we have a big announcements yes we um are releasing our new model starting on uh on Monday uh it's a module that is uh based on the whisper architecture um and is called The Whisper zero because it doesn't have any alucin naations we're going to talk about inference and hallucinations which is one of the big problem that you have with llms and architecture such as whisper which is the same as the llm architectures uh when you're performing in France so we released this new model it's uh 60 times better than 60% time uh 60% better than the previous model um and we use 1. 5 million hours of audio in multiple environments meaning like clean environments not only clinical environments but like environments that are complex also phonetic uh sorry phone calls um as well as very nice environment uh on the street for instance um and so this model is performing like extremely well with an extremely high accuracy that has not been seen in the in the market so far so we're very excited about that wow thank you for this announcement yeah thanks now Marian you you are a new CTO yeah so can you share with us what are your challenges about AI today so so yeah indeed it was a Le two weeks ago and I'm the CTO um so at P we have many Tech technical challenges um so mainly as we process videos um we have challenges not only um regarding the runtime of AI but uh also with the prep processing because like a 4K video can reach like 160 GB of of data and you need to process this in a few minutes um and so you have uh to optimize your model a lot you have to optimize the pro processing a lot because uh you need to decode the video you need to to be super with your formats and to gpus properly and we do all this at the edge because we don't want to have an a neck with the network so that add another layer of complexity thank you Maran and now Jeff uh how are you helping Tech teams to build uh cost efficient inferences please yeah that's a that's a big part of what we've really been doing it it comes in a couple different lay layers um obviously we're working with people like scale way to provide infrastructure um that end customers can take advantage of but it goes beyond just just that um it's also us partnering directly with uh companies that are building or running uh AI models uh we have our software libraries called aiio uh that makes it very easy for them to get up and running so if they're using p torch today they don't need to change anything they can go and run on our processors right away uh and get extremely optimized performance uh but it also we end up working fairly closely with uh with some of these companies um at an even deeper level to help them actually optimize their their models and help economize things um one of the things that often happens is that uh companies train their models on big GPU clusters and then their first instinct is when it comes time for inferencing to run those models just deploy them on the same machines uh the problem is that gets really expensive really fast and so uh companies often come to us asking whether there's a more efficient solution uh that'll help them to be able to run their models more broadly not run out of money um and and also deploy them in more places so we often work with them to help to um not only get them up and running on our Hardware um but of often times it also involves um working with them on um their models themselves uh to see if there's actually ways to um you know optimize the model size in order to um you know be able to run on more efficient Solutions so kind of it it runs runs the gamut uh from the hardware side to the actual kind of full software and use case super thank you uh just a question for the room please to know a little bit our audience please raise your hand if uh you already deploy an inference llm inference please okay and keep your hand if you uh keep your hand up please if you have uh llm inference in production please okay thank thank you very much so um now we will uh spend more time in a debate debate style don't hesitate to react each other and I will stay silent I promise so the first uh theme will be uh about machine learning operations Hops and uh automation as part of your workflows so how do you keep how how do you maintain a great quality of inference uh with automations as you said um with the zero hallucination how do you how do you do that in your companies do you want to start you want to start well I think maybe yeah I think you you can probably talk more to what you specifically do uh you know since we're Building Solutions um you know for people that are actually building models um kind of what our big part in this is really making it as easy as possible um in order to um provide the most computing power so that you know whether it's improving training accuracy whether it's helping to quantize things with greater levels of accuracy you I think we're we're kind of in the in the game of making it easy for you to be able to do more and for it to be very very simple to deploy but I'd love to hear more about what what you're doing specifically with your model itself uh in this area so for the model there are multiple ways to see it you talk about automations automations can be Hardware or even um let's say software Automation and then you have process automation which are extremely important so we deal with audio but you can see you talk about machine learning machine learning is like widly spread in the organizations and some of your cases are for instance fraud anti frud systems so it's not only about how you deploy in FS right on CPU gpus it also goes into how do you automate your workflows within the company how do you collect the data from the customers how do you reinforce your system to make sure that that the model gets better over time and how do you under what is called the drift of data how do you make sure that your model doesn't overfit the reality and to avoid having the processes that are coded but are not fitting the reality anymore so that's the one thing on the process side and then you have the question of the hardware and the software and that's extremely complex you have zero done time that's what people are asking if you're training something training is easy training you can break the machine you can relaunch nobody cares you're going to be for one week if you break the machine for instance in our sector for transcription you don't have any medical transcription anymore you're breaking a medical service if you're breaking your sales process with your customers you're losing all the transcriptions the CRM won't be up to date you're breaking the the productivity of your of your teams okay and maybe they were not taking notes during the meetings they were focusing on the customers looking at them in the high to make sure you have a good engagement you're losing all the informations that's bad and I can go on and on and on so you have two things to consider the relability of the infrastructure and the relability of your processes and the relability of the software chain and the data chain all over so it's not only about it it's not only about infrast it's also about the data that flows in and out of your systems thank you so on our side we we also have um topics side to infrastructure and a lot of CI to to manage uh the models we have around 50 yeah 50 models um running in our app so we we need to maintain all of them um we need to be able to be sure that every time we release uh we are sure that the model don't regress um but we also have to attach a lot of meta dat both to the models and to the output we are producing because as our models are running on the device of our users um if there is a crash we need to have a way to get back logs to understand what happened um um and so we need to know uh which model was the one which crashed um why uh for example on which data this model was trained so to um to be able to grasp really quickly what was the issue and to solve it as quick as possible okay thank you do you want to uh B well I I I like what you you brought up about the differences between inferencing and training in terms of how reliability matters because you're right in in training reliability matters but a different way it's recoverable um and you're right that on the inferencing piece having incredibly reliable uh systems with high up time is is incredibly important and kind of a related piece to it is is also um having predictability and consistency in the results for inferencing matters a lot more as well in training um having variability in in in the training runs isn't really that big of a deal um it doesn't matter um if it changes from hour to hour how much throughput you have for instance but in inferencing it really matters if the latency changes dramatic aut atically from minute to minute um you need to have consistent latency those latency taals kill you because the time that the inferencing result took you know a second instead of 100 milliseconds that might be a completely lost result for somebody um they've moved on they've missed you've missed the critical time frame and that is one of the areas that we really focus on from a hardware perspective is ensuring that we're delivering Hardware when we say scalability we don't just mean lots and lots of compute we also mean that you can scale um to many many users and many many workloads without losing that consisten in latency because once you lose that consistency in latency it's useless okay and now um developers um they in the in the room and online watching us uh they want to have practical advices about how to cut their spendings uh from test to production it's it's obvious so what are you doing in your companies please um to control your inference cost and um to reduce your impact on the environment it's also something sustainability is very uh big quite quite now so far as the approach is quite radical uh we don't use the cloud for inference like we use um we are a specific um case because our customers most of them are either streamers or video editors so like they have really good Hardware have workstation with good gpus and large amount of memories um so it easy for us to to run models on the hardware um and so we we exploit this computational power um and deploying um or models as an API don't have like a really huge additional value for these customers thank you um so I'm going to go Jeff you already gave a big part of the answer um there are many techniques to optimize your model uh not only optimizing um in a sense that the quality B goes so to the size of the model um and what's funny is in the latest development of of AI it seems that the two of them are correlated so if you distill a model for instance there are ways that you can get rid of many parameters in your models and you have like we had MERS before we had like many companies talking about TMS these big models lot of parameters some of them are useless even worse than that some of them are penalizing you and so getting rid of those is helping you to have a better memory impact reducing the size of the hardware reducing the time to get your model up to speed into the memory reducing the speed of uh improving the speed of inference reducing the latency and so with that you able either to fit more models within the same uh systems or you able to have smaller GPU card for instance and a big card like we said v00 is 250 Watts what is 250 Watts it's two MacBooks like old MacBooks Intel MacBooks I'm not talking about the arm MacBooks they are way more efficient so it's like two big computers that's one card if you go on the T4 or L4 that was announced 60 watt so it only not only matters on the quality that I said it matters on how many models you are able to run on the same infrastructure and it matters on what is the impact in the PO consumptions and lastly because you have a smaller model or you're reducing for instance the representation of the numbers within the card maybe you don't need Fa l bites maybe you need 16 or eight and you get rid of things that don't matters what's going to happen you need a bit less transistors and so if you look at the end cost for the environment of those cars and those those elements that we are building 80% around 80% to 90% of the CO2 that is being emitted is not at the cons ition level it's when you use trucks to excavate all the the the hold the Earth and do lot of treatment to actually get the materials to build the cards so 80 to 90% of the impact is not when you're running and savings I don't know 200 Watts it's when you're building and so the last transistor you're going to have the better impact you're going to have in the environments so you have many levels to optimize but we need to think h hand when you're building a cloud or an infrastructure so goes back to the cloud and INF France in France is really good goes back to the cloud should you buy yourself gpus or not we are going into the cloud for impacts that are good impact in in CO2 because if we don't use infrastructure during the night because everyone is sleeping in Europe someone else is able to use it so we don't have to buy to to build two cards we are building twice less cards better impact on impact on the environment so we need to think hand to hand when doing Ai and we have a responsibility toward that okay thanks Jeff you want to add something yeah well I mean I think that the the elements of power efficiency are really and sustainability are really important because yeah we a lot of times we think just of the uh the power that's being consumed when you're actually running the CPU or the GPU and those numbers are you know can sometimes be kind of Staggering I mean you look at something like a an h100 C that that can be 700 Watts um so you've got the power that's being consumed there the hidden cost of power is the cost to cool um that Hardware um you don't directly see that but behind the scenes a lot of power is being used to actually draw all the heat um away from from that CPU or GPU so you know being able to to run at lower Powers has a dual benefit there and you're right um there is all of the other um impact from the actual manufacturing of the CPU or the GPU so I think that's where if you can improve utilization um use every single cycle on that CPU or that piece of Hardware you that means less of them need to get uh need to get get built uh that means a smaller environmental impact and I think that's where I think from our perspective you know being able to run with CPUs means they utilize all the time because when you're not running inferencing someone else might be running a web server on that same piece of of hardware and and and just to give you a tiny number you should look at something that is called pu po usage efficiency which is the ratio of the number of wats that you're using to run the system and the number of whats that you able to get from from the calculation mhm and if you look with all the systems it's extremely interesting to look at different Cloud providers and if evenin the same cloud provider depending on where you are in the glob it won't be the same so if you're running a GPU in Canada during the winter it's going to be very good if you're running in Canada during the summer it's going to be very bad if you're running the same card in the same conditions but in a state where you're using cold to run the uh po the I would say the U the electricity is going to be bad we are luy we are in France we have nuclear energy so you have a good pu we have a a good WEA pu is really good and the CO2 impact is really good and all those things should help you to make better decisions um towards the optimizations of your inference um regarding what you said um with uh us more chips and like reducing ha extraction um by using Cloud um I'm not sure I fully agree with that because um today um if you look at let's say any iPhone you have pretty powerful chips in it uh I don't know if you run test with the last um Apple silicon chips uh but you can get pretty decent results uh when running models on them um so I wouldn't be so partial let's say towards the cloud um for me it's there is value in using chips that are already there because everybody has a phone as and a laptop agreed and don't agreed I think it's the point there uh homeair is doing an amazing job as pushing the same type of architecture you talk about apple chips this is this is the architecture that you have right uh in the morning we have announcements for M2 chips in the cloud so still you can get very good consumptions and price for the type of flops that you're going to have um and and finally you have a questions that is really interesting because we don't have the same type of personas and we don't have the same type of customers your customers don't really care about privacy they're running their own stuff forsel um when you're running an hospital I don't know if you went in hospital I will let you check the type of computer they have and Doos they're running probably Windows NT 20 years old you can't run a good speech recognition and I challenge you to run those models with good um transcriptions of the of the drugs that that the the the physician is spending I'm sure 100% you won't be able to have any wor coming out of it so it depends on your customer depend on the Privacy it depends on many factors if you're looking in the Enterprise grade I think you have not really big choices except going into the cloud because there is another factor that you need to take into considerations is what is called the capex the capital expense you're going to have at first for your project how much money you're going to put up front and the cost for money let's be honest right now it's pretty expensive two years ago it was pretty 0% interest or very close now it's very expensive should you put your money there money there or should you go into the cloud cloud is a bank cloud is a bank they're taking money buying also I would say uh like big spaces building data centers managing the cooling making payment of fronts to buy a server and you're only paying a rent on a day so so that all these questions about privacy Etc security resilience okay those things are super important like if your customer is losing or your your user is losing like Po in this apartment should you stop the transcription of the Physicians that that are there no you still should have still the the the process running so all these things are extremely important depends on the use case on the person and so again for instance you should Define what is the appropriate use case and and the tools you're going to use you want to rebound again just maybe a bad joke data center can take fire yeah we remember I won't say anything so enough remember so so let's talk about about the edge approach um I think at humare you have um a lot of uh companies asking you how to build quality with Edge devices uh used cases can you explain more about this yeah that's um that's is a big part of our our strategy we talk a lot about Cloud when we talk about Cloud we're really talking like very very broadly uh we're talking about cloud infrastructures or Cloud architecture so that could be a big public Cloud that could be an Enterprise running their own cloud but increasingly it's it's a lot of edge uh deployments that really look like the the cloud because what's happened at the edge is that you used to have a lot of dumb devices that were just uh ingesting data they were sending that data back somewhere else it was being processed and then it was being sent back again and that doesn't doesn't really work now because you're trying to do more analytics you're trying to apply a lot of AI to the data that's coming in and a lot of times the reason why you're collecting that data at the edge is because you actually want to use that data at the edge to do something else think uh your traffic monitoring in a city or some sort of autonomous uh driving solution so you need a lot of comput sitting out there really close to where the data is is being collected and um the challenge is that many of the devices that we're sitting out there traditionally just don't have that type of compute they were low power devices but they didn't have a lot of compute because they didn't have to um so what we've spent a lot of time uh on and we have a number of of customers uh deploying our Solutions into is these High Performance Edge environments where maybe you don't even have the ability to cool um there's no fan uh maybe it's a really dirty environment maybe you just can't afford the power of a fan so being able to run say a 40 watt CPU that has 32 cores um out in these extreme locations is is incredibly important and it kind of fills that Spectrum in of different ways that you might be running aiing so quickly on that you talk about coing that's never you need to under the conditions I'm going to give you just numbers um we run many benchmarks but if you take a GPU and you go overover for from 40 degrees which is the maximum operating temperature you should have and you go to 50 degrees so only 10° 10° the PO of execution degrade by 20% just to give you how difficult it is to operate those systems so again we talk about CL providers environments so maybe you can run on hedge but you don't handle exactly the condition is running so 20% mean like it's thr down by 20% the numbers if you go like above 50 50% it's like 50% above sorry 50 degrees it's like 50% degradation in speed so it's really important to have good conditions as you as you mention mhm and Marian um at uh po GG you have a lot of devices you are deploying on on the devices of your customers your users uh do you have over ages approach age approaches well um for the main challenge when running uh oi on edge um is ending the difference you can have in Hardware um so it can be having different instruction sets uh in CPUs for example uh you you get different performance when you have a ax2 or ax 512 uh available on your CPU um and then regarding the GPU uh I would say there the software is as if not more important than the hardware um because I don't know if anyone in the room um has tried to to deploy a model on AMD GPU uh but it's really not fun at all okay you two you already deployed on this Hardware uh AMD or yeah okay so it's a good transition for the next question uh about the hardware so um when you need to choose the right Hardware what is your advice because we have CPU gpus clusters we love all these s but uh maybe you can go deeper in the tech how to Choose Wisely the good Hardware mhm I think from my perspective you know it it starts first with uh just understanding what you're trying to actually accomplish um you know what your key objectives are and that's where that's where it changes between training and inferencing because like I was saying in my talk earlier training your objective may be number one priority is just time to train um and power and and cost are a lot lower down on the list um and also things like predictability and things maybe not they don't matter as much uh on the training side but on the inferencing side um you know it's it's prioritizing what you really care about because you're you're probably deploying at scale um and so now all of a sudden you know cost efficiency power efficiency consistency of performance those all start to um start to move up the up the ladder in terms of the the priorities so I think first it's it's you know it's understanding what your priorities are it's understanding the requirements for the actual model you know how big is the model what's the memory footprint is there a way to actually optimize that um you mentioned earlier a lot of the parameters useless I was talking to a company um that um has a big foundational model and after they did the initial training um which was many many billions of parameters I mean they they found now that 95% of the parameters um have no bearing whatsoever on the final results now you wouldn't know that until you actually went through the months of training um so they did they successfully proved out that they could build a model that big um and that it could generate maning results but in the end like they can go and cut out 95% of it um with no decrease in accuracy and and you're right sometimes actually some of the data is actually um detrimental put more data in so it's also looking at where the where are the um ways that you can actually economize the model itself and then it's about picking the right piece of Hardware um that it's going to most efficiently fit into U which from our perspective most of the time you know we think is is a CPU for for most models but if you need to use a GPU um at least pick one that consumes the least amount of power and is the most efficient thanks yeah so F story are the big nerves in the in the room it's easy to see you like you have long airs and you have some oodies so quick story and and funny story I I did run like lot of benchmarks on Hardware this is my my job in the past if you're doing training what's important for you is the bandwidth between the dis and the GPU and what you're going to do you're going to load your truck like with big batch and you're going to launch on the GPU so you do care about the size of the highway when you're doing in France you're looking at cost with thing and latency and so I did run some test say how can I optimize my cost and how do I put as many card as possible GPU cards as possible on the same motherboard because a part of the cost is the motherboard and then you have the um the different gpus it happens as you take the PCI Express so it's been technical but like the size of the highway and you divide by two doesn't change when you're doing if you divide Again by two doesn't change if you again you divide by two you start to have like 5 to 10% loss in in in performance it means that you can put like four six eight times more uh cards on the same Hardware because you can split the highway and having one line per GPU if you're doing training if you do that you're going to kill the performance and you're going to kill the price per dollar that you're have so Define what is your use case what is your final goal and find the way to have the best setup so there is always a button neck if you're running let's go into databases if you're running in databases your button neck is the memory it's the ham if you're running like web servers your bottom neck is the number of CPU and number of phases you're going to have and after you choose what is the key element that is the blocking then you're going to find what is the next best Best Hardware so if I took take a datab B the best price for me the bottom neck is the ram I'm going to take the highest Ram possible and then I'm going to find which CPU is able to handle that big amount of RAM and then I'm going to choose my CPU don't do it the overway run if you're running a web server you need to have as many fret as possible and then you choose which Ram is is applicable and then you're going to choose your nvme do you need nvme actually you don't so stop buying the Top Gun infrastructure find what is the best for use case the same for AI has been the same for database been the same for web server it's been the same like for years and years it's just a different way to buy it because the cost of the item that you need to Leverage is way higher than what it used to be so GPU is probably 90% of the hardware that you you're buying when you're buying a server like that so okay it's just like being bigger in terms of problem but the problem remain the same what is your case how do you optimize you want to add something about Hardware um yeah I agree a lot with what you say Jeff um When developing model um you are tempted to make it work uh that's your main goal first uh but it's easy to overlook the compression phase and just try to um find the proper Hardware uh too soon uh it's really important to uh to focus on compressing the models first uh because most of the time you have a bunch of us of useless parameters in them um and AD powder uh we uh always try to make the model runs uh run on CPU first and if we really don't achieve this then the GPU is the last result um but that's our main focus yeah CPU first okay and so uh if I understand your opinions is that the two capis are latency and cost for your inference we agree mhm okay so now about the future if our startup is growing we want to scale so scaling is challenging so um what are your tips uh to get ready to scale and maybe you need to think when you are building not when you are scaling because it's too late maybe what are is the same today if you're doing like speech recognition you need to have your website up to date uh very likely to scale between 8:00 a. m.
and 8:00 P. M that's not a new problem it's the same for the website it's the same for Black Friday the same for like weeks of days the same for your it for the size of the VPN and that's what we saw with the covid um the vpm were like needed to scale Etc so you need to think about it as a website and you need to to think about the traffic and you need to find what is the right provider to have the stock and the flexibility while not eating your cost and and the problem is like we have a big Starion of GPU I was in San Francisco like this summer it's funny you had some meetups that were dedicating on how do I get availability for the gpus because every time there are gpus coming in the market they are preempted by the big Labs so how do I run in France and how I'm able to sell my product and that's a big question because you're going to have have either flexible providers and you will be able to have the availability between 8:00 a. m.
and 8:00 p. m.