Scaling AI with Open Source Tools in the Public Cloud

Our next talk is scaling AI with opensource tools in the public cloud with Brian and Ron. Brian is a solutions architect who supports mutual customers of Microsoft and Intel to leverage and optimize technology and enhance business objectives. Has a rich history in cloud, IoT, AI, and customer-driven products.

a diverse and comprehensive understanding of the industry from roles at Microsoft, New Relic, Atachi, uh, and the Department of Homeland Security. Ron is a customer focused technology and sales leader with over 20 years of experience driving cloud adoption and digital transformation across the Americas. Uh, he has uh, led initiatives that help organizations modernize and migrate Linux, open source workloads to the cloud.

and they are here today to talk about scaling AI with open-source tools in the public cloud. So, welcome Brian and Ron. >> Thanks Neil for having us.

So, welcome everyone. So, I'm Ron Abair as Neil mentioned. So, we'll be talking about the scaling AI uh and taking open source to the public cloud.

Uh and Brian is with me as well. So, my counterpart from Intel. So, Brian, maybe you go ahead and um go and take it away from the the agenda standpoint.

Yeah, folks. Uh, good to be here with you today and, uh, going over this. So, you today we're just going to be building on the architecture for AI.

So, there's obviously, you know, Microsoft's in the forefront of a lot of this going on and then Intel is uh, still supplying a lot of uh, the CPUs and other other solutions underneath. So, the ability, you know, your ability and your capacity is one of the things that we want to talk about today. Security and compliance is obviously going to be critical.

So we, we know, we're hearing a lot about what we're doing, but we want to make sure that what we're doing is also secure in what we're doing. And then obviously your scalability and elasticity. So how do you scale up?

How do you scale out? And then how do you return on your profits and making sure that you're not overusing an environment. So we'll be touching on some of that.

And then uh you know just going through the major pain points that we're facing or we're seeing folks facing in the industry around organizational changes. you know, how open source is so critical to AI and then tackling the security challenges with AI. There's just a lot of places that we want to go look and then we'll be finishing up with a case study with everybody.

So, real quick, you know, going through the major pain points, the one thing that I want to bring up, and this is something that you, if you start listening to announcements out in the industry, you're going to hear a lot about uh where the industry's at, uh where the bottlenecks are at, and how big it's getting. Um to give you an idea there some of the off uh color concepts that are out there right now is around how many gigawatts will we need and that's going to be somewhere as as large as 90 gigawatts in the next u four years. Um you start breaking that down and you start looking at how many racks that would be you're talking about several uh 100 million racks that we're looking to put in the United States alone.

So we do feel that this is a huge environment for everybody to be in. Um, and we really want to make sure that we're growing this uh with our community uh and making sure we're there to support you all. So, Ron, I mean, there's a bunch of industries like what what are you folks doing, right?

Absolutely, Brian. So, thanks. Yeah, you know, it's not just chat GPT that we get into, right?

AI is absolutely exploding and and as uh Brian mentioned just the amount of power that's needed to scale AI is it's a huge issue that we need to solve but it's also just goes to show how how often and how fast that we're starting to consume AI in a day-to-day environment. Um so if you look uh we've been working with a lot of customers in all sorts of verticals. In fact you know our our teams now are verticalized because customers are looking for very specific you know assistance and help on their their own AI use cases.

It's more than just chat bots. So if you look at these slides here, you know, everything from from manufacturing to healthcare to retail, AI is just taking over the the vast majority of the the use cases and all the new projects that are happening today. Uh everything from things like new robotics, right?

Robotics need a lot of compute to figure out how to uh leverage um you know the the physics and be able to move and do surgical care, for example, or even just doing fraud uh analytics and things like that. Um, you know, historically it's been running on CPUs, but it needs things like GPUs and accelerators to be able to um to be able to get the amount of data since there's just so much data that's um being created right now. Uh so just the fact that there's just all these different use cases that are happening within each individual vertical just goes to show that not it's not just going to be LLMs, it's all sorts of different models that companies are are adding their intellectual property to and then they're being able to you know produce services and better products for their customers as well as save cost and make things more streamlined.

So there's a ton of these use cases that I won't be able to uh to share in such a small amount of time, but it just goes to show that AI is absolutely real and you know this talk is really to help customers plan and prepare for the scale and leverage the the uh AI open source tooling that's available uh to customers today. All right. So on the next slide, uh we do want to talk about like why is it sometimes hard for for our customers to get started when when implementing AI.

So if you look at the slide, you know, there's a a few stats that I thought was really really interesting, right? So 95% of decision makers say it's key to this implementation, but the path toward it is is unclear, right? or that 63% that it's repeatability being able to not just make this like a unicorn project but a be able to leverage this you know this knowledge to other projects in the future.

So there's things like like those and just the fact that they have to scale their ef AI efforts. So is it the manpower, the compute power, it's like how do you take one AI use cases and be able to do 50 100 of these um at scale. uh and and also one of the other biggest challenges is like how much time does it really take to go from um you know like a an idea an AI idea to actually implementation and getting business value out of it.

So there's a lot of these different things that customers have been you know saying like hey these are we want to get to the to the business value u as quick and as uh you know cost-effective as possible but how how do we make it happen right so as you as the previous slide showed there's a ton of use cases when we won't be able to tell you exactly how to do it for every single one but what we will do in this talk is really talk about some of the building blocks from the AI tooling to be able to make it happen so Brian go ahead and take away on on why open source is crucial. >> Yeah. So, you know, I think folks, you know, open source has been around for a long time.

We know that back in the days and I think it's been great, you know, working with Microsoft and folks like Canonicle around how do we make open source really crit uh a key part to how we're adopting and so over the years you we've seen a lot of things in open source, right? It it allows you to avoid vendor lock in. Um you get to go and pick your projects and get involved.

you get a lot of freedom um throughout the community and and and honestly as as a uh using some of the projects almost too much freedom you don't know which one to choose sometimes but you know that's where it's important to go and make sure that you're working with the right folks um that are driving these projects forward and that's where Intel likes to work across the communities and make sure that not only are we involved um you know and making sure the x86 platforms in place but also that um what we're adding into the open source community is really driving um driving to a bigger change. So, it's important you don't want to be in a closed system. You want to make sure there's flexibility.

You want to make sure that your migration costs are being, you know, looked at, right? Because obviously going from a proprietary lock in system to to a nonproprietary. There's sometimes there's good things that are happening.

There's sometimes that it's going to be a little costly. But that's where again a lot of us um you know, both Microsoft, Intel, and uh Canonicle can go and work with you a lot on that. um and we do every day.

Um and then we have open source you know and just itself right you know the workloads are are going across uh the world right on different open source programs and I think it's important to to look at the scale function here is that over the last couple years it's just been um booming on how many new projects are coming out how AI's been adopting the open source community and this is allowing for a lot of things like we'll talk about security but it's allowing for security to become more robust because more eyes are on these these solutions. So, this is a very good place for everybody to be and and and kind of be driving together on how we're going to work across uh open source. And in that, like we want to just remind everybody that Intel is sometimes the the um the uh the man behind the curtain, right?

Obviously, we're on the bottom at the x86 platform on your CPU level. Um but at that place, we're doing a lot still in the open source community. And you'll see here that like Intel on the x86 um ecosystem advisory group has committed committed um the um AMX solution that we have which is renamed to ACE now um so that other folks like AMD in the x86 environment can get this.

So it means that the work that you do with us and in these open-source environments, you're going to be able to leverage across all your products and you're going to see a future with it through all of our competition as well on x86. And that's important to make sure that we don't want to leave anybody behind, right? And that's also what the open source community is really good about is making sure that we're pulling everybody forward, that everybody has the opportunity to get involved um you know at more of a global scale and then have the feedback that's needed.

And so we were working with through making sure that we have seamless integration with these open source frameworks. Um that's TensorFlow, PyTorch and then on the Intel side we have one DNN and we're making sure that when you go in and adopt these you're getting both your performance tunings in place you're getting you know best on platform uh operations and that you have uh a big ecosystem right that can help you um when you have questions and then your performance scaling is also critical like we know that more and more that load and capacity um especially with capacity challenges coming that we want you to be able have a high performance and scale. So I imagine Ron that you're working with a lot of folks in the industry, right?

And some big names. >> Yeah. As a global black belt at Microsoft, I've been working with customers since the very beginning of of cloud.

Uh so it's just really cool to see the the growth of where customers are taking their ideas and applying different type of tooling. being able to lead the open source initiative um you know in our global black belt team we work with so many different technologies and tooling uh in the industry. So, uh, that's the that's the great thing about open source, you know, to to Brian's point is that you have the accessibility to all this tooling and and in AI things are changing so fast.

So, you want to be able to see what's best to breed, what actually is going to be using for uh for your use case. Uh and and Brian um you know didn't mention this earlier but part of the reason why that Intel and Microsoft are partnering for this talk is that we are consistently the you know two of the top five contributors to open source right so not only are we using open source technologies you know for for our own technologies but we also want to improve it for the community and make it better especially in this day of AI where things are just changing so fast. So the the ability to have access to this tools is really important.

So these are just some of the things that we do in cloud in every day. So the fact that Microsoft for example is working with hugging face and Intel as well. Uh so it gives you know developers and you know folks that are you know working on large language models or whatever models that they want to the ability to have like a a GitHub type of experience where not only they can collaborate but they can you know create projects and be able to just essentially take those ideas and leverage different toolings that you know that hugging face has um has access to.

So Brian mentioned TensorFlow, but there's also things like open source data solutions, not only databases, but analytics. That's going to be really important. That's where customers are being able to take, you know, their their own data and create their own intellectual property because now that they can train their data, do rag patterns and all sorts of interesting AI things with with their own data, right?

And so that requires databases and data platforms like you know my SQL or Postgress which is fantastic for you know their vertex capabilities. Um so there's just so many different things that that Intel and Microsoft are working together uh to make these platforms better. So every time there's a new version there's you know performance improvements there's new capabilities for AI.

So, it's really important to trust, you know, um, you know, companies like a Microsoft and Intel because we're always contributing back. We're part of the community and making things better. All right.

So, the next thing is really about, you know, tackling the security challenges. Um, you know, it security has always been one of those things that's top of mind. Um, but we also in this day of AI, it's just really important to make sure that um that you know we share some of the best practices and things that can help keep you protected as well.

Um, and I can't remember if Brian or you and I had had this slide, but it's it's really important make sure that your company uh does not land in the front page of one of these, you know, of these uh news articles because that's really the worst case scenario that our, you know, some of our, you know, companies that we work with that don't take, you know, security seriously, you know, end up having a breach or data loss or anything like that, right? There's that old adage that it's not if you get, you know, compromised, it's a matter of when. And those, you know, attackers are getting definitely a lot more um sophisticated because they themselves are leveraging AI and they're writing, you know, malicious code, you know, at extreme scales nowadays with AI.

So, it's really important to stay ahead of that curve. So, it's not a problem that you can't ignore. It's something that you always have to to stay on top with.

Um, so that's where you being under the umbrella of having, you know, technologies with with Intel and Microsoft helps give you some of that protection, right? Brian, if you wanted to, you know, add some color commentary, feel free to, but I know it's this is an area that's >> definely your slide, but I did want to add a point to everybody. So, we're coming into this world called postquantum, right?

And postquantum is getting so I want to put everything out there to to folks to think about is postquantum people are starting to actually take the information that's already encrypted going across the transits today and they're storing it so that when postquantum really happens they have access they're going to go and look to to decipher it in the future. So this is why it's so important to start today, right? And that so what you know what what Ron's talking about here, all these other issues that are attacking that's a now problem, but the now problem is also the tomorrow problem and we need to start working harder now to get the security out there.

So Ron, I'll let you continue, but that's that's the one thing I would endure there. >> Yeah, it's a great point. you know and you know working with customers you know trying to get these workloads in cloud you know we get the question all the times like why can't I just do this on prem like why can't I do this you know on my own and not have to trust a cloud vendor right one of the things that um I've learned over time just experiences with literally thousands of customers and just going through the growth of those number one it's like hey they need the capacity first right and you know with the now number of data centers and the capacities that that Microsoft has um globally.

That's number one. It's like you have to have access to the raw horsepower like the infrastructure to be able to start these things. Next is you have to have the tooling which you know I mentioned earlier is that hey you have hugging face and all these models these toolings that are available through either our marketplace or you can bring your own.

But the the third piece of the puzzle that people sometimes forget about is that security aspect. It's the access to the tooling. that's access to the capabilities, right?

So, there's things that um people don't realize that, you know, especially if you're a small shop, you know, there sometimes it's just way too price prohibitive to be able to do things like DOS, you know, type of mitigation and all that type of stuff. By putting your workloads in Azure, you're already in the umbrella of our security, right? We're protecting our endpoints and all those things right off the bat.

So that's kind of where um working with with the cloud providers really gives you know our customers like that inherent you know uh protection uh underneath the covers of what we do. So you know it's from from a Microsoft perspective security is actually our number one priority. Um and that's actually a public statement.

So it's even above and beyond what we do for AI. We want to make sure that our customers trust our cloud because it's their their data, right? It's their, you know, 24 by7 application that needs to run.

So, these are just some of the things that we we work on from a day-to-day basis and as an employee, you know, we have our commitments internally to make sure that these um are robust and that these are going to be trustworthy for our customers. And we leverage a lot of the Intel technologies like the confidential computing which we'll talk about here uh in a minute about those are sort like the the base ingredients to make sure that we have um these secure AI you know platforms for our customers to use. So, not only do we have security as our number one initiative within Microsoft, right, we help out with data platforms, um, and we make sure that we secure that whole end toend software stack, which we'll talk a little bit more about here in the future as well.

As well as sort of like the things from a um, you know, maintenance perspective as well, like when customers have to um, pass audits and that sort of thing like how do we automate that? How do we make that more uh streamlined especially as the number of auditing tests just keeps growing at scale and then you have you know multiple uh number of projects that you have to audit as well. So these are all the you know three main things that you know we often u talk to customers why you go into cloud and how does open source help solve a lot of those problems for you.

So here's just a quick slide of just basically like um our our our software future initiative that I had talked about. Um when you look at basically technology, it's multiple layers of the stack per se, right? So not only is it the hardware layer from the chips, it's also basically the power, the cooling, it's the, you know, the software, the operating systems, the middleware, and then you know, and things of that, right?

There's networking involved. So from a Microsoft perspective, this is a kind of like an eye chart to talk to a little bit about, you know, we have a security by design. We basically have a no u no zero trust type of architecture.

We have, you know, red green teams that are always um you know, working behind the scenes to keep um our platform safe and our customers safe. Um so there's always continuous improvement on this. We know that the attackers are also getting more clever and more sophisticated.

So we're trying to be ahead of that game, right? And doing it on behalf of our customers. Another thing too is um it's the software life cycle, right?

It's it's really easy, especially in the open source uh space because you know code is available uh for for customers as well as people that are doing malicious things. So it's really important to have those systems in place to be able to scan them as well. Right.

So, I know Brian, you know, you you and Intel have some really cool technology um to talk about this, but feel free to jump in and talk about where Intel is adding uh to this as well. >> Well, with Intel and and obviously with Microsoft, I think what I want to step back and just remind everybody is open source, right? This is a big thing and theme here, right?

And with the open source, we also have OCP, right? And OCP safe. And that's something that Microsoft and Intel both are involved in.

And we also have the confidential computing consortium and we're both involved in that. So this is why it's so important to look at these open- source platforms. It's it's bringing security out and transparent.

And then it's no longer security through obscurity. I remember that term from the 1990s and the early 2000s like the more obscure we make this the more safe will be. And that's not true.

Um, what we're finding now is it's important to get into these frameworks with these folks, especially when it's AI from your data ingest all the way to your feedback and integrations back and and reporting. We need security across the platform. That's what we're doing as a team.

Uh, Ubuntu is there with enterprisegrade um solutions as well, especially with Ubuntu Pro coming up. And that was the big thing. Even in open source, a lot of people think it's just going to I'm going to go in there for free, but everything costs money.

um you know whether it's your maintenance or someone else and that's great when we're seeing these folks in the communities in the open source community saying look we're going enterprise grade we can be there to support you and I'm loving this I'm loving this right from the Intel side is more and more of these ISVS are are giving the big group hug and saying from the enterprise we can work with you as well and we can help reduce some of your costs for those type of operations and that's kind of critical so like securing your software stack and then Ron this is, you know, a place that you folks obviously focus a lot on, right? What I'll say is Intel's there from the CVE, from the hardware perspective. We're making sure there's as least risk on the platform as possible at the hardware that can be and then Microsoft is obviously working with us and other partners like Ibuntu to be able to take it from the top.

So, Ron, what are you guys doing up there? >> Yeah, and and being a software and cloud company, you know, we take code seriously and we make sure that like that code is secure. There's obviously a lot of testing that's involved.

Um, part of the reason why, you know, we partner very closely with with Canonicle is that they're also a huge, you know, contributor to open source and we partner with them. You know, they're the basis, the operating system for a lot of our, you know, Linux based open-source, uh, technologies that are in Azure. So, they're a fantastic partner with us.

Uh so I just wanted to put a a special call out to them because not only um does Microsoft have to focus on you know the the actual code being secure so does you know and they they have a product called Iuntu pro that takes it to the next level. So from a supportability standpoint, from the patching standpoint, right, they're not going to leave you hanging, you know, unlike a lot of open source unfortunately where projects go die and there's no, you know, there's no code updates and any bugs that are there are going to be there forever. Uh so that's where being able to have the supportability uh in the security both from an Azure perspective and then with a canonical through their Iuntu pro gives you a lot of these benefits not only from you know the fact that they you know give you patches and updates but they give you that you know 10-year um you know kind of headache-free um you know license where there will always going to be there for you especially if you are in a heavily regulated industry where you have to think you know you have to be fed RAM compliant HIPPA compliant.

Uh so that's one of the reasons why when we work with customers in those specific verticals, you know, these are some of the baseline ingredients that we need because they're already secure by design and and makes, you know, creating new solutions a lot a lot easier and a lot safer right off the bat. So a few things about the uh security and compliance from a Microsoft perspective, right? because not only are we working with just a handful of partners, we're working with a, you know, whole open-source uh ecosystem.

So, there needs to be a baseline, you know, security approach. Uh, and because a lot of these, um, open- source solutions, you know, start off with Linux as, you know, their OS um or their containers that are Linux containers, there has to be protections in place. So there's um some automated uh security and compliance that we that from an Azure perspective we do on a you know day-to-day basis.

We continuously check you know the software and we um do audit test on your behalf to make sure that those things are compliant. So those are just some basic things that you know we um we provide uh from an Azure perspective. And so for a lot of customers out there uh that may not have access to these if they were running on on prem, but we give these as a service um for for our customers.

So I'll turn it over to Brian to talk a little bit more about the the confidential computing that we mentioned earlier. >> And just in the in the interest of time and we're going to move a little bit quicker through these um at the end here because I know we're running up on it folks. But what's important here is so take a look if you haven't done the research on what a trusted execution environment is this is critical.

So in the past with the first thing that we've always done was you know uh encryption at transport right so that was your SSL your TLS right then we had encryption at rest and this is where storage encryption on disc started happening and now we have encryption at runtime so your trusted execution environments are are safe against other malicious actors whether it be the bare metal OS itself or whether it be an admin on the system these f these places are very important and with Microsoft we're releasing this type of solutions into the general Azure populations. And so you'll see this here with the confidential VMs. And in these VMs, like I said before, we have AMX, you can start running AI today.

And that's important, right? So you can start looking at your um solutions to make sure that they can scale. What we want to do is like so with the Xeon 6 processors, so these are going to be um online definitely um coming to the cloud near you, you know, that kind of thing.

But at the same time, uh, what we're having, um, and looking at the most again is AMX, your TDX security. We're making sure that your secure and scalable deployments are in place for AI because that's going to be the big thing here. So, it's not just how fast can you process and train, but it's your inferencing, it's your data and the security, it's a lot of other things.

So, we can't forget that as we move forward through these these times. It is a whole solution. Um, we're, you know, we got to remind everybody, and that's what I've been doing, is that you can't just allow your data to go untouched and un unccleaned, um, around this time.

This is important. So, you want to be working through that. So, Ron, like towards the end, I mean, you folks are doing this for, you know, in the wild.

So, what's that looking like? >> Absolutely. So, you know, coming at the end of our presentation here, now you you see a lot of the the building blocks and the ingredients to making a good solution.

And now we'll talk about an actual customer you know putting this into a real world you know scenario. Uh so you know I want to talk about one of our customers which is Western Sydney University right they're they're very typical for a lot of customers where you know they have a lot of hardware on prem they had to maintain that you know they're getting into some security um things that they need to um focus on but they also had requirements like hey sustainability how do we reduce cost you know do do better with the environment but also leverage you know new technologies like with AI right so this is where you know they work closly with Microsoft and Microsoft's you know you know global black belts with the account teams and and very specialized resources uh to help them understand exactly what their problem statements are what their visions are and maybe even help tweak that to get them you know make the art of the possible happen. So in this case that they wanted to modernize a lot of their infrastructure on prem right where they needed to be able to leverage you know platform as a service so and and be able to take advantage of AI so this is one where they were able to do a lot of lift and shift and and modernization from onrem uh to Azure with using tooling like our Azure migrate um and be able to use some of our AI tooling to do assessments to see exactly what you know what that environment would look like um and also make that transition that migration a lot easier and simpler uh from on-prem to Azure.

Uh so that's really one of those where it's a full you know full team where it's it's really helping out understand the customer better. It's really about solving you know pain points and leveraging AI tooling to to make things better. So, not only they're saving money, they're able to meet their sustainability goals, uh, and make, you know, their infrastructure a lot more manageable now because now they're able to manage it in the Azure portal.

But this is just, you know, one of the many customers that we see this this story um, you know, unfold, you know, the same way, but maybe different tools, different AI, you know, technologies that that they leverage. But this could be um you know your company in the future if you're you're thinking about hey how do you leverage Microsoft Azure how do you leverage Intel you know start from a legacy environment that you've been maintaining for years uh and that you need to modernize and migrate to the cloud. So, uh, Western, uh, Sydney University is just is one of the many ones.

You can feel free to take a look at our our website for other customer stories. But that's really where where we ended at as far as being able to showcase, you know, the different, you know, tooling and building blocks and then also showcase a customer that were able to put everything together, you leverage the technologies and the people and create, you know, some really cool, you know, end goals in in solving a lot of their business problems. So, just wanted to end here.

If you ever wanted to um you know contact some of our friends from Canonicle for example, feel free to take this QR code or or send them an email. They're one of the partners that we both collaborate with. So they're uh kind enough to be sort of like the the intake to be able to take some of your questions and and comments and maybe some of your scenarios that we can help out with uh and solve those uh together.