um so my name is Dan I'm a PM I work on our compute platform and our uh in our Compu networking product area and I'm Nick I'm an SRE working on our compute platform and our networking infrastructure so if you don't know Spotify um yeah it's that one uh we do music things um and um and yeah we uh we work on the uh the platform there so um one of our core values is uh playfulness um and sometimes what that means is you get team names that don't really mean much so we work on
a team called Alf um which is yes from the show from the late 80s um and um you might be thinking what's the best place to get support for a team that's called Alf and that would be of course at melm um so that's super confusing for our users but um that's kind of how we do things um so in terms of scale um these numbers are actually we got them approved so that we could share them with you we haven't talked about these publicly before uh well the stuff on the left is all public
but all the things on the right um that's all brand new stuff so yeah in terms of scale um about 62 million uh monthly active users um in about 180 markets um from the platform side of things we're about uh 2800 Engineers over 500 squads that's another thing we call them squads as opposed to teams we do about 2900 production deployments a day over 3200 microservices and then in terms of like compute itself um we're over 40,000 VMS these are all at Peak numbers over across about a million cores four pedabytes of memory about a
half a million kubernetes pods um and then yeah the big one there 1.5 terabytes per second of egress which is actually larger than the U Amsterdam internet ex exchange so yeah we uh we have quite a bit of scale there and um of course the fun things happen at the boundaries so across clusters and things like that um and all this is to say is when we have downtime somebody notices so uh January 14th of uh last year was not a super fun day this wasn't compute related but something that the ALF uh Squad actually
had to deal with as well um and you can see there uh when we have downtime people notice and they are very vocal about it except I guess if you have a zoon um but yeah one of my favorite things to do there pop open Twitter whenever we have an outage which luckily isn't that often but um but yeah people notice and they definitely are very vocal about it um so what does our back end look like um actually looks like this um this is a graph of all of the microservices that we have um
on our clusters the sizes there are um the amount of traffic that they're doing um so you can see this is yeah this is a picture of our uh entire back end not super helpful for anything but um yeah it's just a cool image um so what we're going to be talking about today in the next uh 27 in minutes here is we're going to be talking about what kubernetes looks like at Spotify today um we're going to talk about why you would want to rebuild all of your clusters and then we're going to talk
about how we actually went about doing this so let's talk a little bit about what kubernetes at Spotify looks like today um the golden paths or the pave paths or the pave Road or the you know insert however you want to uh think about them um is something that is very very big at Spotify the golden paths are really a way to deploy something to production it's it's like a tutorial essentially um and we have most or quite a large number of our backend services that are using these golden paths to get to production um
what that ends up looking like from a kubernetes perspective is we have these massive multi-tenant clusters um we probably have fewer clusters than you might think that we do um but um they actually are they tend to be long living and um we upgrade them so we've had clusters that um have been around since we first moved to kubernetes which was years ago um so a lot of the kind of the our first iteration at uh kubernetes and the way that we thought about it was for folks that are using these golden paths and supporting
them getting on to the Clusters I think the big problem with that of course is that when you think think about kubernetes on Aggregate and what it looks like at Spotify you have a pretty big portion of those workloads as I was saying that are on the golden path but then you have a whole bunch of other ones that are not right and of course as a platform um especially as a PM we need to we can't just address a portion of our users these numbers are not accurate by the way more just for representation
than anything else but we have a ton of other stuff on our multi-tenant clusters things like non- golden path services especially for optimized Services um those big big circles that you saw in that that graph uh earlier those ones can't be on the golden path because they have optimizations that require them to you know be a little bit more performant or fine tuning because they're doing so much so much traffic um we have things like cron jobs so yes those are also on our production clusters um and we have other things we don't actually extract
abstract way um the kubernetes API at all for the most part so our users have access to clusters mostly at the Nam Space level um and can kind of deploy not whatever they want but they can kind of deploy things to those clusters as they see fit so what we end up with from a platform perspective are these massive multi-tenant clusters that see quite a bit of traffic that we have to support for these kind of golden path things and then things that mainly could be anything right um and so recreating clusters for example is
uh is is definitely a big task that Nick will talk about in a minute um so the first step here is when we're thinking about recreator our clusters um from a product perspective we need to think about a couple things one is the users of our platform so who are those people that are actually deploying the things and what are they trying to accomplish what are the trying what are the things that they're trying to get done um and then what is our impact to the business at as a platform right we provide a layer
on top of kubernetes itself um and we always want to be thinking about kind of our impact there um and yes there will be several gratuitous Alf gifts as we go through this presentation um so we always have to keep in mind these two things whenever we're building anything um and that um first starts with who our users are so this is the kind of mental model that we use or we uh well simplified I guess but this is how we can think about the that use our platform right those developers themselves and it's a
really a spectrum from everything from I don't need to care about kubernetes and I don't want to all the way to I want my own clusters right we do have folks that do require their own clusters for whatever it is that they're doing um maybe they've created an operator maybe they um you know are using a third party service and they need their own cluster so we get folks that come to us and ask us about a kind of a whole Myriad of things and we want to be able to support them the best of
we can um and having one kind of kubernetes offering obviously doesn't meet all of these um all of these people and so from uh I don't care about kubernetes at all perspective um we do have the golden paths right like that's a way that you can be in production that's a way that you don't necessarily need to care about kubernetes for people that want to do yaml things these are the people that want to fine-tune their services or you know they they want to kind of dig into things maybe they have other needs we support
them with these kind of golden path things but not as much as we would for folks that are on the golden path the problem of course is for people who want to do things like write operators or own their own clusters and we don't really support them very well right so SpongeBob here yeah we don't have a great answer for those kinds of people and that really is the kind of the the the product piece that as a PM I'm really focused on is how do we best support all of the people who use our
platform as opposed to just the people who only care about one thing um as a platform team this is always the kind of thing that is in the back of our minds what is the kind of cost is proportional to reliability right talked before about the kind of you know if we have an outage people notice um we could make something extremely reliable it would be extremely expensive we could make something extremely inexpensive but it would fall over all the time um this seems super obvious but it is really the core thing that drives a
lot of our decision-making right so is this thing that we're doing decreasing our Rel reliability in favor of cost or is this thing something that um is going to increase our reliability and how does that impact our cost and somewhere in there is a certain level of nines that we're happy to accept that you know we don't measure by the number of tweets that we get but um somewhere in there is the right level uh of cost for liability and that's something that we are always mindful of whenever we're designing something or working on our
platform um so of course um as a PM there you know there was something that sounded something like this when we decided to look at this so the idea is how do we create these differentiated clusters themselves um and ensure that we have the operational control and stability of them while keeping in mind of things like cost and reliability for example um a product manager uh obviously said this um unnamed um but this is the kind of the idea here where we're trying to um create these differentiated offerings in order to meet our users where
they're at um and ensure that we're hitting those kind of key things that we care about as platform um so going back to our kubernetes willingness and knowledge um our first kind of pass here at separating out these use cases is really along the lines of the folks that uh don't necessarily want or need to care about kubernetes um and the folks that want to do yam things um and those folks are really the ones that are looking at computer as the product right they're really looking for just raw compute to run their services or
do whatever else that they need to do on the flip side um these other folks are really looking for the Clusters themselves right they need um you know they need maybe more fine grain access to the Clusters or they need um other things that the folks as the folks that are more in the computer as a product bucket uh don't necessarily need right um and as I said before these are folks that are more advanced they're writing their own operators um they can manage their own clusters although it might not be their day jobs to
do so um and so this is kind of where we've drawn the line today there may be more gradations here right um and there's very there's really a kind of a I think the the devil's in the details of where you actually draw the line between the folks that need one thing versus the other but for us in terms of our size and our scale and our ability to manage and support our users this is kind of our our pass at it um so our current cluster offering of a single you know multi-tenant service um
definitely doesn't isn't this um and this is one of the kind of key reasons or the key reasons that we were looking at rebuilding our clusters um another piece of it or what this actually looks like in practice um is something like this so for the folks that um are doing the kind of computers product these are the people who don't necessarily need the whole cluster for example um what they have are things like name spaces on a multi-tenant cluster they could have higher levels of abstraction uh depends on you know um depends on themselves
depends on what they need um like I said before we don't really abstract away the kubernetes API at all um but we could at some point if we really wanted to lean into the kind of uh golden path or paved path um things like slas um as a platform team we guarantee things like cluster stability and reliability to a certain level um of nines for example um governance is actually a key piece here right so whenever you have these clusters where you're not abstracting away pieces um having governance on these clusters is extremely important to
ensure that you know what's on them right uh cost also a huge concern for us as I was mentioning before um and uh our Squad in in in Sp at Spotify doesn't exist within a vacuum right we have other folks that their job is to take care of cost reporting controls and things like that um and so we are controlling the Clusters from a cost perspective um and feeding that back into different parts of our organization so this is super super key for us um ensuring that we have these clusters that exist within the ecosystem
of Spotify um specialized nodes this is becoming more and more important I think I've heard AI mentioned about 4,000 times in the past hour so um we care about this as well um things like access to gpus are very important for our users um and um and and so having access to those node pools is super important things like model serving for example will have different needs than training um and those folks are all um on the same cluster sometimes um upgrades so we still do upgrades um cluster life cycle management is something that's key
um when you know you're managing clusters at scale um and for those computers of product folks it look it's fully opaque for them they don't even really necessarily need to know when the Clusters are being upgraded um and they shouldn't need to know if we're rebuilding clusters which is something that we've we've done and Nick we'll talk about in a sec for the Clusters of product people um these are folks that need that additional kind of fine layer of control so they they don't need necessarily High higher levels of of abstraction they need direct access
to the Clusters they still need things like cluster stability and reliability which is what we offer them um but they get a cluster out of the box that fits in the ecosystem of Spotify um while allowing them the control that they need um so things like managed upgrades if they're doing upgrades we help them with that um and cluster life cycle management is super important and then of course uh the flavors of clusters as I mentioned before access to gpus um so this is kind of one of the reasons that we decided to rebuild all
of our clusters to support this model um and I think the the key question that hopefully you're all thinking is how would you actually do that um and this is where I'm going to hand it over to Nick um to take over from here cool thank you Dad so once you've resigned you need to do this you know there's discrete stages of work before you can do it and most of the work is done before you do anything live uh and it's important that everything is going to be done live we as you saw our
users can't handle any downtime and uh you know we're striving to maintain that so we're going to go into each stage uh talking about first our architecture in terms of how we can actually support doing this live uh talking about our clusters themselves their shape their configuration how they're managed uh the workloads themselves and how they interact with our clusters in our ecosystem and then how we actually do migration tooling to do this live so to start with our clusters or our architecture I should say you know let's take a very simple global Service you
have it running in many regions you have a global load balancer in front of it routing to the closest client so you can have low latency um and when you think like oh I need to re I need to recreate my clusters I need to move workloads from old clusters to new clusters you know obviously the first thing that comes to mind is your networking setup um we could have chosen to to do this with you know very segmented partition networks um but we didn't think this was a good enough experience for users we thought
something could be done better something could be done within region so we decided to do this by you know running service Discovery beyond the cluster boundary uh and having headless services within the Clusters so what this means is our clusters themselves can attach to service objects um you know similar to what you would within a cluster but be routable to uh within our broader Network and this has given us a lot of flexibility so as we bring up new clusters we can deploy existing workloads on running clusters to new ones and still Route traffic uh
and this has been a game changer for us in terms of uh actually being able to support something like this so when it comes to preparing your clusters obviously uh you know Dan mentioned a lot about the golden path but it's not just the golden path you have to support it's everything else um you know there's a there's over 3,000 microservices and frankly it's impossible to know them all so there's certain things you have to do as a platform engineering uh organization to protect yourself so that starts with you know what does your clusters actually
support you know and it starts with you know your apis and your kinds available assume it's used on if it's one on one cluster assume it's used on all of them if an accelerator or machine type is available on one cluster assume it's used on all of them and more importantly uh your ecosystem of tooling for your platform or needs to understand that all of it has to work across the cluster boundary when you deploy your workloads it needs to be cluster di agnostic it needs to work across any of them and your monitoring alerting
needs to not be tied to the cluster itself it needs to have a strong level of continuity um and on top of that uh everything else that you know a lot of best practices we talk about in the industry basically become table Stakes uh things like infrastructure is code or automated cluster life cycles that's just an expectation here not a not a nice to have so we have homogeneous clusters now right you know they're nice equal boxes shout out chat GPT for this one uh and our services are available in any number of the boxes
but now we have the problem of fitting them in we have to put them places they need to be somewhat consistent they need to be reliable they need to be in a healthy state so the first thing you have to ask yourself when you start thinking about your workloads and the cluster ecosystem that they live in is do you know where your workload are because if you don't you're going to need something like a software catalog you're going to need a source of truth you're going to need uh you're going to need to get a
lot of information from these Services it's impossible for us as a platform engineering team to actually know enough about all of our services and all of their unique bespoke requirements because they're all going to find different ways to do different things that are you're going to have to support and backstage is sort of the key uh the key piece here to actually having a source of of Truth to dive into these workloads um so that you can eventually support their migration um but past that get getting enough information about your workloads you have to actually
optimize them for the Clusters uh and we're solving that in three ways so as Dan mentioned you know we want to give users you know our developers the best possible chance uh at component creation and that's you know the golden path templates uh and that's sort of the best sort of configuration for their you know for their coding language and their framework and that's the best kubernetes configuration possible but unfortunate unfortunately services are long living uh you know many of ours you know predate as far back as spotify's existence and automated improvements need to happen
constantly uh spotify's made really significant investments in this in a program we called Fleet Management um this is where we're shipping code and configuration changes automatically to uh all of our components constantly uh and I think we've shipped over 8.8 million lines of code and configuration changes to all of our repos so think about every kubernetes version upgrade you've done that's changed the yaml spec just slightly well we don't want all of our Engineers to have to understand that to take that cognitive load we want to handle that for them and that's where Fleet Management
comes in so think changing HPA Behavior or deprecating some other API we we can handle this for them because we can we can what we can Fleet manage it and for everything else There's caverno You Know Dan mentioned governance uh cerno has been a key part of that acting as our our governing source of consistency across the cluster but also acting as uh an easy way to handle Dynamic environment injection so it has really strong uh abilities with admission mutation web Hooks and this lets us add you know dynamic um content or dynamic hydration to
all of our users workloads without them having to know you know what cluster they're running on which region they're running in so so far we've optimized our workloads um we've built an architecture that actually supports running workloads across the cluster boundary with an ecosystem of tooling as well um but unfortunately you know the final step is actually moving workloads in a live setting you know people are streaming uh music and podcasts and audio books now and uh we can't stop that we have to do this live so the important thing when planning the migrations and
preparing them is understanding that you know the migrations are going to break break toolings are going to break tooling is going to break but the workloads cannot um and you have to cover all the bases you know humans are going to make mistakes and your automation is going to have bugs it's it's just inevitable so the key thing is that your migration has to be done safely and uh we sort of plan this it with sort of the main principle being admiten when you're doing a migration in a live setting so the first stage obviously
of doing a migration for one workload and then applied it scale is validation is the is it actually safe to start it for this workload can you move this across the cluster boundary um is it in a healthy state has the input been is it valid um and what does this workload need um the next step is pinning the workload I know this sounds simple but you have developers working across several time zones they're shipping features in code we're also automating lots of code changes and we need to ensure sort of a consistent environment uh
we want two versions of code running in two places with an inconsistent experience um you know as we're constantly launching new features and constantly shipping more code um The Next Step being double deploys so we're deploying to an old and a new cluster or potentially multiple clusters um and this is where you know Dan mentioned cost versus reliability this is an explicit trade-off we're making here there's other ways we could have gone about this so that we don't have to double deploy workloads which can be expensive at scale but this is something we believe is
is super valuable for ensuring uh you know quality of service uh and the next step sort of Builds on that which is traffic shifting so you know graceful traffic shifting gives you a lot of things but most importantly your users have no idea that you know maybe their connection is moving across a cluster or across a region um finally now we can reap the rewards of our you know excess cost for reliability and we can clean up and finally we can unpin so we can allow sort of the Spotify machine of automated changes and developers
to constantly ship new features to our users completely seamlessly so we've done a lot to get here by now we have a network architecture that supports multicluster Services we have a completely homogeneous set of clusters and a cluster diagnostic ecosystem of tooling our workloads are as healthy as possible to support this in a live setting without without a sort of human interaction and we've created a strategy on how to migrate in a in a live production setting that can fail at any stage so putting all that together you can kind of piece together what our
principles are but we can sum it down as these two the first one being take the pain uh we don't believe our users should really incur you know the they shouldn't feel any pain for a decision that we made as a platform engineering team we chose to recreate our clusters because we believe it gives us a lot of value you know but they just want to build features and and bring value to the business so we're going to take that for them uh which leads to the zero developer interaction you know with 3,000 microservices and
3,000 Engineers uh it can't be expected for all of them to interact just that's not how it's going to work for us it's kale so you know we we believe the the users almost shouldn't even really know where they're being deployed and if we've done sort of our migration as as best as possible they wouldn't even notice they're being migrated um but we'll let them know on Backstage usually and with that not a beat was skipped [Applause] yeah so we're happy to take questions here um also would love your feedback or anything else so just
hit the QR code there's also some links in there to learn more about things like Fleet shift or that fun incident that we had last January um so yeah I think we have a little bit of time here um if you have some questions feel free to on one of the two mics here um and uh yeah hello oh that's a lot of people hello um in terms of managing that many clusters at that amount of scale and you having users who don't really want to give a about like kubernetes and all of their stuff
uh how do you make sure that those developers who are owning those applications feel like they actually own them and if they break then they are responsible for fixing them or are you guys responsible for kind of fixing stuff and what's that sort of setup and culture bit like so we we're a really big believer in Ops in squads so the squads themselves still own all the operations of their workloads um but it depends really on the outage right if if we're doing a big change to the underlying Fleet and we break their workloads you
know we're going to have an automated way to reverse it or to get them back into a healthy State uh and a large part of that too is also the Integrations with backstage that we're building to support all of our users so in the best case users don't even they they don't even have to actually directly talk with us to fix their issues uh they can likely fix it through backstage and if not then you know we have all the normal on call goalie style support uh hi sorry we'll we'll just go to one I
think there's another mic over there yeah go for it thanks could you just elaborate a little bit on your use of kerno and what you guys are actually doing with it sure uh we use caverno for the two the two main things for us are uh it's as a sort of a generic admission mutating web hook so you know injecting uh environment or cluster specific variables that workloads may need to operate within their region obviously we operate in 180 markets so that means a lot of different things around legality so workloads often need to be
aware of the content they're serving and where they're serving it from so injecting things like that is one thing we use for Cal for that type of configuration uh another thing is sort of ensuring other types of like best practices on the workloads themselves so our workload owners get a lot of control around uh editing their workload but say we want to enforce a best practice like your memory request equals your memory limit you know we can set that as a cal policy to ensure or n dots equals 1 for example yeah yeah any uh
how did you uh achieve the uh Ser developer interactions in the migration CU that seems like a bit of a thing to do especially given the demands on the engineering teams and how much you need to work than that uh zero interaction was kind of done a few ways but the the big thing is that you know we we have uh a lot of platform sources on where workloads are running where they need to run like what regions they need to run in um we have access to the repos we have you know tooling around
triggering like API driven deployments so for us it's really with like the whole ecosystem put together you can you know you can start to build automation to actually codify that uh and then with backstage is sort of your central source to tell users where like what's what's happening to their workload thank you all right we'll head it back over here thank you very much I have a question about your netwalking setup you talked about headl Services how did you implemented the multicluster networking uh as you did as you said you know thank you so I
can tell you our services exist beyond the cluster boundary unfortunately the rest of our Network architecture I'm not allowed to talk about sorry about that okay I have the same issue so too bad all right back over here hey thank you for the talk really interesting I had a question about the cross cluster Communications um you said you were using some kind of service registry where did you put the those Registries inside the Clusters or somewhere in your region so the service Discovery happens outside of the cluster okay yeah so then multiple clusters outside of
multiple clusters within a region can register their services to something external to the Clusters okay thank you and there's there's another Spotify talk today Alex and Yanik who work on the squad that deals with that most closely so you can ask them too and they might give you a better answer then okay thank you just kidding but thanks for your talk um I had a quick question regarding data stores and how you all actually manage to whether any of your data stores were deployed in kubernetes or if they are externally based did you recreate them
as well uh and migrate them to a more golden path you want to take pH search maybe kind yeah I think one use case in particular was um our use of vesa um so that's something that was not historically on kubernetes but was a use case that we were looking to support with this kind of cluster as a product versus the computer as a product model um and so that is something that we now support search for example um in its own own kind of um cluster as a product offering um I'll say um I'm
not sure if that answers your question you're talking about other kinds of data stores but but that is one in particular that we're using this model to address got it but are the remaining data stores not deployed on kubernetes and you'll have it Standalone within your infrastructure we do we do use quite a bit of managed uh data stores so in that case you know it's managed right so we don't need to deal with that um but yeah so yeah maybe we can talk offline maybe more about about specifics there but yeah hey thanks for
the talk um I had a question about the first step that was like validation of the of the workload does that actually ever require like knowledge about the workload that's running and therefore require like interacting with the devs um yeah yeah so you know as I was saying with like the Ops and Squad ODS we have to make certain assumptions about what we can do safely so if we you know do a validation on their service and we don't believe it's healthy to migrate that's immediately a signal for us that okay this is an exception
to the rule and we need to reach out to a team to get their their service in a healthy State before we can do a migration so the thing is before we do any changes sort of to their production workload uh and actually start it you know migrating it we need to make sure it's good so that's that's our first indicator really it's the validation step thanks thanks hello uh I'm just interested if you could give us in um insights about the infrastructure is being used uh for the these kubernetes clusters uh do you have
like bare metal or do you on on virtualized uh environments uh and the other question is for which strategy did you decide for uh life cycle management for the for these uh long running clusters do for example node replacement or uh how does how does it work yeah so um first question I think we're a pretty big gcp customer so yeah we don't have anything on Prem um so I think that answers the first question then the second question is yeah we're we're we're tied to obviously some upgrade Cycles uh via gke um and uh
and so that's kind of the the process that we follow for upgrading things um as I mentioned the talk we have clusters that I think we've had for I don't know five years for example and we've constantly upgraded them um yeah and uh so it's a little bit of a different different model than maybe um the kind of ephemeral cattle approach um and uh and yeah so our clusters are super super long living this model is a little bit changing that of course um as we get demands for things like like I was saying ephemeral
clusters for testing or whatever else um but yeah for the most part our our clusters today are are long super long living so for for some of the some of the life cycle things as well a lot of it's dictated on what upgrade you're doing so say for example like the docker to container d uh deprecation you know that's that's a full node pool replacement uh or maybe you know avoiding the c groups V2 bug that's another one where you have to replace the whole node pool again uh and then I think obviously a lot
of that's going to be dictated by how sort of node pool upgrades are done by the platform you're using yeah so maybe we'll go back over here yeah yeah so I had a question about um and managing Network policies so um uh when it comes to sharing microservices uh what is your approach for um configuring the network policies per workload like uh how what what is your approach for um yeah sh uh creating these Network policies is that a responsib of the workload teams or is that is that your responsibility uh I can't talk about
too much of this because I think security teams would get mad at me but uh yeah it's it's done Ser like it's it's within the networking plane I can say okay great and and Alex here said we could ping these two they have a talk later today they'll be able to answer your questions better than I can on the networking side okay great thanks yeah back over thanks um have you encountered a scenario when one of these squads move across the spectrum of willingness uh or care about kubernetes and how complicated would be to support
such a scenario yeah I think from from our perspective right if they're staying within those kind of use case product boundaries then it doesn't really matter but if they're for example um they've decided to entirely shift and have like move from one to the other it would just be another user of that other type right um we want to productize these things as much as possible and not have kind of snowflake configuration for any one supported use case um and I think that's where the kind of level of abstraction comes in right so for the
computers the product folks they they do have access to a namespace so for example if they wanted to do more yam things like I have in the slide they could do that right if they started off as a golden path service but they needed to configure or optimize their service they can go ahead and do that um and that's kind of what I was alluding to in the presentation around like where you're drawing the lines between the different things is very important um because it's going to dictate how you can support and how you interface
with your customers thanks hello thank you for the great talk uh I have a question regarding continuous delivery tooling you use for multicluster deployment and for migration between clust uh yeah we use a couple tools uh a bunch of them are in-house unfortunately uh and then we use like you know a lot of the bunch of Open Source things like config sync and Argo CD with a bunch of you know orchestration type apis to then handle you know the different cluster to Cluster assignments and things like that thank you yeah two more questions I think
we have a couple more minutes but yeah go for thanks a lot uh I just got a question question regarding the double deployment and traffic uh shifting are you using some specific load balancer during the the migration how do you manage that to go from a full first version to the the second one and with continuous traffic do you have some yeah balancing or percent percentage that you can set or something like that yeah so our workloads are all configured with you know graceful connection termination to start and because workloads can then connect to the
same service object across a cluster across multiple clusters we're not doing any special uh special load balancing per se and instead of just putting more things behind the service object and letting that Round Rob in and then by enforcing scaling in the two directions you can you know you can shift traffic okay thank you all right there one two two more question uh thank you uh well my question maybe partially of topic but I'm curious uh I have seen your your slide where you said that you have 1.5 terabytes not even terab but terabytes per
second of f trffic uh but normally as far as I know the encapsulation uh the encapsulation algorithms for bots in kubernetes like vxlan genev and so on they have a limitation in the in the broadman sometimes it's 1 GB per second in so how do you deal with this bottleneck is are you using a special encapsulation or I think with half a million Pods at Peak it's it's just spread out across them it's just such a large amount of containers across so many thousands of VMS that we haven't run into individual node limits all right
thank you yeah last question hi um my question goes towards the cluster as a product uh offering um you mentioned you you as your team is responsible for the stability and the Rel liability of the offering but still you probably hand out cluster admin or more or less cluster admin to your user how do you where do you draw the line there or how how how does this work out for you yeah so I mean so far so good I guess um it's something that we're we're definitely keeping an eye on I think the folks
that fall in that boundary are very knowledgeable about kubernets anyway right which is why they want their own cluster so there is a fair amount of trust there um and when it comes to doing things like upgrades it is a little bit more Hands-On for us right then fully opaque in the case of the computers product um so a lot of the time we view that that as kind of a partnership right um we'll see as theoretically more people move from one to the other how scalable that is um but so far it's something that
we've been okay with um but I but yeah it's a great question because I think it's something that we're not worried about per se but it's definitely something that's on our mind right how much is too much uh support there um of course you know we're also a small team um and yeah that's that's that's our concern but it's a great question because it's definitely something we're thinking about thank you thank you uh yeah I don't think there's any other questions here so uh we will um happy to chat offline or outside of the room
as well again if you have any questions feedback comments or you just want us to uh to chat via email feel free to hit that QR code and um and uh yeah we'll get back to you but thank you everyone thanks everyone
