hi thanks for joining this session i'm mark groper senior product manager for amazon dynamodb and i'm thrilled to announce this session speaker attilio jouer director of content discovery for disney plus i had the privilege of working with attilio as disney built scaled and globally launched disney plus and i've got to tell you this was no small feat as attilio will describe in today's session but one that has been very successful both technologically and commercially and on a personal note i want to extend my thanks to atelio and the disney plus team for launching a critical
service for my own family's mental well-being over the past several months attilio over to you thanks mark this is the first time i've worn a college shirt in eight months so i'm very excited to be here and also glad we can help with a little bit of distracting entertainment just to give you a little bit of background on myself i've been working for disney for a few years now after they acquired the company called bam take bamtech media which was spun out of major league baseball advanced media all told i've been working in various forms
of this company for about 12 years in the beginning with baseball statistics and ticketing systems later when the company branched out into providing streaming products for third parties businesses i worked on things like wwe network hbo now fox sports go among other things and now as part of disney one of the things i'm working on is disney plus which is what i'm going to talk about today so enough about me i'm going to talk a little bit about what disney plus is what content discovery does for it then i'm going to talk about some of
our use cases for dynamodb and then i'm going to wrap it up with some basic learnings and other tech takeaways that we found from launching and operating disney plus for a year now this is just the obligatory introductory introductory slide and then let's go right into what is disney plus so it's a video streaming service that hosts a lot of disney's most popular brands which you can see here behind this giant disney plus logo you have pixar marvel star wars national geographic in addition to just disney branded content right so i'm sure most people have
heard of at least one of those things besides disney and you see in this image also it depicts the top of the home page of the application while that looks pretty straightforward there's actually a lot of services both in the foreground in the background that support rendering this home page and we're going to talk a little bit of about a few of those today content discovery team is teams are the ones that i manage they're responsible for apis that serve the content metadata for the disney plus application so that's basically the metadata around the videos
themselves right so everything that you see in this sample of images is powered by one of our apis including the home page and other brand landing pages so you click on marvel and you see essentially a home page for marvel recommendations and trending content on the home page we're going to talk a little more about recommendations later continue or resume watching functionality on the home page series and movie detail pages we'll talk about that later in relation to bookmarks which i'll explain about the explore page is powered by our site search service so that's you
go in and you search for frozen uh you'll you'll see stuff that's related to frozen watch list pages right so that's where you can add items that you are interested in watching maybe not now so you'll you'll add them to your watch list you go into that page you can watch from there i'll talk about watch list a little more and all the images themselves are served out of our image apis so as you can see like all of these services combined together a pretty essential set for the disney plus experience let's talk a little
bit about our scale so november 12 2019 was the day that we launched disney plus in the us and canada we had been running a technical test in the netherlands for a few months up until that point the traffic on launch day was much higher than anticipated to say the least with over 3 billion requests to our content apis part of the reasoning for that you can see the character there in the upper right corner is known as the child this is a pretty popular character in a disney plus original series called the mandalorian which
is part of star wars brand uh probably mention the mandalorian a few times more since launch we've expanded to over 30 countries with more coming soon 30 countries and territories and are delivering tens of terabytes of content metadata a day hundreds of terabytes of images a day hundreds of millions of recommendations a day are delivered by our machine learning team over kinesis and inserted into dynamodb and billions of bookmarks ingested a day over kinesis and into dynamodb and talk a little bit about bookmarks and what those are in a little bit but yeah let's go
on to the next one here just a final note about what what our stacks look like we're in many regions with pretty much identical stacks in europe and north america where most of our customers are today we use this multi-region setup for all the obvious reasons uh failover right and regional issues could cause us to decide to just remove a region from our ralph 53 and just divert all the traffic to a different region and also reducing latencies around 53 behind cloudfront is routing traffic to the closest origin right so there's that aspect of it
but ultimately what we're trying to do is make that guest experience of using disney plus as magical as possible right it's magical as one of the words we use a lot of in disney so moving on we're going to cover the dynamodb use cases first and then go into some learnings afterwards and some tips so one of our first use cases is pretty basic it's called watch list basic concept for streaming service you're looking around the catalog you're browsing around the catalog you see something that you want to watch in our case you click on
a plus sign it gets added to your watch list uh you then like i mentioned before you go to the watchlist screen you see your latest watch list items you can start watching them from there when you're done if you remember you can remove them from your watch list and our architecture for watch list is backed by a global table it's also equally simple which you know i highly recommend simplicity definitely helps when you're reasoning about systems and bugging them but you can see here we have a service that's in front of a global dynamodb
table which lets you query thing do query against it do things like check if a particular series or video is on a watch list get the latest set of watch list items obviously add and remove stuff from your watch list there's really nothing too complex here or too special going on here other than the the table itself helps us you know keep those watch list entries in sync across all the regions which we're deployed in with very low latency right so if we had to whatever for whatever reason fail over to a different region no
concerns there about a stale watch list one of our next use cases is called is around bookmarks mentioned it before basic concept for video streaming service you start watching a video you pause it you pick it up later where you left off or you start watching it on your phone and you want to continue watching it on your tv if you know you still happen to own a tv you're watching a series and you want to continue watching it with the next episode serie of the series when that's available these are all your basic user
stories and they're handled or with the use of bookmarks right which keep track of where you are in a stream for a particular piece of content why it's called the bookmark i have no idea these are clearly not books but that's how we refer to them i guess video marks would sound kind of weird here we have the bookmarks architecture right so as you're watching a video the app itself is sending a stream of bookmarked bookmark data to a service a telemetry service and the nearest aws region to the video player all right that service
takes that bookmark data writes it to a kinesis stream and then we read that data from the kinesis stream and insert it into a global dynamodb table that exists in the regions where our content api is deployed to right so clients then request that bookmark data from one of our content api services when they load up home page or series or movie page and having the architecture broken out like this allows us to decouple where we're reading the bookmarks from uh to where we're actually you know serving the clients with the bookmark data so we
could deploy to any number of regions and just by adding you know that region to the global dynamodb table and the telemetry team can deploy to whatever regions they think are best for them all right so that actually opens up a lot of flexibility for us recommendations is another seemingly basic concept that we offer on disney plus that utilizes global tables right so you can see some examples here recommended for you on the home page based on stuff that you've watched before or other signals we recommend you watch this other content right because you watched
a particular piece of content here's some other things you might like is also another use case there so our architecture for that is a little more complex but still fairly straightforward we have a machine learning team they generate recommendations and write them to a kinesis stream in a single region we read those recommendations from kinesis we do some work on them and then we place them into a dynamodb global table from there again we serve them up from our application in whatever region the user is in so this allows the machine learning team to not
have to worry about delivering recommendations to what they believe is the aws region that the client will be closest to they can just focus on creating the best recommendations that they can and delivering them to us and then we'll figure out how to deliver them to a client we don't have to read the recommendation stream in every region and write to like a local dynamodb table so it reduces a lot of overhead from us we just read and write it once and let dynamodb do the replication and that really simplifies everything for us and the
final use case is pretty non-standard it's around content caching but we felt like we should share this use case uh with everyone because it did help us around launch so dynamodb is not the source of truth for our content catalog for that we use a different document datastore but we do use dynamodb to cache the results of queries that we run against the document datastore and we cache it with the ttl so the basic pattern there is somebody will query for some piece of content or some set of content we'll check this dynamodb cache if
it's missing or it's expired based on the ttl we'll pull it from the primary data store and we'll put it in the cache serve it out from there this helps us buffer requests to our primary data stores any sort of cash would but you know we considered using a standard in memory cash service you know which would be typical before the u.s launch but we ultimately chose dynamodb because of a couple of reasons but mainly around there was a lot of uncertainty in terms of the number of subscriptions and traffic estimates that we were going
to get on day one so made it very difficult for us to analyze like instant sizing that we would want to do with an in-memory cache but with dynamodb and it's sort of extremely low operational and administrative overhead we were able to set it to an on-demand mode with some pre-partitioning which i can talk about in a little bit it helped us quite a bit in taking the pressure off the main data store and to scale to the levels that we needed to on day one of our launch in the us and canada all right
so now i'm gonna those are the basic use cases and i'll talk a little bit about the takeaways that we've we've had since launch so some of the ways that dynamodb has benefit benefited us uh global tables specifically um when we evac we we evacuate regions i mentioned before like for for certain things like failover right there's issues and we need to you know move to a different region to maintain you know the reliability and the the performance of our service or we may be doing some planned maintenance that requires downtime so we may shift
resources to a different region or we simply just want to test out being able to remove a region and make sure that everything continues to operate without degradation uh so the replication that dynamo offers us with latencies that are in very you know low digit seconds allows us to do that sort of traffic shifting without really worrying about data inconsistencies or anything like that or even scaling concerns in addition as disney plus continues to grow and we launch in new countries right being able to leverage an additional aws region just by adding it to the
dynamodb global table right gives us an enormous amount of flexibility in terms of being able to just stand up our stack in that region the table's there the data is there we're able to just function pretty simply then scaling benefits we we continue to grow our user base and the number of recommendations and bookmarks continue to grow right so dynamodb grows along with that user base so with very little operational overhead on our part i think that's a big theme the operational overhead the burden of managing dynamo is very low which is excellent for us
and then finally having the ability to switch back and forth between this on-demand mode or a provision mode really helps us when we launch in a new set of countries where we don't necessarily know what the volume of requests what the traffic's going to be like on day one so having that ability to do that very easily is is pretty awesome for us the next two things are you know basically in more in the line of tips right so one of them is around pre-partitioning i mentioned that before um so dynamodb partitions data as data
storage grows as uh throughput grows uh the number of partitions grow as well in order to maintain a high level of throughput so on the november launch we expected a pretty large influx of bookmark data which basically into like an empty table which if it wasn't partitioned to meet the demands of that scale we would have experienced throttles so in order to meet that anticipated demand of read and write through put without throttles we decided to pre-partition our tables prior to launch so the way we did that is we set a very high provisioned throughput
right value and let dynamodb do its partitioning based on that and then we switch back to an on-demand mode prior to launch in order to scale higher if we needed it and using a combination of those two things we were able to avoid any sort of throttles on those tables that we did that too and then finally there's a tip here around concentrated read traffic supporting that right so an item lives in a partition which has its own limits on reading and writing in our content cache which i mentioned earlier some content is more popular
than others and can result in disproportionate lookups to a partition which then results in throttling right so to solve that we employed a strategy of appending a sequence number to the end of the key and writing that data to more than one entry right so on a request for a piece of content we would use a guide that was on the request and we would hash that to one of those sequence numbers and then we would look up that key so as an example here around content id abc123 we would map it to 1 or
two or dot n of that and distribute the read traffic amongst those entries uh keep in mind you can still use a secondary caching layer in front of this to reduce the cost and further reduce the latency but as is the case with caches sometimes they're unavailable or sometimes they start out cold we're pretty confident we know that dynamo will be able to support the full brunt of the throughput that we require of these tables if those cases occur so hopefully you're able to get a couple of tips or best practices from this presentation that
will help you with your own aws dynamodb needs thank you for listening certainly if you found any of that interesting to you you want to learn more about the opportunities that are available with disney visit disneytech.com and if you found if you're interested in learning more about dynamodb here are some other sessions led by some great subject matter experts too on data modeling and two on advanced design platform patterns and yeah thank you very much for listening
