um so this talk is called dream this dream a high velocity event processing with the converged database um it's it's a talk that's a mix of some high level Concepts and a little bit of marketing slides um and some tech mixed in um I'm happy to talk with anybody after this uh this call or even during about deeper under the hood kind of technology advancements in this space okay uh let me go forward okay so here's the agenda um I want to just give an overview of what is event stream processing um and then I
kind of shift a little bit in terms of converge databases versus specialized databases and we'll talk more about this but this is the idea of having a specialized database just for event stream processing or um you know any other type of workload graph spatial there's lots of specialized databases versus a what we what we call a converge database which is a database that handles all different types of workloads um so we'll talk about that very quickly and then I'm going to talk about um in more detail what an event stream processing database needs um and
all the requirements and and how in this case Oracle meets them with that with ITS Technologies and we'll we'll go into that then time permitting uh we have I have some demo slides that go through an example of how we use the database for an event stream processing solution okay feel free to interrupt me anytime you have a question by the way okay so what is event stream processing so event stream processing really has three components one is continuous ingestion we're going to be ingesting high frequency event data right rapidly bringing it into your system
um the second thing is real-time analytics so what do you do with that data so you want to be able to process that data in real time if if this data is coming from a fleet of servers that's indicating uh the temperature you need to know right away if any of those servers exceeding exceeds some temperature threshold because you want to be able to resolve that so any customers writing their applications on that server aren't impacted and then the last part is data reorganization and here the idea is because you've ingested so much data that
over time you're not able to keep that data um either in memory or Flash or whatever you want to be able to move it off to cooler storage and you want to be able to compress it or and or summarize it so that when the data is at rest you can still uh be able to store as much data as possible okay so that's kind of the high level what is event stream processing now what about the data itself what kind of data do we receive so normally each event sounds like a discrete data record
and it comes from like a large form of data sources and those sources could be a variety of different things right it could be like just devices sensors meters like your electric meter or gas meter servers if it's a devops monitoring scenario desktops or smartphones for that matter right so a variety difference uh devices think of the internet of things kind of scenario and it typically consists of the following information so if you look at these three different examples on the left side you've got a media reading and there's always going to be a time
stamp there's an ID associated with the device where it came from and then there's a reading and the reading itself is you know the value associated with that metric as well as some attributes so in this case the reading just shows all of the values like electricity used was 0.4 kilowatts per hours five gallons were used uh two gas therms are used so this is an example of a meter in the middle you have a phone again timestamp and an ID and simulate the times of attributes are like the GPS location of it the battery
level right and the same thing on the right hand side with cars you got you know GPS location how fast was it going at this time what direction was it going and so on okay so that's like the event data um now the properties of event data are as I mentioned a very high arrival rate okay we're receiving tons of data coming from different sources and one example is like a Billing System that's receiving millions of smart media readings every few minutes um the second thing is this High obsolescence rate so what that means is
when the events come in they're hot and they're the most recent data and you want to do reports directly on it and now the older events are going to be used more for like historical reporting that says something like hey in the last quarter what was my average uh temperature reading for example okay and then as I mentioned these events then do get compressed and summarized as the data gets older and older and older so if you look at the picture on the right you know you see recent events going down to older events and
eventually they get archived okay okay so now what are the requirements of the event stream processing system so number one is in our view we want a flexible data model and the reason here is that you are going to receive events from a variety of different data sources and those events could have a different representation may have a lot more attributes than other events and you're building a a data store engine for this you don't want to have to keep changing your schema to accommodate for new attributes and so as we'll discuss for a flexible
data model we like to use Json and Json is schema-less it allows me to change things on the fly as needed the next things I mentioned is high speed ingestion that's a requirement you're going to have to be able to sustain billions of events per day so whatever system you build needs to be able to scale out and parallelize and be able to ingest rapidly the third thing is whatever system you build has to have a rich analytic query capability as I mentioned you're doing real-time analytics on this but what kind of analytics are you
doing um it's not just simple aggregation and counts and predicate evaluation you want to do prediction and you want to run ml against it you want to be able to do things that are more complicated um as you're streaming this data in the fourth thing here is this real-time analytics capability I want to be able to respond immediately and be data driven um and so that generally requires me to run at in-memory processing speeds right I don't want to be bounded by IO I want to be bounded by CPU and potentially bounded by by memory
as well and the last thing is you need to have some kind of life cycle management built into your system you want to be able to do automatic data compression automatic summarization of your data automatic archiving after some period of time of unbounded growth okay any questions so far on this okay so now let's look at this converge versus specialized database story so there's an increasing complexity due to Modern apps modern apps basically use a variety of different data Technologies we have lots of new data types text and graph and spatial documents a lot of
new types of analytics that are happening machine learning social graph you know networks and then different workload types right so because of these different Technologies it's it's making Building Systems more complicated and that complexity then introduces more problems like more bugs have to deal with security concerns how do you deal with upgrades if one system needs to be upgraded how do you deal with interfacing with another system so it brings more complexity to the systems that we're building so now one approach is to build a specialized database for each application and each specialized database can
excel at that app's requirement but they don't it's like a one-trick pony as they say it's only good at one thing so you see all of these different specialized databases that are built for these different types of data analytics and workload types and we believe that the specialized database has a lot of problems associated with it so number one it requires that the application architecture that you're building is heterogeneous and distributed you're working with a variety of different systems built by different people that some of it might be open source some of it may have
some support associated with it um and if a problem goes wrong uh right it's like very it brings it brings unnecessary complexity to the entire system so a lot of moving Parts there and your data becomes fragmented because some of it might live on one system then it has to be transferred over to another system um and there's another big problem is that these different specialized databases have limited asset consistency right you might have a very different asset model on one system versus the other and so this is why we feel like the the right
approach is a converged database so I'll talk about that in just one second if you look at these different event stream processing Solutions this is just for event stream processing influx DB and your Prometheus and time scale and so on so lots of different systems built for one particular workload okay so what we have and what we believe in is a converged database Oracle is a converged database and that it supports all the modern data types and your workloads in one product it just allows you to have different workload types and different data types but
all fit along the same components so if you look on the picture on the left you know Json and graph and Tech all of that is supported under one converged database and it's got a lot of benefits right you don't have to worry about a unified security across all these systems you don't have to worry about data transfer from one system to another and so on and a good analogy is like a smartphone okay in the past you had a separate phone you had a separate camera a separate video record a separate GPS separate music
device most of you guys are probably only known about smartphones but before smartphones you had all of these other different devices and now everything is fit in a smartphone and so that's kind of a great analogy it all works on just that phone you don't need to have all these other separate devices okay so that's our model uh we believe in this converged database to be able to handle all these workloads so now going back to an event stream processing database and what does it need so let's go to each one of those so flexible
data model so we start off with Json okay and why do we think that Json is necessary so if you look at this event stream data again is is highly dynamic as I mentioned the formats can change um instead of your reading values can change over time the attributes Associated can change over time so we like to have a single schema like the one on the bottom left where you have essentially a sequence number that identifies the row you have a meter ID you have a timestamp and then you have the Json document describing the
reading and if that changes over time let's say the device that's sending let's say you upgrade the device to device 2.0 and now the readings are more sophisticated more detail your data engine underneath doesn't have to change right now going back to Json really quickly the problem with Json which we'll address is Json is going to be slower right than a relational column if I'm doing an aggregate or a predicate on this Json document it's going to be slower I'm basically doing like some kind of string parsing I have to go to this document and
fetch out the attribute so we'll talk about how we make that faster later okay another thing that you need okay for for this requirement is if an easy capability of communicating between the device and your database and this is where rest comes into place it's the ideal protocol We Believe for clients to connect with the underlying storage engine so you want to connect through rest endpoints so here you get your lightweight clients that are uh you know sending out a URI that has um you know attributes associated with it that you then map and bind
and then generate SQL in this rest server process that generates SQL to the underlying database and that does the put if you will or the get if you will in the database then responds back because it's got SQL results it needs to send that back into Json so the SQL result comes out of the database into the server process that handles the rest endpoint and transforms that into Json and then Json gets sent back to the to the lightweight client okay simplifies and standardizes apis to access data we want that simplification so that any device
can connect to your storage engine okay the next thing is high speed ingestion so what we'll talk about now is an oracle specific technology that makes ingestion a lot faster so we have a feature called mem optimize row store which brings fast ingestion into the database tier and the idea here is we want to insert directly into memory and once you insert directly into memory you respond back saying I'm done so let's look at this picture here on the right so you've got these events coming from these sensors and and different devices and it's going
to stream those events into an application server right that's kind of like in the mid-tier area and on the right side you see the database on the bottom as that disk picture and you see the memory associated with that database SGA stands for system Global area which is just the memory that's used that's shared between different processes on a database and you want those events to stream and be ingested directly into memory into a pool of memory that we call the large pool and once it gets buffered it gets buffered into that memory in batches
essentially and after it gets buffered there it immediately returns back to the application server and back to the device saying you're done it's been ingested the reality though is it hasn't been persisted it's not in the database itself on disk and so there's a risk associated with losing that data if your database goes down so I'll talk about that in a second but the main thing is you ingest the data very quickly into into memory and you come back out telling the device oh okay send me your next batch now what happens is we'll have
drainer processes that are running in the background that will constantly look at these batches of events and flush them to disk okay so that's happening periodically as the buffers get full now what happens if the database goes down okay if the database goes down we have an API that allows the device to developers to check to see if those inserts have been durableized so you the developer of your device can periodically check and say hey have you written up to this point yes or no and if the API returns back saying no that tells you
okay something happened I will go back and re-insert those those events okay so that's kind of end to end this is what allows us to get to really rapid data ingestion and we're just being optimistic about it that hey most likely your server will not go down but if it goes down you can check and see what's going on okay um I'm not going to go into too much detail about this because most database databases have this but it requires scale out this is just basically scaling out your database so here you see on the
picture on the left I don't just have one instance I have two instances on something that we call Rack or real application clusters and rack allows us to scale out your processing across multiple nodes or multiple servers and with Oracle all of these rack instances share a a common pool of storage so Azure data set grows you want to be able to scale out and say okay let me add a new server to my cluster because I got a lot more devices bringing their events coming in so I can scale out performance with rack I
can scale out fault tolerance so if an instance goes down no problem I can connect to another instance and because the show storage is shared that instance can be able to still retrieve all the data sitting on that shared storage okay um that's the main point I want to make about that another thing now along the lines of fast ingestion is partitioning you want to be able to partition your data as it's coming in because partitioning allows you to parallelize data ingestion so let's look at this example on the left at the very top you
got this temperature reading right there's some meter that you have or some sensor that's giving you temperature readings and on the y-axis you have interval partitions which means I want to build a partition of your data from 6 pm to now my my hottest partition um my next hottest partition was my previous partition that went from 12 pm to 6 p.m and before that was 6 a.m to 12 p.m and so I'm building these partitions of data every six hours and once the six hours completes I freeze that partition and I open up a new
Partition for data to come in so that's on the y-axis on the x-axis is going to be sub partitioning so I want to now partition with sub partition within each partition based off of my sensor ID so I don't want all of the sensors to go to one area within the partition I want to distribute that workload across the partition so by being able to partition and sub partition I'm able to move these events into different sub partitions at the same time and that allows me to parallelize data ingestion over time and by Source ID
okay hopefully that makes sense it avoids contention of space allocation it's very important whenever you're dealing with scalability problems again in real world database you know data data storage engines um you want to be able to avoid contention on either space or time and the way of doing that is kind of partitioning your workload uh now this is taking it to the next level so we talked about partitioning we talked about scale out by being how adding multiple instances this is now called sharding and sharding is doing sort of geo geo uh scaling if you
will meaning I want to partition based off my my geography so if my events are coming from all over the world I want to keep the events that are in Asia in Asia I want to keep the events that are coming from North America in North America and basically I want to have a single global view of a database but in each one of these regions they should be separate databases so it's sharding is one giant database divided into several smaller databases and this is super important again for scalability and availability issues to avoid that
you want to be able to have separate charts okay um the rest of this is kind of more details here in terms of what I just said but that's the the idea okay any questions so far again on on that if you do feel free to add it on chat if you like and I will continue to monitor on that on the chat side okay so now we're moving on to real-time analytics so how do we get immediate responses based off of the data being ingested so we have a solution called database in memory which
is basically bringing a column store into your database into our database and now what's different about our database versus other databases out there when it comes to real-time analytics is we basically maintain two copy two formats I should say of your data two in-memory formats of your data uh one is a row store or a row formatted version which is great for doing transaction processing and the other is column store which is great for running analytics so if you've seen this picture over here here's your row store this is your column store the data is
a single copy of data sitting on disk your sales table sitting right over here and the optimizer then makes a decision should I go through the row store or should I go to the column store based off of the query that's running okay um this is not a separate engine that we plugged in this is this is natively built into the data engine and the benefit of this is that all of the existing things like the the clustering we just talked about the real application costs there's a rack that works seamlessly sharding like we just
talked about work seamlessly all these things just work the trick is to make sure that these two stay active and consistent with each other so any updates that happen to the data on disk get reflected in the in-memory formats that we have here okay um very quickly this is just a picture that shows how this works we take the data that's sitting on disk in the row format we populate it we call this populate to bring it into memory into columns which means we have to transpose the rows into columns we use interesting formats data
formats um dictionary encoding run length and coding things like that to keep the data compressed and keep it in a format that can be queried without decompressing the data okay okay so the key technologies that we have number one as I mentioned column format it allows you to access Only The Columns you care about um that saves you on i o costs um and even within memory I'm only running on just that column I care about I avoid bringing into Cash all those other columns um Vector processing um I don't know I'm assuming you guys
have talked about simdi Vector processing is that correct I just want to make sure I I probably mentioned it but we don't discuss in detail okay okay no problem so very quickly on this um this is huge for columnar databases common databases get their speed because when they operate on a column of data that is uh you know could be compressed or dictionary encoded and so on the way we do filtering is we grab a bunch of rows or column values at a time into a vector register Sim D Vector register simdi stands for single
instruction multiple data and allows you to do evaluation on a batch of column values at a time so this picture shows you hey let's say I had a column of States California Wisconsin and so on and I want to know which of these states is equal to California so I load a batch of them into a single Vector register let's say I load um 64 of them at a time and then in one cycle I can do a 64 comparisons of each one of these to say which one's California okay so that's the that's how
you get to billions of rows per second instead of millions of rows per second with a row store okay um we have something called storage indexes that allows you to do very fast pruning so imagine you have a batch of rows here a batch of rows and a batch of rows but we maintain the Min and Max values for each column Within These batch of rows so if I'm looking for a value of nine well I know it's not going to be in this batch because the Min and the max 9 is outside of this
range it's outside of this range and it's only inside of this range so I can quickly say this is the batch of columns I want to look at not these okay and the last thing that's very important is compression I need we want to be able to uh keep this data in memory and be able to scan it very quickly and for that reason we really have to have solid compression technology to be able to to achieve that okay um I'm just going to scan through this very quickly all aspects of analytics for event stream
processing you need to be able to have very fast scans you need to be able to join between Dimension tables and fact tables very quickly and you're going to be able to do reporting aggregation very quickly so we have a lot of Technology built around this to make scans joins and Reporting faster and I'm going to cover just a few of them in a bit um one thing I want to talk about now is how do we make mixed workloads faster so when you think of an event stream processing solution you are doing two things
you're ingesting your data so a lot of new data is coming in and being inserted in and you're also doing analytics against it you're doing scans and joins navigation against it so this is considered in my mind a mixed workload because I'm doing two things I'm doing some DML operations I'm changing my data a lot of data can be coming out of order because not all devices are sending the events at the same time so it takes a long time for it to come so it can be out of order so how do we make
mixed workloads then faster so in a mixed workload you're typically going to have one to three indexes just to be able to quickly get to a row these are called oltp indexes and then you can have a bunch of analytic indexes to do your fast scans or filtering and so on and the problem with those analytic indexes when in the mixed workload is every time you do an update to your table or an insert into this table all of these indexes have to be maintained every single one of them requires an update and that's very
very expensive and so what we claim is that you can replace those analytic indexes with just your column store and now whenever you do an update over here you need to maintain it in the column store but because that column store is not persistent the updates are very fast it just requires setting a bit and we'll talk about that that says hey this guy's been updated and that's it okay so we'll we'll talk about that um here in fact okay so let's go through this example I'm going to skip through the words and just show
you the animation so you see that there's again our architecture our row store here in memory our column store here in memory and the data sitting on disk now let's say you do an update okay to this table you want to update this row it's going to update the row in your buffer cache in memory it's going to say okay this row has xxxxx it's been updated okay now let's say you run a query okay sorry let me go back so once that happens what we do is every block of rows in the on disk
are mapped to their location in memory so as soon as you do an update to some Row in some Block it's very quick to identify where in memory in your column store that was updated and so we just have to have a bit Vector that maintains mappings of rows to their location on disk and set a bit that says it's invalid it's no longer valid to look at that in the column Stone so now when you run a query okay that wants to do an analytic query we go through all of the call all of
the the column values but we skip the ones that are invalid and we keep going okay so by skipping it that means everything else is valid except that one and so for just that one that's invalid we will go back to the buffer cache to get its updated copy so that's how we stay transactionally consistent between our column store and our row store okay hopefully that makes sense the only problem now is if there are too many invalid rows in the column store it starts to become expensive you're going to do this columnar scan that's
going to skip a bunch of invalid rows and then you're just going to go back to the row store to get those you might as well potentially just go to the row store and do your analytics on it so this is where we have techniques that will quickly refresh the copy in memory the in-memory formatted version in the column store so that it gets rid of the invalid rows and brings in the fresh rows so that everything is fresh and new okay hopefully that makes sense um and that's again really important for event stores because
even though event store processing aren't doing really updates they're doing inserts and when you insert you're also for saying hey there's a hole in this column store because I brought something in that's new go here to go get it and so that's why we want to make sure that this thing stays nice and fresh okay um oh there's some questions let me just look at this really quickly what if only row store is down and column store is not what if row star is inconsistent with Comstock which one is more trustful okay so the real
star is never so as I mentioned um the rose store has the truth it's the source of Truth so the column store will simply say it's invalid and that is always true it's it's it's kept in luck with the row store in terms of that role being invalid and so as like I mentioned as soon as we detect that that row is invalid in the column store we go to the row store to fetch that row and that's why whenever you run your query it will always be correct okay um can we join stream data
don't we need to complete tables to perform joints so yeah so you can join a stream uh streaming tables with another table um and that's something that a database does very well at whereas a so for example let's say you have streaming data that's coming in and based off of that streaming data you want to look at another table that maybe has spatial data that says hey based off of all these events are coming in I want to know um from my from my spatial data which restaurant I should go to based off of all
these high rankings that are coming in in real time or something like that okay so then there is a join that happens where you you're going to run your query against the streaming table and then based off of the results of that you you'll join with another table that might be based off of location ID or something and that's the the primary key foreign key relationship between those two tables and then you pull out relevant data from the other table so yeah you can do that okay let me keep going um and what I'm gonna
do is I'm just going to spend a little bit of time just talking about a few key things on real-time analytics and then I'll I'll move on to the next requirement so this is an example of how we make the column store real time and as fast as possible so let's just look at the picture here and ignore the words um oftentimes you might run an expression like this um Price Plus price times tax okay that's equal to your net price price plus price times tax so you have to read the price column you have
to read the tax column then you're going to do price times the tax column and then you're going to add it to price again okay now that can be expensive what we want to do is say why don't you tell us that this is a virtual column net is Price Plus price times tax and once you tell us that this is a virtual column we will go ahead and populate into the column store a new column an added column if you will that is the materialization of that virtual column of Price Plus price times tax
so the next time you run this you're just going to go through here and you don't have to pay the cost of going through price and tax okay you can do this manually you can declare this manually or we can Auto detect that this is a very frequently executed expression and we'll populate this ourselves okay um so I'll skip this example as for date queries it's it's a version of this let's skip that for now um and now I'm going to say how we make joints faster so both joins and in memory Expressions as you'll
see from the demo slides uh hopefully we'll have time to show you will help us make event stream processing really fast so here how do we make joints fast well let's suppose that you are joining a line item table with the orders table okay when you're joining these two tables on the order key again if you tell us by creating what we call a join group that you want to join these two tables we do the following we take your line item table we look at the order Keys okay now these order Keys could be
random values now the line item table is the fact table right so it's going to have foreign keys of the order key but you look at the orders table and all of those orders are going to be distinct they're going to be random values but they're distinct and you only have from 0 to in this case 2698 distinct orders now what we next do is we build this structure what we call a global dictionary and it Maps orders to dense keys or dense values and the maximum dense value is going to be 2698 because that's
the max number of distinct orders you have okay and so all of these orders order numbers get replaced with these dense Global codes okay so now what we do is we replace the hash join hash joint involves taking this join key hashing it looking it up into a hash table then doing a key comparison to make sure these order keys are the same in order to be able to determine if you have a match instead you can now convert that hash joint to a very simple array lookup so you got your code and you want
to say is there a match with the other with the orders table or with the line I am table you look it up into this array of in the orders table of just 2698 entries and if there's a hit then you there's a match and if not there's no match and that's it everything is based off of now a simple uh random potentially a random memory access okay but it's that's it's an o1 lookup and depending on the size of this uh of how dense this is it could be random or may not it may
already be cached in your l1l2l3 CPU caches okay um hopefully that makes sense that's how we make joins a lot faster there okay um now I'm going to skip through this converge workload but the main thing I want to maintain here is in the column store we maintain different representations for different data types so for spatial we have a unique representation of spatial so that spatial operations are faster same with text we keep instead of having your document your text document here well in addition to maintaining a text document here we store a text index
or an inverted index in the column store so that you can quickly match all documents all all resumes in this example that have the database keyword in it so you can quickly get to it same thing with Json on in the column store we do something very unique and I'll explain that in this next slide we columnarize your Json document so if you look at this Json the the three Json documents here one two and three and they all have a first name in this case first name is John Allen Clara so we take those
documents and we create columns out of it with name being the column name and all of the values Alan Clara John and then we do the same thing for all of the other uh paths that are in those Json documents now so we call these path value columns and so now when you want to run a query that's a Json query that's looking at age less than 34 and name equal to John and address to the equals Iowa City these now become very fast CMD vectorized scans on essentially relational columns okay so this is how
we make Json fastest now going back to event stream processing where we're dealing with Json documents as the attributes for every event now you can see why we can run very fast uh predicates and filtering on those Json docs because we columnarize them and then we put them in memory and so it goes a lot faster okay all right so I'm gonna largely skip this section but just just summarize it um Rich analytics query functionally as I mentioned it's very important um so analytic window functions allows you to do things like this where you want
to select the max energy over a partition of time so for every this is one window for one hour of readings I want to know what was the max temp max energy used to for the next window it was 1.5 so it's doing an aggregate Within These when within this window of time and that's an example of very rich um analytic functionality that you want your event stream processing system to to be able to support okay so I'll skip through the words over here similarly if you can look on the right you want to be
able to do more complex analytics so here I want to do what's called a match recognized that wants to look for specific patterns so you can define a pattern here so for example let's say you're looking at the stock market ignore this for a second but let's say you're looking at the stock market and we all look at the stock market these days it's down down down but you're like trying to predict when am I going to see this v-shape kind of uh graph on the stock market where all the stocks go back up again
and so you're looking for these v-shaped patterns and that's the pattern that you can describe um here and then run and match recognize function that will then look at all of your data even the data that's coming in in real time and see you know do I have a match that fits this pattern so again you need that behavior in your event stream processing system and then the last thing here is you want to have machine learning capabilities again for the purposes of prediction you know if you want to predict from whether there's going to
be fraud that's happening based off of all of these new transactions that are coming in or you want to do sentiment analysis based off of readings that are coming in from news feeds there's spam filters so you need to have that analytics capability in your system okay okay more things there okay and the last part I want to talk about from requirements is data life cycle management as I mentioned you know when data comes in it's coming into this right optimized region if you will right optimize means I want to ingest the data efficiently that's
I want to optimize for write speeds then it moves that data slowly moves into this read optimized category which is saying okay I want to be able to do analytics I want to be able to run real-time analytics on this data that's that means I want to optimize it for read performance and then the last thing is okay now the data's cooled down a little bit now I want to be space optimized I want to be able to optimize this data for space and compress it or summarize it and so on so that's um what
this uh bit says over here right optimize Zone read optimize Zone space optimize Zone so how do we do this um we basically have a way of automating movement data movement from one region into another um and I think I have a point over here uh yeah uh right here at the bottom okay so this gradient of write read and space optimization is achieved with something that we call automatic data optimization and a dbms scheduler what these two things do is they say you allows you to create a policy if you will that says after
one hour of no rights or no updates to this event compress it for example I'm just giving an example or after this partition has exceeded one hour compress it or move it to a different tier so this kind of facility to automatically compress your data or um or reformat your data over time is important and and we have a schedule and capability to make that happen okay um last thing is it's called down sampling and this is just a very simple concept that says hey your media readings have been these very large six hour readings
every six hour readings you do five minute consumptions every you know within a six hour partition now I want to summarize all this into essentially 60 meet it 60 Minute consumptions so for every 12 readings here that 12 times 5 makes sixty I want to be able to have a single event that captures those 12 events and that's called down sampling it's basically saying Hey I want to round off or some summarize that data so down sampling again can be achieved through the combination of that scheduling capability and the automatic data optimizations feature all right
so this is the last summer slide and I'm just going to show you some demo slides really quickly and this just says look um for event stream processing we need that flexible data model we use Json for that we want high speed ingestion we talk about all these things over here with partitioning and mem optimized tables and rack for clustering and sharding and all that stuff for for ingestion we want streaming analytics capability right we want a very rich analytics Library we want real-time analytics and that's where database and memory comes into the picture and
then the last thing is the automatic event life cycle management where you want to be able to do automatic compression using a scheduler capability okay okay so what I wanted to show you guys now is just a quick set of demo slides that shows you how we deal with devops monitoring so devops monitoring if you look at the picture on the right it's very simple the idea is like you know we're Oracle Cloud we have Oracle Cloud there's Amazon Google Cloud everybody basically has a fleet of servers where you as an application you know or
customers running your application on them and we need to make sure that those servers are running up to up to speed right we want to make sure that there's no downtime that the customer is going to experience so these servers are giving off metrics things like CPU utilization things like memory availability disk space availability all that stuff and those metrics are going to be streaming into an event stream processing database and from there you're going to run queries against it things to do like anomaly detection you know which server out of all my servers has
exceeded at server temperature you want to generate reports again or dashboards okay bottom line is you want to do some anomalous detection okay so that's devops monitoring now we looked at Prometheus and I don't know how many of you have to Prometheus but it's a very popular monitoring open source monitoring and alerting solution and they build a whole ecosystem out of it so they begin with their own uh storage engine their own database if you will that's like a Time series database and on the left side they have what's called data exporters which is for
every server or every system that you want to monitor you run a an application called a data exporter that collects metrics and uh the the data storage engine pulls those metrics into the data engine okay so that's how it gathers in now on the right side to to run your queries you run with grafana that allows you to generate dashboards and you can run queries against it using Chrome ql their their own language and then it allows you the data engine itself allows you to generate alerts so it says like hey if a a specific
threshold has been exceeded um go ahead and fire off an email saying hey the alert has been you know reached thresholds will reach so this is their whole ecosystem and to kind of prototype this so the initial prototype we took away Prometheus okay we wanted to say hey you know what if what happens if we take away Prometheus and just replace it with Oracle so we did that we got rid of the Prometheus engine and we replaced it with Oracle and then we added all those cool Technologies we talked about we brought in rack for
scale out we brought in partitioning support uh we used our own user-defined functions to be able to pull metrics from those exporters we had the mem optimized for right to be able to ingest directly into memory then we use Json to store the data uh database that I mean for real-time analytics a new plugin for graphite to be able to run queries etc etc okay we brought in the compression aspects of it all that stuff we use so that the whole ecosystem stayed in place but we just replaced the Prometheus data Engine with Oracle okay
and that was our prototype and we were able to show that everything worked fine we were able to just stay directly into the database we were able to run the queries just fine through gafana and that proved that okay we can do this and then now for the real demo what we did recently is there's a real product out there called exadata real-time insights and what this is is on our exadata database machines these are machines that are in the cloud that our customers are running on their applications on and we with xcity real-time insights
it's generating those metrics okay so it's a devops monitoring solution it generates these statistics and those metrics for us to observe and monitor okay in our setup in the in in the real setup we have over 2 000 events that are collected from each and every single server every second okay and all those metrics that we talked about are what collected okay so then what we did is we're going to build this demo that monitors over 500 Exodus servers in the real Oracle data center in real time and this came out to about 2.5 billion
exadata metrics captured over 24 hour period in one day and then we showed in this demo how we're able to detect anomalous Behavior very quickly using database and memory for real-time analytics versus without database in memory okay so the setup was like this you had all these extra data machines this is an example reading an example metric value was a temperature its value is 25 degrees Celsius here's the time stamp and then these are the attributes it came from this Fleet in Phoenix the server name was this there's a unit value in Celsius so that's
an example metric we use again the database server we use the rest facilities to ingest the data again we had interval partitioning and use Json storage we had grafana to show the dashboard and we were able to use that to run queries against him okay and of course database and memory um let's skip to that so what we did right off the bat is there's a database overview page it shows over 24 hours summary this is how many metrics you're able to get the data size is almost you know 450 gigabytes in memory we compressed
it by 16.4 X so again this compression is very important because we want to keep this data as much as possible in memory so compression is super important okay then we looked at a one hour summary with without in memory when you're running this dashboard these dashboards queries are running in real time and it's saying in this one hour I want to know how many metrics were there how many of those metrics came from how many different fleets and how many different storage engines database servers and storage servers were there so and you can see
that in that one hour um you can't tell because I'm not writing the demo live but it pops up very quickly and with that in memory it's still spinning because this is our first indication that it takes some time to run these queries without in memory now um now why why does it take time so look at this query that's actually being ran okay the query has three different important things one it's doing a Json lookup Json value it's trying to say give me the server names okay now as I mentioned why is in memory
faster because we do columnar Json we take each of those columns those attributes in Json we make them columns so that's going to be number one why why we're faster number two it's a like predicate okay so any of these filters that we do we make those scans really really fast so it is a like predicate and then the last thing is aggregation count distinct again within memory in our data formats we can do aggregation count this thing all that stuff an order of magnitude faster than without memory okay so this last slide let me
check my time I have what one more minute let's say this last thing just shows hey I want to now detect a specific problem with a server so this is an example where I check to see find me any servers that are hotter than 40 degrees and it popped up one of them one server is hot now when no in memory it's still spinning okay that's the idea the idea is we want to be able to dive deep into the problem look into this server right away um so that we can figure out what's going
on so I dive into the server and I say Yep this storage server was hot right here it went to 40 degrees celsius right around this time and then it dropped for whatever reasons okay it was going up got hot then it dropped so okay fine now I look at the disk controller battery and it says okay that also was hot around that same time that's really odd and I look at the storage server CPU and says well before it got hot there was some CPU activity you know 50 all the way up to about
70 so something was running not really hot but it was hot and then I look at the last metric it says working fans and you can see from the working fans we went from 12 working fans all the way down to two right around the time when the temperature exceeded and you can say okay there's a not a root cause but definitely an explanation you know something might be wrong with the working fans something may have caused the working fans but it went back up so that tells the the IT person they had to go
look at that server check on the working fans okay so while that whole investigation happens we show that hey within mem with that without in memory you're still spinning okay that's that's the the real-time analytics part is I can quickly determine what the problem is okay all right so that's it I want to just go past this really quickly skip through these slides and just simply talk about the research and development opportunities in the space um we care about compression we care about Hardware acceleration how do I make these different workflows like event stream processing
run even faster on different Hardware um with gpus and fpgas and so on um how do I there's something called approximate indexes you should learn about that if you haven't read about that but that's another area of good research in this space um machine learning mixed workloads and memory technology all these are kind of active engagements that we are involved in as an organization uh to make this specific workload run faster let alone all of the other workloads that we're trying to optimize right the different Technologies they've ever invested getting okay so um thank you
I apologize for going right at that time but I hope that was helpful