InfluxDB IOx Tech Talks: Intro to the InfluxDB IOx Read Buffer

Hello everyone and welcome to today's session of influx db ios tech talks my name is ken croft i am super excited to have you all here today we'll just give all of our friends a couple minutes to join so we'll just go over a couple of friendly reminders this session is being recorded so it will be Available for replays later today as well as the slides will be available for review as well so don't worry obviously we'd love for you to stay for the whole time but if you need to drop out you will be

able to go and check out the slides and the recording uh later today um this is zoom so please feel free to post any Questions you may have in the q a box or the chat we will be monitoring both and we will get uh paul ed and andrew to answer all of your questions at the end so don't be shy they love hearing from you guys and seeing what you guys are interested about this new project and just getting your feedback on it as well so We look forward to seeing uh comments from you

come in a couple of other friendly reminders we do have the community flat channel which is a really awesome resource to stay connected with the influx db community which includes a lot of input state employees uh including all the engineers who are building uh these products so it's a really great resource and it's also fun to just see What other community members are doing we of course have our forums as well of course events this year again are going to be pretty much virtual so super excited to announce that influx days emea 2021 will be

happening on may 18th and 19th it will be virtual uh call for papers is still open call for papers closes this friday so we're super excited to see all the different things that our Community is doing and reviewing your submissions so we look forward to seeing you all there as well we also have all of our other webinars and other virtual events such as virtual time series meetups which have been really fun uh we've been doing a lot of hobbyist topics recently so be sure to check those out so without further ado i am going

to hand things off to paul dicks All right thanks caitlin can you hear me sound great all right great uh okay so thanks everybody for joining us for the tech talk today uh i just wanted to give a quick update on the project before i hand it over to ed who's gonna talk about uh much more interesting things uh so for those of you who don't know just in case influx db iox is basically the new core of influx db That we're building it's written in rust it's using apache arrow is the core underlying technology

it's basically a columnar time series database that uses object store as its persistence layer and it's a number of other things as well but that's for later uh so updates hold on a sec um so the progress so far uh we have new team members uh dom who joined us Actually a few months ago and then marco and raphael who joined us uh last week and we have another team member na who's joining us next week so that makes uh a much bigger team which means hopefully we're going to be making faster progress uh over

the next or the next uh few months uh so we made uh ed in particular has made a bunch of progress within the reed Buffer which is basically the compressed column restore that he's talking about today execution engine uh andrew has made great progress on basically wiring up the mutable buffer which is where rights land and the reed buffer which is query optimized in the system so that they'll be queryable across uh sql across influx ql and flux uh our uh helpful contractors at integer 32 have landed apache aero flight at the api into iox

and actually to get that working we had to do some contribution to uh the rust implementation of flight so now with the recent release of aero 3.0 the flight the rust flight tests run like those integration tests run in addition to python and all the other first class languages uh that apache arrow has um and then the other thing is over the Last week or so we've had an internal discussion some of which has been on the github issues and a lot of which has happened kind of synchronously over zoom uh about how we're going

to handle basically a bunch of ioc servers working together how they will replicate data uh how the right ahead logs work across them and how you'll be able to split roles from servers that are receiving rights versus servers that are Receiving queries uh so uh dom wrote up a document kind of capturing all of that uh obviously i don't think you can click the link here but i will put it up on twitter right after i get done speaking so i'll put a link to that doc it's public so people can read through it so

we've also made some decisions on the api we kind of this very very recent we've Been debating about these things so obviously all this is kind of subject to change uh over the next few months is we really like hammer out the details but this is kind of where we've landed so the management api will be uh over grpc which just makes it easier to have us you know have a declarative thing this documented all this other stuff we'll have the command line interface for common tasks like creating databases And stuff like that to make

it easier to use for writing data into iocs we'll have uh inflective 2.0 line protocol which we currently support uh we also want to add support for just posting flat json objects uh to the database events and then we've also been thinking about protobuf somebody brought this up last month as uh something they wanted so we may have That open because the set of data type types that we want iax to support is broader than what lime protocol supports and we'd rather just uh piggyback off something like protobuff that'll give people like strong typing and

all this other stuff on the query side of things uh we'll you know expose querying vhttp they'll return csv currently you can do that with a sql Query but we also want to support returning json or something that's basically just like you can display uh it's kind of like a pretty print kind of thing uh aeroflight which obviously we just added support for and then we've also been talking about adding support for querying via postgres wire protocol so what's next obviously the management api We landed on the decision that's going to be grpc now we

just need to start moving forward and converting some of the previous work we've done over to that and implementing all the new endpoints a bunch of persistence work basically persisting the immutable buffer and the read buffer uh in parquet files to object store some of this work has already been done but it needs to be reworked a little bit with the new system Um recovery from object store and that's both from the right ahead log and from the persisted partitions and chunks that are parquet files replication and subscription replication for the write ahead logs subscriptions

for query servers that are pulling data and then obviously official builds and documentation we want to get to that originally we were targeting uh the end of this month For that uh based on all these changes we've recently made and basically this internal back and forth we're probably not going to get to that we also have our company yearly kickoff uh the week after next which is going to impact all of our uh ability to to work on iox so now we're targeting late march for that but with the goal of having a lot of

This stuff wired up in the some initial stage where you can actually test end to end uh and start playing around with things and that is all i have obviously i'll be around for the q a after ed's talk ed hand it off to you thank you i'm just going to share my screen i hope this is all working anyone see the screen okay and hear Up great me brilliant um okay let's get going so hi uh so i'm ed um i'm an uh engineer at influx data um working on the iops project i've actually

been at influx data for about four years uh but for about five years four years of which i was working on influx db uh mainly on the storage engine and the right path and indexing So i'm kind of familiar with um the use cases that it's really great for and where some of the some of the kind of issues are around other use cases and what we're trying to and how we're going to try and solve those problems with iocs um which i've been working on and it rests with uh paul and andrew and is

now for just over a year so what are we what are we working towards so like these are just some of like the Things that like i guess we think about a kind of a high level around like performance scalability in terms of ios right so we effectively want to be able to handle unlimited data um that means realistically that we're going to need to leverage things uh very scalable like object storage um it means we're going to need to think about things like compression we don't want to be storing like Unlimited amounts of data

in a in a large raw way we want to think about being able to support unlimited cardinality um sometimes this gets referred to as infinite cardinality but i think unlimited is perhaps more appropriate but that effectively just means that you just don't have to worry about cardinality right so if you have a high cardinality use case you should be able to just Treat that like more data right not not different data um and so in order to kind of be able to handle unlimited amounts of cardinality we have to think very carefully about data organization

right and so one of the things that we that we're making a bet on in iris is that we we don't want large secondary indexes because they're something that are expensive to maintain when you have huge amounts of cardinality um and we Want to like be amazingly fast and performant for analytical type queries um and we think to do that we're going to need to do some um in memory uh processing we're going to be able to have a memory execution we think that's the best medium for that data to sit on uh and we

think that a column the data layout column architecture is is the way to do that And then there's just like lots of uh lots of other fanciness that i'm gonna talk a bit about now um so these kind of as paul just mentioned these are some of the things that we're leveraging right mainly rust it's helping us in a bunch of different ways i'm not going to not going to get into an apache area brilliant in memory format and data fusion is also a really Interesting rust project that provides us with a query engine so

this talk particularly is about a particular subsystem in ios that we call the rebuffer it's a new kind of query query execution engine um we something i've kind of been working on i guess for about six months now and we've been sort of spiking out Various prototypes and kind of sort of settling on a design and the idea is that we're going to work on data that we hold in memory um particularly on the rust heap so we're going to limit in terms of the read buffer scope there's basically no i o at read time

so the read book has either got it in memory on the heap to process or it just doesn't have that data we are not interested in like Mapping and mapping things in and out and we don't want to be fetching things from like other storage mediums that's going to happen in ios but that's not like it was the concern of the read buffer right up to other systems to make that data available in the read buffer so we can think of this data as being immutable which is another nice property right that means we can

leverage certain Techniques and certain approaches to improve the read performance um and there's just a bunch of like interesting column store stuff that we that we're kind of doing in the rebuffer so obviously we have like a column of representation we're as much as possible trying to work directly on this compressed representation so i'll explain in more depth what that means In the talk um we're basically leveraging things like vectorized execution and particularly we're trying to make use of modern kind of hardware features such as cmd and things like that so we're thinking carefully about

how we organize our data so that we can leverage those those features uh we're gonna be able to do um interesting things like um predicate and projection push down which i'll talk A little bit about later but it just basically means um getting um all of the constraints around the data that you want from your query down as far as possible as low as possible so that you can remove the need to process as much data as possible and you're just doing the minimum amount of work um to basically answer the the queries that you

want answering And as part of that kind of stuff that sort of touches on what we call late materialization which is another kind of well-known column the database technique that effectively means you save the expensive part of your um you know often referred to as two for construction right you you save the most expensive part materializing all your raw underlying values as late as possible until you've Got rid of as much data done as much processing as you can um before you go through that stage so i'll talk about that later in talk um so

like at a kind of very high level like we want to just be able to do um any kind of time series workload in ios but we also want to be able to do particular time series workflows like things around events like all of the observability things that you want to do around logging tracing Large analytical workloads some of these like workloads typically are very high cardinality or high volume in terms of data and so we think that um ios can provide some kind of um some help there so like as i'm sure some of

you on the on the meeting already know we already have a time series database right so influx db um so let's just do like a really quick Refresher about where we're kind of coming from in terms of what influx db is really good at what it's not so good at and what kind of things we're trying to solve with iops so influx db is really great for iot for monitoring but other time series use cases right there's loads of things you can use it's got really really good block compression Of most field and timestamp data

um sometimes you can get absolutely incredible compression sometimes you get not quite so good compression it really just depends but typically for time series data it's very very good um it's got a very flexible data model so you're not strict on like your schema you've got to kind of decide what things you think of tags or things you think of fields those are Those distinctions are very important i mean you do have to kind of think about that up front but aside from that you can add new tags you can change um like you know

bits of what your schema looks like um influx has got really good performance on selective queries so what selective queries they're basically queries where you're only interested in a very very small proportion of the date of the underlying Data set right and you provide predicates um and you know constraints and ranges to like specify exactly what you're after influx is really good at answering these kind of questions if you've got like like billions and billions of samples over like millions of different series like influx tv is great for this because it can handle that kind

of cardinality around the series so when we talk about Cardinality we can influence we're talking about combinations of measurements and tag keys and field keys and if you're in the millions there it can handle that quite well and then the billions of samples for those that amount of cardinality is not a problem because of all of its block compression which is like really really good so i'm not gonna like jump into like all the details on this Slide but like i don't know if maybe some of you recognize this but it's essentially like an internal

tool we have that it allows us to inspect the data format that influx db persists its time series data in which is called tsm files and when we look at when i look at something like this which i pulled from a an actual real like iot use case um i can see that like we have almost a two Gigabyte file um and only about 75 megabytes of this file is the index right the index are basically the series keys that determine like where our data lives within the file but the file although it's not only

2 gigahertz under 2 gigabytes it's got 525 million points in it right so there's only 3.6 bytes per point so this is like very very good compression that we've got going on here We've got lots and lots of samples for relatively small number of series so i figured out if based on the contents of this file if we just stored them uncompressed just as like timestamps of 64 bits um there's lots of floats and integers and here they were 64 bits this would be like 67 gigs of data but we were able to compress it

down to 1.8 gigs which is really really good so unfortunately though um If you have a different kind of use case or a different sort of different kind of workload that you use with influx db you can have problems and you don't leverage some of these benefits right so in this case we're looking at a tsm file which you might use for like a tracing use case so often in tracing as i'll talk about a bit later on um every single sample can essentially be unique you end Up with like one point per series because

often you have like very high cardinality tag values like these might be span ids another example would be if you're like doing a lot of monitoring in a very like ephemeral environment like kubernetes or something you might have like loads of container ids that are like constantly changing so you can have these like very high cardinality workloads with very few Like samples per perlex series collected so in this case we've got like a 1.3 gigabyte tsm file but like 1.1 gigabytes of it is the index because it's just full of like unique series and we

don't have much data per series right and so because we have to wrap all of our data points in our block compression that we do in influx db and we have to store them in various places and we have to point to those Blocks the tsm data for the blocks is actually um uh have we got that wrong around so that basically the uncompressed the uncompressed tear set the uncompressed data for the for the blocks um would be like smaller than the the tsm version i think i've got those two the wrong way around but

basically we're doing a worse job than if we had just hadn't compressed it At all so this is like not where influx wants to be because you're not you're just not leveraging um the the features of like influx db and tsm format so there are ways around these like use cases by the way so you can just like swap out tags for fields some people do that but once you start to do that you're restricting like the like the things you can do with the query language and those sorts of things So um like in

summary like in pathological cases like i've just described the index dominance tsm and the compression that tsm's so good at isn't very useful when you don't have much to compress right so it's not kind of this is like kind of obvious um we also use mmap in the tsm and influx db csm formats in influx db and we've found like through many years of like operating influx db and And enterprise versions and things like that that sometimes map can be unpredictable um it's controlled by the kernel you don't have a huge amount of control over

it it can be difficult as well in containerized environments things like c groups and stuff like that it's just sometimes you just end up with beams when you shouldn't really be having them and you're basically paging in like four Ks of data and you're hoping that all of the data in that 4k page is like relevant to the to like the workload that you're running and often it isn't because of the way we we organize the data so the data in the tsm file is distributed according to the lexicographic order of the series keys right

because it's an ss table but often you're querying not on the full series keys sometimes you're just querying on a single part of a Series key like a host or a region or an environment or something and so your data ends up being completely randomly distributed through the file and that's often like orthogonal to the like the access patterns that you want um the other problem is that for very high cardinality workloads and what i'll call non-selective queries right so that's where effectively you want to actually read Most of your data set to answer some

query like a big a big aggregate or something um influx db over indexes because you've got this huge index that's like basically there to answer like very specific selective queries and you're you're basically asking that index for everything all at once right because you want to aggregate across large volumes of data so some of the like bets that we're making in iocs are That with a columnar architecture and if we organize data in certain ways that we think makes like processing faster and keeps the footprint smaller then we're essentially going to be better at some

of these workloads that influx db doesn't do so well to do that we think we're going to have to ditch these large secondary indexes so influx db also has these large indexes called That's one called tsi for example and we think we need to not have those sorts of things in iocs because like fundamentally like a secondary index is is only useful to you if the cost of reading and maintaining that index um to find the data that you need to answer your query is is less than actually the cost of just processing the underlying

data Right that's the only time that secondary indexes are going to be a win for you um and we're going to say no to nmap we think that it's better to have predictable memory usage so if you you having all of your data or a portion of your data in memory it's on the heap you know where it is you can control what comes in and out um you know you you're not like reliant on map having The right things paged in the right places um the other thing is as well is like we want

to effectively only keep like relevant data in memory so this is a bit of a higher level concept in iops it's not so much about the read but it's more about we want to have systems that make sure in ios that the right things are being moved in and out of the read buffer at the right time so that We've got the data that we really care about being held in memory so we're thinking that for high cardinality like analytical workloads we're going to do really well because of the design and then we also think

that the trade-off here which i'll talk about later is that um even though we're not going to have indexes big secondary indexes and things like that that are really Good for selective queries and selective workloads we think that we can just keep close enough with basically very like smart engineering in terms of how we lay out the data how we take advantage of vectorization like materialization all these column store techniques that effectively brute force will keep us close performance wise to systems That are like heavily indexed all right so the rest of the talk um

i'm just going to sort of dive into the rebuffer now giving a bit of a bit of an overview on where we're sort of coming from i'm going to look at how we lay things out the representation some of the compression stuff uh what do i mean by late materialization in the read buffer and then also it has some Early numbers right some comparative numbers and we'll talk about future improvements so let's just talk about where the read buffer sits um this is like a very rough like right path diagram now this isn't these squares

aren't um pieces of like different processes necessarily or pieces of software but it's just more how data flows Through the iox system so obviously at the top we have like writes come into the database right so those rights will end up landing in the right headlock in the in what's called the well buffer that allows us to have replication to other ioc servers it also allows us for recovery so if we lose the server we can replay this right headlock but obviously right headlogs um maybe not obviously but Right headlogs are not very good for

querying right because it's essentially just an append only log of events so we need to be able to query the data like as soon as it comes in or very close to where it comes in so for that we have this thing called the mutable buffer um that allows us to query this like written recently written data it's mutable because it has to be updated all the time right new data is coming in we Need to be able to query it so um it's a it's similar to a mem like a mem table if any

of you are familiar with that term um but at some point though we can't keep all of this data in the musical buffer right so we need to sort of think about durability and we need to think about getting it into a format that's more read optimized so at some point the ioc server Basically will be able to move data out of this mutable buffer we'll put it into an object store that's like our storage persistence layer for durability and we'll also put put some of it or all of it into the read buffer right

so that when we put it into the rebuffer that's when we're essentially saying that this is now some immutable thing that we're gonna optimize uh the layout of The format to make it very very fast for read workflows uh yeah exactly so that's like how that's how the rebuffer looks on the right path but on the read path so now we want to read from the system this is kind of how things look so we have like these different um or we will have all these different uh front ends right so you'll be able to

query with sql With flux with influx ql maybe other frontends in the future and effectively they're all going to go into the query engine which is powered by data fusion very interesting project there's loads going on there and that query engine is then going to be responsible for figuring out what's the best place to go and get an answer to this query right where where do Who has the data is it only in object storage do i need to pull it out of object storage has it been written in recently can i get it from

an immutable buffer or is it something that the read buffer is going to have right so the query engine will be responsible for running against all these execution engines and then obviously like combining the results and all of these execution engines are going to be able To uh you know support every single possible kind of sql query and whatever so the query engine is also going to be responsible for like figuring out you know what who it can ask for what and what it has to do itself basically so i'm only really going to focus

now on the reader right which is just one of these execution engines so let's look at how Data is organized in the read buffer it's very similar to how data is organized in other parts of iops it's also quite similar to how data is organized in um things like park a for example right there are definite similarities there but like a very high level we just organize things by database right and so database is kind of similar to a Influx 1.x database if you're familiar with that and if you're familiar with influx db2 it's kind

of similar to a to a bucket so we can have multiple databases inside the rebuffer and then within a single database right we can have these things called partitions these are basically just um sections of the key space of all the data that was written into this database So most people we think at the moment most use cases are going to want to partition by time i think that's quite a common way to do things so you know you'll basically have all the data written in for an hour in one partition then the next hour

and the next hour but you know you can partition by different things you might partition by table or by host key or by some other thing Within each of these like partitions you're going to have these things called chunks right so partitions are like a logical concept you decide as an operator um or the system decides what's the best way to logically like separate out your data into sections but chunks are more like a physical concept so um if you're partitioning by an hour depending on how how heavy your ingest Is that might be hardly

any data or like huge amount data so we have these things called chunks so chunks basically are just like a a point in time where their data is a certain size or it's been sat in a mutable buffer for example for a certain amount of time and we want to just close it off and roll it over and make a new chunk and so you have these like Chunks within a partition and they're essentially just like subsections and they all have their own metadata so it's actually very useful for allowing us to do like pruning

and deciding whether or not we need to go and read data out of one of these chunks within a chunk you have um tables right and if you're familiar with influx db you'll see that these tables like the labels I've given them anyway look very similar to measurements right and so and they are effectively so if you're writing inline protocol into arcs for example your measurement name will become the table and so you have these various different tables and each one of these tables has a different schema right because it's for a different thing And

then inside each table right you have these row groups so row groups are like now we're just getting a bit more concrete okay so row groups are basically just like horizontal sections of table they're just like multiple rows within a table and the reason we have you might have one of them or many of them is well there's various implementation reasons but effectively They're all they all have the same schema you could think of them logically as just one table but they're just physically separated in some ways we have like metadata on tables like we

have min and max for the various columns each row group also has a metadata that we'll look at in the next slide and obviously we have a schema so all of this like metadata is very useful when you're running a query to decide whether Or not you even need to go and read into any of this data or whether you can just skip it now we're getting very concrete so a row group now probably looks more like a columnar table as well so a row group is like the lowest sort of unit here before we

get to a column it has it has a schema it has obviously metadata for each of the columns and that metadata again has a min max And that sort of thing right once you get into a column now you can kind of see this is one of andrew's slides that i pinched now you can kind of see what how we map line protocol to this model right so what the blue box is basically like a row group and the colored columns are iox columns and you can see from the circles like how we map Um

how we map uh sorry how we map influx db line protocol to these different like column types um i'm going to talk now about like all the different types of things we can talk about in the columns i'm going to talk about how we can press them down because that's like a really important thing that we do so you can we can support in iops in the rebreath at the moment but we will be supporting Soon at least like these uh logical types on the left so you'll see that a lot of them are very

similar to the ones that we support in influx db um and then the semantic column types is like so we want iops to be able to support the influx db data model and lion protocol and all the use cases but we want to be able to support like other data other data models as well and so we Basically map influx db um things like tags and fields and timestamps onto certain column types but we also want to be able to have um just non-influx db specific columns and that might have be able to support data

types that for example you can't like have in-line protocol so basically we have a mapping and effectively right you write line protocol in and we're going to map It to certain types of columns in ios so for example all of the tags that you're writing will get converted into string columns the fields will get converted into whatever the data type is of the line protocol field that we should you know we support all of them um time stamps are in uh signed integers and then iox columns could be basically anything so here's some of like

the features in The read buffer that are like i guess pertinent to time series stuff and to influx db type use cases so the rebuffer can do scams um essentially like you give it predicates uh pointed at table you say oh i want all the data as much as these predicates i could do group aggregates so you can do effectively give me the sum of all of the temperatures grouped by Like cpu or something it does windowed aggregates it's very similar to google aggregates but with obviously with the time component that's like the vast majority

of graphs that that i see on like influx data dashboards are grouping things by times very common and it can also do like the schema exploration stuff so if you're familiar with influx db1 or 2 where you can do like show check keys show tag values Show measurements and all of the possible constraints that you can put on those meta queries it's going to be able to support all of that as well it also does like i've just described tried to describe it that very hierarchical data model so it does pruning at all the different

stages um actually the query engine does some of the very high level pruning Um but the re the rebuffer can basically do all of the table and road group pruning as well so it can figure out whether to just ignore like swathes of data that it doesn't need to look at to answer a query um we can do like predicate push down so you give it all the predicates you need for your query they all go down to the read buffer it filters out all of the columns and parts of columns and rows and things

it Doesn't need based on the predicates if it determines that for example a column doesn't satisfy predicate at all then it can just short-circuit the query and we just return no results immediately for example um this is something that i think is kind of interesting because it's something that's been missing from influx db for such a long time is that we can do comparator operators um if you give it a constant so like Region greater than west um on tag columns because we can do predicates on any column in iocs and tags from influx db

line protocol just get converted into string columns so that's kind of cool because you can only do filtering in influx db like filter on tab column equals something not equal something called regex something but You'll be able to do like range based comparative ones without and you can also aggregate any column as well so you'll be able to do min max of type of uh tag columns which is kind of interesting um other aggregates all right so let's just talk a bit about how we actually get data like into the read button so there's basically

like a a spectrum of types of compression that You can do columnar compression and this kind of applies not just to like iox's new columnar architecture but this is also like relevant to influx db because you can think of influx db as a as like a sort of uh has a columnar architecture on certain bits of the data model particularly the timestamps and the field values and it does compression on those right So the one side you've got no compression at all so just don't compress anything um that's great in terms of processing because you've

always got the data that you want in front of you um but it obviously means that everything gets stored in the largest possible format the other end you've got loads of compression right so you just use some general purpose algorithm to compress everything down and you get a really really tiny Footprint but then you've got this huge potentially huge processing cost because you have to decompress this data to do anything with it right and so you can just sort of think of this like spectrum along this line um of various like ways of compressing things

so you've got very like um heavy stuff down this end um i would put like gorilla simple 8b these are algorithms that we use in influx db to Do block based compression i put them down here as well um you get a really amazing footprint from them as you saw in that early slide with the 500 odd million points in under two gigs but you you have to decompress all of that to read anything and then you can work your way down until you've got like a vec of t which is just like you're not

compressing it you're destroying the raw the raw values i'm going to talk about The fact that the reed buffer chooses to choose some of this like middle ground right we think we're making some interesting kind of choices there and the reason why we choose some of this middle ground is because the choice you make about how you compress your data your column data depends on the location of that of like that data right so the example here is that If your data is living in the cloud over a network and you can only really pull

it down to the cpu at you know a few hundred megabytes per second but you can decompress it at you know a gigabyte a second if it's you know some fast like compression algorithm then it's worth compressing it right because like effectively you can decompress the Throughput here will be higher because you can decompress it faster than you can pull it over the network so that's fine and as you move but as you move through this like medium you get faster and faster throughput right so once you run to an ssd now you can like

pull things at gigabytes a second once you get into main memory you can pull things at tens of gigabytes a second so you know do you compress things Or not because if you can pull things at tens of gigabytes a second and you can only decompress something at a gigabyte a second there's not much point in compressing it right except the problem is that as you move through these storage mediums the price just to have a you know have a gigabyte or whatever there it increases right so the cloud's effectively Free for all intents and

purposes ssds are getting cheaper and cheaper and cheaper but memory's still relatively expensive i said it's 10 gigabyte it probably isn't but my point here is that it's significantly more expensive per gigabyte than it is in other mediums and of course like it doesn't really scale in the same way right because if you want to add like scale out memory you end up having to Scale up processor and all the other bits of hardware that need to you know be around that so actually it turns out that it is it is important to think about

this compression question um and to like get the most out of this really expensive medium and so that's kind of what we've done here and that's like what a lot of other column stores do as well so i'm going to talk about the fact that we've Tried to find encodings and compression schemes that we think will give us that bang for our book right they're going to give us really good compression so we get more out of our memory but we're still going to have really high performance and we're not going to pay a huge

processing cost to get the data out so the first one of the first ones we look at is dictionary encoding so we do Dictionary encoding on a lot of columns in in ios it's really good for higher cardinality columns the order of the data in the column isn't really a factor it's great that you get constant time access um to the data um and you can operate directly on the date so i'll just talk a little bit about this quickly right so if you've got all of these like values here that you're trying to Compress

essentially all you do is you create a dictionary mapping it's really important that these are ordered and then you end up with these encoded integer representations then effectively you store um this dictionary here and then you store this encoded column here and obviously these are like four byte integers for example so you're going to reduce the size of Your column if all of your strings are significant and large if your strings are actually really really small you can still get a lot of performance benefits which i'll talk about in a minute so it's still kind

of worth doing it we found so here's an example of like how you would actually filter with this encoded column so here's our encoder dictionary column we want to do a filter like where region Equals e so the first thing you've got to do you basically look up the east of the dictionary you turn it into an integer and now you're basically just doing integer math and integer processing which is significantly faster than string based processing and you can basically just apply that i want all of the rows that contain x equals 0 and i

get rows 0 2 7 15 back right then the nice thing about dictionary encoding though is that you can do other types of predicate so you can do comparator ones now you can say i want to do all of the rows where region is greater than north and so effectively all you have to do is convert again into the dictionary and now you're just doing x is greater than one and you Can basically still answer this question the reason that this works is because the ordinal relationship of the encoded ids is the same as the

ordinal relationship of the underlying data the key takeaway here is that we're doing all of this processing we're finding these rows and we're not actually like manipulating any strings we're not reading strings we're not comparing strings we do all on integers So this is what we mean when we talk about working on the compressed data directly um another kind of uh sort of piece you can do after this that we do on a lot of lower cardinality columns is something called run length encoding so try to keep the pictures similar here so you can just

see how it follows like effectively these first three bits are similar to dictionary encoding And there's just one more stage right and that extra stage is that instead of storing this entire um encoded column we can press it down so if there are repeated values within the columns like here zeros and ones and zeros and threes we just basically store those as what we call run lengths so we'll store zero three times one is three times zero three times and so on and so forth it's just an extra layer of compression So for low cardinality

columns where you've got lots of these runs you get incredible incredible compression um so it works really well if your columns sorted or if it's like secondary sorted or tertiary sorted you can work really well there it's not as consumable by consumable i mean that you don't have effectively constant time access to the to the compressed data because once You've got it in this final compressed format here with these tuples you've got to undo those tuples to figure out where each row like logically sits right like you know that it's somewhere maybe around here but

you've got to figure out exactly where it is um so we kind of make a little change here so we pre-compute the bit set we pre-compute Bit sets for each of these um entries in the dictionary so it's a bit of a cheat right so east will have a bit set that contains all of the rows that belong to east north south and west will have their own so that if we're doing like if we're trying to do a query where we want to effectively look up all the values for region equals east instead of

having to decompress this Representation we can just look up a pre-computed bit set so we store these pre-computed bit sets here and we want region equals east well that's like id 0. so we know that the rows we want here are just already stored in this bit set we actually use um roaring bitmaps for this which we've used a lot in the past in influx db and they work really well because they Also themselves do run nothing coding so it kind of works works well in both ways and again you can do like regions greater

than north you can do the same thing um you can either union these bit sets or you can just do the decompression step so that's kind of how filtering works there it's give you some like idea of performance if you're doing um dictionary encoding On 10 million rows in a column which has say a cardinality of 10 000. this isn't a single thread by the way if you just draw store a vector of strings you're going to get maybe a throughput of about 200 megs a second or something 200 million values a second but if

you're using dictionary encoding and we have got lots of like cindy and stuff on top of this then we're able to Do about 4.2 4.3 billion values per second so each value is like a 4 byte integer so this is about 17 gigabytes a second which is near the limit of my memory bandwidth for my laptop so it's pretty good pretty good performance so it seems like dictionary encoding is definitely all we need if it's this fast um except like in some cases rle is just off the scale More performant right because we've got a

cheap right we've got pre-computed bit sets for certain um certain types of operations so in this case region equals east you can basically answer this question in just a few hundred nanoseconds and the footprint memory footprint is pretty similar to dictionary encoding so for a card column where you've got like a relatively low cardinality this is Rle is definitely the one that we tend to prefer but if you've got a high cardinality column like something like tracing like span id where it's essentially a unique value per row in the column dictionary encoding and the the

original vector strings are going to perform exactly the same because they've still got to read through all of the rows in the column to find them Find them all um but now rle will give you a really fast answer to this question but it's going to cost you a gigabyte in memory right because if you've got no run lengths in your column then you've essentially got as many run length encodings as you've got values and so it's just overhead so rle is no good for higher cardinality columns if you're really concerned about spaces So this

is kind of interesting trade-off between dictionary encoding and really encoding and to be honest we're still like working through empirically like where we're going to move between the two encodings so what about materialization materialization basically means okay so instead of finding some rows in your compressed column like i know which rows i want right Maybe i've gone and checked another column and i want you just to return the values that are there so um in this case we'll see we see that when you've got constant time access to your underlying data like the vector string

or the dictionary um you basically it's very very fast rle is very very fast when you've got a low cardinality because it doesn't have much decompression to do But as soon as you've got like really high cardinality in a root on this column that's where release starts to struggle so again this is like the price you kind of pay having this really fast filtering operations is that you pay in materialization so again this is like things that would just you have to sort of trade off so to summarize um if you're working on The compressed

data you get a lot of benefits over the raw data smaller generally lots of integer math all cmd vectorization all that sort of stuff so that means that even if the column is bigger is a bigger footprint than the raw data it can still be worth doing it just for the throughput um ra is brilliant for filtering but the size can grow um the dictionary filtering is Proportional to the size but the materialization is always brilliant it's constant time access um we also have numerical encodings as well so for example you can do um you

can provide i-64 used for f-64 and we'll just trim the high order bits so we'll store them in the lowest size um physical representation we can so in this case here we can take like a bunch Of i-64s and we can turn them into a bunch of u8s and so they only take six bytes instead of 48 bytes these are some of the things that we're like thinking about for numerical compression in the future so we want to keep the compression stuff lightweight so we can do things like frame of reference or where you take

away like base values from the column and then everything is just scaled that way So we're still kind of thinking on how we're going to do this it's just really important that we keep this constant time access so we're really trying to avoid like block based compression like we have in uh i'm just going to talk you through quickly like how we do some of the execution and the like late materialization um so the first thing we try and do is Just like try and prune away data right we don't if we can avoid it

then that's what we'll do but as soon as we have to like read some data out of the rebuffer this is kind of how we do it so here's like a query at the top like a select query so i want these columns i want to price some predicates so this so the first thing we do is we apply all the predicates individually to each column and these are all compressed in different ways That i just talked about we do all of that like filtering on the compressed values and then we produce these bit sets

of of like rows that match the route of match the predicates and then we basically do a turn of like integer uh set operations on these bit sets to provide the final set of rows that we know we need and only then once we know exactly which Rows we need do we actually materialize the original values right so we get things like you know host equals a1 host equals a2 instead of like working on these integers so this allows us to basically reduce the kind of um uh like increase the performance and reduce the amount

of memory and reduce like any of the wasted resources by messing with strings Until the very last minute we also can do this with grouping as well so we can basically similar process you apply your predicates to each of the individual columns you get your bit set and then the interesting thing here i think is that you actually build your group keys so when you're doing grouping you effectively have to build your group keys and then you can Associate all your aggregates for each of those group keys and we do that all using the encoded

compressed data so we don't even use any of this string original string values for group keys until we've actually built all of the aggregates we've got the final answer and then at the end we materialize the individual values that belong to each of those group keys so that works really well for Us as well um just looking at time we've got 10 minutes left so i'm just going to go through some numbers really quickly and uh i'm just going to like preface this with so these numbers are going to compare influx db and biox and

the influx db numbers we've done on a very big machine with like 24 cores and tons of memory um an nvm v drive and the ios numbers were done on a Laptop with like a single thread so the use case um is like a synthetic kind of high cardinality tracing one so the idea is that um effectively um we're going to have columns here that have got very high cardinality like span id so effectively unlimited cardinality we've got other things like trace id and lower cardinality columns and we can press them in different ways then

we're going to compare how we can Perform these certain queries on influx db versus files so the first thing is like space so iox is up to 10 times smaller footprint in terms of the read buffer size than it is on influx db so for 60 million rows of this kind of tracing data influx needed 36 gigs to store it on disk and iox rebuffer was storing it just under four gigs in memory and so that's quite a big significant Difference okay so what about like a needle in a haystack problem where an index is

really good and useful so this is something that influx db is just going to be very good at if we can get close to the influx db performance then we're going to be happy like in iops so you can see that input tv obviously just really works it's got an index to fall back on it can just find this trace id from the secondary index Whereas the rebuffer has to actually go and hunt for it and and like go through columnar data to find it it's still keeping up we're able to do and under like

20 milliseconds basically to to find a single trace idea across these like 60 million rows so we're kind of really happy with that this is like the worst scenario for the read buffer as well for the common database the better scenarios When you want to do like big aggregations over high cardinality so um this is like give me the give me like the total trace time for all my traces and group by trace id so if you try and do this in influx for one million rows you can do it in about 30 seconds in

influx but it will take 10 gigabytes of peak memory the read buffer will do it in like 45 milliseconds and it won't use much Memory at all influx db was able to answer this on a 10 million row data set but it took 18 minutes and it used all of my memory 128 gigs and a lot of swap space um whereas the rebirth can do it in under a second and it uses about 150 megabytes but that's not even peak that's it's kind of i won't get into it but that's just the total memory usage

so it Doesn't even peak at 150 megabytes and then for the 60 million row case influx just can't answer this query it's just not it's just not what it's designed to do well whereas ix is able to do in in a few seconds so we're really happy with this kind of like performance here because these are the kind of use cases that we're really targeting is being good at and we'll take the kind Of trade-off of the previous one where it's it's keeping up basically but it's just not you know it's fine it's fast enough

it's the way we're looking at it you can also do like schema exploration with the read buffer as well so like what about if you want to do the equivalent of like show tag keys how does that look in a columnar architecture without an index well it looks pretty good so influx is Obviously very good at these kind of queries because it's got an index and it's designed to like to answer these kind of schema exploration queries but the rebuffer can do them in you know several hundred like microseconds and that's because we just ask

the question in a slightly different way we can take advantage of a lot of the vectorization a lot of the Compressed data we work directly on it so we're just asking a bunch of questions of integers rather than a bunch of questions of strings effectively um some of the future work that we're looking at doing in the rebuffer so i won't spend too much on time on this but we're going to add more data types we're going to add more operators so you can do like regex and like and things like that We're going to

add more encodings um we've got a proposal for delete support so you will be able to delete things out of the read buffer it's immutable so we'll be solving that problem with things like tombstones um um there's still just more implementation to do and to round out and then also we're going to add things like predicate caching and buffer pooling like these numbers for iocs are just sort of the Beginning they're just going to get a lot faster we think and then some more some more stories around like how we're going to do concurrent execution

that's about it i think i've just gone over on time thanks ed uh looks like there's a couple of questions here um i know paul already answered this but just in case anyone else had this Question does this new columner database draw any familiarity with cassandra db well it's it's i mean it's a column oriented database so there's that kind of similarity but um maybe paul could answer more on like how he thinks operationally it's going to compare uh yeah i mean i think i say it's a completely different design um the design of iox

is more similar to Things like vertica or click house but also with the caveat that is designed to be operationally like working on data that's in memory and in object storage like ideally your working set fits in memory um so it's it's gonna be very very different than cassandra it'd be hard to draw parallels between the operational characteristics of cassandra and the operational characteristics of iocs Is it possible to have a configurable selection between rle and dictionary or is there an architectural limitation requiring a decision between the two so i think that the short answer

to that is that we probably will at some point expose the ability to to choose ahead of time as an operator how you want to like store certain columns but ideally we would like the ios and the rebuffer to kind of have Very good heuristics in there so that for the vast majority of cases it's just going to do the right thing the nice thing about the rebirth as well is that it's immutable so we only have to make that decision and stick to it we don't have to worry about that data changing and then

having to reconsider the encoding yeah we know a bunch of the statistics up front of Each of the pieces that are stored in the rebuffer and so you can make i think there's a good input to drive those heuristics that it's talking about i think the one of the interesting things also is that because data moves from you know this hot state and immutable buffer to like essentially a snapshot state in rebuffer or you know in parque files it's totally possible That from partition to partition the underlying encoding for the exact same column could be

totally different and that's based on the actual shape of the data it's based on the cardinality of the data and if you want to get really crazy like we could do things later on like change things based off query patterns and stuff like that but that's like years down the road all right a couple more questions when Will influx db iox be ready for production so our that's great always that question uh so our timeline basically is we have this uh deployed it's we have our own environment in cloud called tools which is what we

use to monitor our own cloud environment so basically we use our product to monitor our product um and we have Uh it deployed there right now taking a little bit of data over the next couple of months we're going to scale that up so that we're using it ourselves in a production capacity uh at a fairly large scale um our our tools cluster gets somewhere on the value on the order of like one point is 1.2 or 2.4 uh do you remember up 2.4 like 2.4 Values a second of data written in and obviously a

bunch of query workloads for monitoring and million values and stuff like that right it's more than two million values not 2.4 2.4 million values sorry um so uh the next step for us will be to release iox as essentially an alpha or beta product that people can opt into for our cloud environment Where we can control it and we can continue to deploy it and we can also have other backend systems that essentially like pull the data out aggressively into object storage to make sure that we don't lose data for example stuff like that um

so our goal is to have uh something available for uh alpha usage by you know late q2 Early q3 um and then you know we'll go from there as far as when you know the builds that we're gonna be producing like the publicly available builds that you can use in your own environment will be available for prod um we're not gonna have a 1.0 release of ios anytime this year i don't think uh so if you're waiting for 1.0 is your delineation for whether or not it's ready for prod like that's not going to Happen

we will have production like ready generally available supported instances of biox available in our own cloud this year but again that's because we can control the operating environment and we can add guardrails and safety around it um so it kind of depends on you uh i mean as we get like i said our goal is first build by the end of march those are obviously just for testing and Playing around with and then as we get farther along we can see how things go and people can ideally kind of pay attention and you know determine

what's what's the best decision for them will influx db iox replace or complement influx db so it will be a complement to influx db uh because influx db2 has a lot of things in the api that are Completely outside the scope of write data query data so the plan is to have iox as essentially a storage back end for influx db so you could use iox individually on its own or you can use you can have inflected be you know 2.x if you're using that we will have um a code that will allow you to

basically convert all that tsm data in influx db Over to iox and then start using iox as essentially a storage inquiry backend for influx okay what kind of object store will be supported so uh s3 azure gcp the local file system just memory if you want to run entirely in memory and then uh obviously any any object store that supports any one of those apis Will be supported but we'll also want to add like seth and mineo and stuff like that awesome uh thank you we'll just give everyone here another minute or two if anyone

has any last-minute questions thank you all for attending today's influx tv ios tech talks session uh so we have these sessions every month so they're held Every on the second wednesday or every month so the next session will be on march 10th at 8 a.m pacific so same time just next month so we look forward to seeing everyone there and of course the team is always super excited to present a project update so it doesn't look like there are any more questions so once again thank you everyone this session will be available On youtube later

today as well as eds and paul's slides thank you you