the atomic is a database for the JVM it's it's written enclosure it's a JVM app and in particular it's designed to be used by JVM applications so there's really two fundamental ideas one is to bring more data and declarative power into applications themselves as opposed to sort of the remote client so everything we're used to doing akin to what cliff was just talking about the other is to bring a different model for information into database architecture in particular to include time and both these things have been enabled by the fact that computers have changed an
awful lot since we designed the databases that we tend to use with them today I don't have enough time today to do a full rationalization or overview of des Tomic because I tried I'm trying to do something somewhat technical so I'll have a little bit of an overview but I'm going to focus on the data model a little bit on how to implement a database as a value sort of a functional database and finally data log which is the language component of this this application so the fundamental thing we're trying to deal with in taking
on another another database design is complexity and I don't know how many people have read this paper out of the tar pit wow this is a great paper I definitely recommend it it's not a people we're talking about something that was done it's a paper talking about a problem and the problem that they were identifying was was complexity and software and in particular the way they characterized it was you know statefulness is the big source of complexity and also control which is an interesting aspect and what they advocate as the solution to this as a
theoretical solution to this is more functional programming more declarative programming and in their case a relational model for data being moved into the application and they had pretty concrete stuff to say about functional programming they had semi concretes stuff to say about declarative programming in particular they were talking bringing relational algebra into into into programs but but the but the the notion of this relational database or this relational data model getting updated was magic it was like you know you've seen the the classic comic or there's all this you know arithmetic on the blackboard and
then there's like and then a miracle occurs and then a miracle occurs and then magically your date has been updated by the outside world so one of the points of day Tomic is sick is to close the loop on this to actually make a database that behaves in the way that that paper talks about but it and solves all the problems associated with that because databases are shared things right they have to behave as values to applications but they're actually a piece of shared stuff so there's a lot of complexity associated with databases in it
they're inherently stateful and state is inherently complex so there's nothing to avoid about that but there are a bunch of other complexities we associated databases that we've added because of the way that we do them in particular we talk to databases we send queries over and every time you send the same query over of course you get what the same answer right no we don't we don't have any notion of a basis to our queries where the databases just don't have this kind of stuff the other problem we have with databases is that they are
over there and there are very concrete problems with databases being over there then that's come up into the surface of programming against them another problem with database is that the whole notion of updating things is poorly defined relational algebra is beautiful because it says start with this immutable thing and then all this algebra works against it but update is not not properly power relation algebra except as a global the whole world magically is different and you know how do you make that happen so I'll talk often today about basis and I just want to talk
a little bit about what I mean in particular what I mean is that when we make decisions right we have some basis for making them we have something we consider to be stable that we're going to go and look at right with our mind or with our decision-making apparatus and it may it may involve multiple looks right we may look at different parts and compare different parts and we may visit even the same part more than once I I looked at this and I looked at that I thought about this so I'm like oh I
found out this new thing is that the same as what I looked at before and as soon as we tried to make that decision making process across time we get into a ton of trouble right because we don't have a basis the other problem with basis is that if we have simultaneous actors changing the underlying structure underneath what we're trying to understand what we're trying to think about it doesn't work right if you try to think about something and you say I had this list of things and I wonder if I can get them all
for less than whatever and blah blah blah and oh I can only fit so many things in my truck let me go look at the list of things and if when you went back to lists of things it was a different list of things you'd have this infinite circular or pointless decision-making process right there's no basis for it the ingredients for your decisions change as you're trying to cook an answer and you can't think another problem we have with databases is just the whole notion of what update means is poorly defined one of the characteristic
things I think is true of most databases that update means new stuff replaces the old stuff and I'm going to I'm gonna label that place oriented programming and databases that work that way as place oriented databases there's a reason why databases work this way right this is a reason why our programming languages work this way right because our first computers were really small right they had tiny bits of memory really small disks and it cost a fortune so we had to reuse the memory we had to reuse the disk you know cliff just said disks
or cheap this were plentiful should we maintain the same approach to using them that we did when they were small and really expensive I don't think so the other problem with updates is what's the granularity of an update what constitutes a unit of change is it a record is it a document is the fact is it a cell is it a key is it a value under a key and finally the visibility thing again part of cliffs talk how do you talk about the visibility of change what can people's expectations be about what they see
and these are all things that are very difficult there are plenty of manifestations of how our decisions about these things affect our programs in particular they make our programs incorrect I mean how many people have ever dealt with the program where some programmer did not understand read committed and the implications of that or the fact that multiple reads and separate transactions in the same script didn't actually have the same basis everybody right it's a mess it's a complete mess the other problem we have is scaling right as soon as you say update in place replace
something you now have this locus of change that's forcing everybody to go there and there's two sides to it one is actually enacting change we actually don't do as much of that as you might think but the other is trying to see consistency we're putting everything in one place means having to coordinate access to it and so that leads us to a bunch of fear right everybody's afraid of talking to the database talking to the database twice to make a decision so you pile on a ton of logical activity in order to send it to
the database so that maybe you have a chance of that activity having a consistent basis and the problem is that makes for programs that are not modular right a modular program would have one piece of logic that says these are the good customers who want to be listed on our good customer page and a separate piece of program should say the good customers we should show their faces and their last date and whatever there was the display portion and the decision-making portion should be different but they're not right what do we do we say well
I got to go to the database once I'm going to go on together those two pieces of logic because while I'm getting the good customers I also have to get their names and their email addresses in there and their image URLs so we're coupling decision-making with with reporting because we're afraid of making round trips because we have no basis for doing that so I think that fundamentally this comes down to a coordination problem right we have to decide how much coordination do we want to have where do we want to do it and when when
must we have it and when do we not need require it and come and get rid of it right right now we tend to have a ton of coordination right your traditional relational database has a ton of nation is very consistent while you're there and we have you know the new the new breed of key value stores which are eventually consistent you know as if that really meant something that was useful for a decision making process because it doesn't but if we if we disentangle that stuff we can see that process that effecting change right
that's saying I want you know something to move from some value to another requires coordination requires some sort of coordination to make sure we're not on top of each other trying to park two cars in the same parking spot but perception should not require coordination but in fact does in a lot of the a lot of designs we've seen so far and it ends up as we keep finding out over and over and you know we just had and yet another talk about a completely new system you know what did cliffs say write once read
only my immutability is the critical thing to bring about new designs and it really does yield new designs so I don't know how many people were here and three years ago whatever and so let me give saw this slide and others like it okay not too many so I'm going to be using terms potentially not much in this talk that I just want to be really clear about in particular I might be clear about the notion of a value a value is something that could constitute a basis for a decision making process right a value
does never changes right and we understand values in the small is like 42 being a value we also understand more composite things now as you know strings are values suddenly and and even slightly bigger things you know if you Joe to time then dates become values you can scale that all the way up composite collections can be values and it's the it's the premise of day Tomic and this talk to show you that an entire database can become a value to your program can be as useful and as reliable as a basis for decision making
as a in an immutable string is right now so if we're gonna say values are immutable then we are going to have things like you know Sally and we're going to have a database is going to have facts about Sally Sally is an identity right Sally something that we would traditionally think changed over time but not didn't necessarily change in place right when Sally's email address changed it wasn't there wasn't a place in the universe where Sally's email address lived and we went and we erased it and we put Sally's new email address there similarly
in your memory of Sally's email address you know her old email address in her new email address there are at places for fax but we do gather up a set of facts about a thing and label the thing the label isn't the identity but the thing is the identity the thing that circles around a set of attributes that you collect over time is an identity and it's it's a conceptual thing and state we'll say is the value of an identity at a point in time and time is just relative so this is a fancy word
for that model right where we want to say entities change over time by by taking on new values all right so there's an identity that's associated with a set of values that is different over time the values themselves don't change right processes is is applying functions to one value and getting another value not changing things in place and in this in this model observation is a matter of observing values so it's not impeded by the processing part and it doesn't require coordination so I showed this slide a couple years ago as a way to break
out of some of the limitations of the object-oriented programming model and we know how to do this we know how to do this in memory with persistent data structures and trees and structural sharing how many people are familiar with structural sharing for a way of making immutable data structures right so just it scales up from linked lists to trees and once you have trees you can pretty much implement anything else sets and maps and everything else can be implemented this way but the fundamental idea here is that everything is immutable and when you want something
new you know a new composite thing you're going to share as much structure with the old one is required then generally you're going to copy a path to the root to get the next value and this is what we do in memory and it's important that you understand it because I'm going to talk about how we can't do this on disk so just like back then I showed you object-oriented programming as a violation of that epical time model a database it's it slots right in here has exactly precisely the same problems the traditional update in
place database right we program against connections that's sort of the the living interaction point with the database there's this thing over there you want to interact with it you have a connection to it and you're going to do what you're going to either send transaction requests over there in which case your transaction is a function of what it's a function of the connection really you're going to hit some random state of the current value of the database when the transaction arrives and do some job but what it encounters there is up to you right it
depends on how your transaction was written was it written in one piece and again a relational database will do a much better job providing a consistent view when it arrives over there then sending off queries to for instance multiple documents in a document in a key value store where basically you've got no basis at all queries similarly go to this black box they go to the place it's like an Oracle and it tells you the answer and and again a relational database does a decent job if you understand the serialization models and how to use
them of providing some integrity to this single question you ask at the point in time it's asked but a lot of other models don't I fundamentally I think we're suffering from the fact that if you went back right and tried to ask that question again you'd have no possible way to make a reproducible decision you would have no way to communicate a basis for a decision I just looked at the database I think something is screwy could you go look what's gonna happen who can go and look at what you saw no one right there's
there's there's nothing there so we should recognize this is the same problems that object-orientation has it conflates identity and value together right the database is a place records are places documents are places we modify it sits and it collapses time right you don't know anything except now you don't know what happened before and this is really inadequate but we know it's inadequate right how many people use file system for your source code control directory you just put your stuff in a directory Wow so most of you don't and for good reasons right and the thing
you put your your stuff in does it does it just wipe out the old values of your source code with the new values now does it have timestamps on things of course right you keep everything we're keeping everything now way all over the place we're seeing we're keeping everything but we're still using databases that only remember the last thing that we're told it imagine if you try to make decisions knowing only the present value of everything but no historical values how good a decision-maker would you be you'd be terrible so this time collapsing is something
that we can avoid because we have a lot of disk space now we don't need to do this anymore so we can plug in this same model right we can we can change what we do in memory and programming using this model the idea behind des Tomic is we can do the same thing with a database and what's different here is are two things one we've introduced something we didn't have before we had a connection before we had queries we had transactions but now we have database values somehow we're going to try to make a
system that can give you the database as a value and you can program against it the other thing we're going to see here is the lines now they don't stop at the identity right you're not asking queries of the connection right your queries are gonna hit database values you're going to be able to query a concrete value the other thing is your transactions are gonna be functions of database values to values there are no longer functions of connections so how do we do this the first thing is what is it mean for an entire database
to be a value because we know databases change right there's new stuff and the first thing to understand is that new stuff is new stuff it's not modifications of old stuff if you if you consider a database to be a value that only expands it doesn't change in place it only expands then that actually satisfies many of the requirements of a value one of the particular requirements of a value is if I look at it again it's the same okay and if you imagine not enough you can see - well if you think about the
rings of a tree right if you looked at a tree when it was this big and you know it grew later if you went back and looked at the inner part it would be the same all forever and ever and ever in other words this if you if you think about an accretion you can think about a subset as being permanently immutable and that's the way to think about a database now you can get change its accretion it's not update in place and you can look at the past at any point in time forever and
ever and ever and keep going back to it you'll never see something different so we accrete facts the past doesn't change it the fundamental problem here is obviously new facts need new space that's ok your average business if it has this much data this year is gonna have how much percent more next year not a tremendous amount unless they start doing something completely different than they did this year yes no absolutely not they don't because because so we want to be more clear about what changes so the the key word here is fact and I'm
going to talk about facts in two minutes and if that doesn't satisfy we can talk later about it but no I would say facts do not change that's the nature of a fact well so that's this no so this is what this model is this model is we're gonna model a database as a value that is an accretion of facts there are other possible models but fundamentally the idea is we move away from places right we do not change someone's email by going to the email slot in a record or in a document and and
erasing it and writing something new in that in the place so what do we how do we do accretion this is one of the tricky parts of this problem right the model I just showed before the one we use in memory which is a succession of routes right every new value of the database has a new route is something that's unsustainable it's untenable on disk although there's there are databases that try to do that because you can't afford to rewrite the the path from the root to end node every every tiny change the other problem
with a root / change is that unlike in process where you know if you lose track of a root you gets garbage collected you don't care no part of your program could ever possibly have a path to finding it in a database you will want a path to finding some older thing which means you'd have to maintain all these routes which we can't do we can't maintain and convey routes you also can't do the global GC so we have to do something different something like that persistent data structure thing but different and that we we
don't try to maintain a gazillion routes or a Ruppert change so the first thing we're going to do is make sure that the very latest value includes all the past right any value of the database includes all prior values of the database you don't have independent snapshots that you're trying to keep track of the latest version of the database is the whole database including all time you care to keep now forgetting is a different operation than updating in place you can forget stuff if you want that's a separate problem so the past is a sub
range of the present and we'll see that's important for the information well so let's talk a little bit about facts and ends up this slide doesn't say as much as I'd like to say about fact so here's the facts about facts right the word fact comes from the Latin factor it means something that happened happened it's actually derived from the past participle it really really is quite critically important a fact has time on it there's no such thing as a fact without time it's just something said in the air all right if I say Bill
Clinton is president right that's not an eternal fact he became president at a certain point in time you stopped being president another point in time those two things are facts they have time and everything else if I leave time out I am not dealing with facts as soon as you are dealing with facts you realize you don't change facts right the fact that Bill Clinton became president of a particular date is not changed by the fact that he stopped being president on a particular date that's another fact was added to the set of facts in
the universe you can go look at the université any point in time and say what was true as of you know whenever blow Clinton was president well he was president was he president in 1960 now is he president now no that's not there's not a president slot in the universe we keep updating with new presidents so that's how that's how you think about it facts have time associated with them and we want to use facts for a couple of reasons one is they allow us to move remove structure I'm not going to talk a lot
about those kinds of problems which are solved by this but structure ends up being a challenge right I talked about granularity earlier what's the granularity of changed if you had a whole record or an entire document and you just kept a last updated date on the document well how do you know what was updated on the last updated date you don't know you have you've associated the time with this giant aggregate and it ends up that the giant aggregates don't exist the giant aggregates are derived all the information to business maintains about a supplier or
a product or a person's interaction with the website is an aggregation of facts there isn't a bag for each of those things so talking about facts allows you to remove structure which makes it easier to get other parts right like accretion and where do you put the time the other thing is that it allows us to get a universal schema right which is the datum we call it the datum which is this entity attribute value and transaction which is a path to time but the other cool thing about reading of reifying a transaction is you
can keep other things on the transaction like who said it or wasn't approved but the critical thing is the facts have to include time the other thing you want to do in trying to solve this problem right this is the round-trip problem of I have a program I'd like to be mostly functional I'd like to have a clear basis for decision-making I'd like the database to be a value right is that what what is going to be what you call change right is it a request I mean if you've interacted with it with a database
over time where are your interactions can you ever can you touch them can you see them can you get a stream of them you can send the sequel strings around these are the sequel strings I said to the database if you look at a database can you tell what happened to it how it got to be what it is right can we refine I think it's quite important to do that and we've already seen this is not new snuffing I'm talking about his new stuff right we've seen this right there's an event stream processing and
CQRS and a bunch of techniques people use to try to keep track of what happened so we want to reify process we want to try to find some representation of it that is what's new there was this there was you know the database we had then stuff happened in the universe we decided was important to our business we wanted to record it what is that that thing the difference between what was and and now is what I'll call process that you want to reify and be happy something you can touch it ends up that you
can construct a minimal representation or process as just a set of assertions and retractions that is the minimal set it ends up its insufficiently expressive but it's minimal for conveyance and I'll talk about how you make it more expressive with transformations in a minute so how do we do this and the first thing to do is sort of look at what what we have okay so there's two aspects more than two aspects those two aspects today topic one is sort of the architectural deconstruction of things and the other is the data model and the functional
model and data log from the architectural side I think what you want to do is always is look at what you have and see if you could take it apart and build what you have out of simpler parts and so when I look at a traditional database server you have transactions coming in you have an i/o subsystem there it's usually in charge of a disk it's also handling queries and it has to maintain indexing but there's one way to split this apart and it's important for that coordination thing I talked about before is process versus
perception when you look at it from that perspective you see that transactions are plainly in the process category indexing is a question mark about when that happens and processes the output right processes how you put stuff into storage right perception is the other side perception is can I see stuff can I make decisions certainly query goes on the perception side and it's the read/write it's how you how you bring stuff into this decision-making process so we want to pull these pieces apart we can talk now a little bit about how to solve that problem of
how do we represent that accretion the critical thing I think for databases if you want something more than key value stores which really don't give you a lot of leverage and closer to relational databases which give you a ton of leverage right that a relational database is gets the name database for a good reason it's it provides way more leverage than a file system way more leverage than a key value store right and it gets that leverage from organizing the information right there are jobs there are kinds of data it's just piles of undifferentiated stuff
I'm gonna have to use MapReduce to deal with it but a lot of business data is not like that and the way we get leverage over the data is to organize it and we organize it by sorting it right that's that simple so what we want to do is we want to have a representation for the state of a database of facts that means a sorted set of facts and but we can't do live indexing and BigTable keep banging something sorry BigTable Google's BigTable is a good example of one way to solve this problem how
people know how BigTable works some so the idea behind BigTable is that you have a server it's in charge of a that of data in general it has a model that's based upon this on disk this is giant big flat file sorted and then novelty comes in and the same server accumulates novelty in memory and it keeps it sorted in memory it'll also log it to disk but the logged version to disk is as it has no leverage right it's just as stuff came in when you ask a question of a BigTable server it goes
and does a live merge join between what's accumulating in memory and what's already been sorted on disk and then it gives you the answer back which looks like one sorted result and then periodically it goes and incurs the cost of merging what it accumulated in memory into a new different file on disk it doesn't update in place all the files are immutable a new one on disk that's sorted that includes the data in memory and then what can I do with the memory can reuse it right now it's now it's committed now it's it's not
not durable in the meantime right it's been logging what came in so we can at any point in time it could restart load the log back up into memory and continue to merge but you you periodically merge and that amortizes the cost of the sort and of the merge because you're you batch you batch the job so we want to do the same thing and effectively what we're gonna do is exactly what you saw before which is the persistent trees right except you're gonna make a new persistent tree only every now and then instead of
every change and you're gonna keep those those trees in storage so it looks like this transactions come in the data gets logged somewhere again there's no leverage to the log it's just to make your system durable and something to recover from but the the novelty gets accumulated in memory and an index that's sorted and then periodically an index merge job will come that will take what's in memory and take what's already been organized on disk and make a new tree on disk and then that's a new basis and you can drop your live index in
the meantime parts of your program are going to be trying to make decisions in order to make a decision you need access to storage and access to the live index you're gonna do a live merge join just like BigTable did and be able to answer questions with leverage the cool thing about this is how much coordination is required for this what I'm trying to read these two things who do I need to get to help me and participate do I need to stop people writing stop transactions from happening no not at all this is completely
free of coordination the way perception really is in real life but it does beg the question as to how the live index get updated so this is a simplified view of the architecture of diatomic where we've taken apart the database and and put it into into pieces the core of it is actually up at the application level and you can consider this to be like an application server we're what we're trying to do is empower those servers to have decision-making capability locally so every application server includes a library that includes a query engine we've take
storage out of the database and we put it somewhere else storage is considered commoditized by atomic so any kind of thing can stand in as the storage services way we can use dynamodb we can use any relational database we can use Infini span or whatever you want any key value store with certain capabilities is good enough to be our storage because storage is treated as file system is by a regular database we just store blocks of indexes in it so a peer application has a query engine it has local caching it also has a live
index we just talked about which has to get updated I'll talk about that in a second and it has access to this storage whatever the storage is so it's completely independently capable of making decisions and answering queries and it's also horizontally scalable right you want more query make query power just add more application servers right and of course you can you can readily find storage services that have our arbitrary scalability so the read side of the system is completely arbitrarily scalable and the query side is arbitrarily scalable and those are real queries that have sorted
data that can you know give you algebraic kind of query capability then on the lower left we see the final component which is transaction coordination so we took we took query move that out we took storage and we move that out and we're left with the server that has a traditional availability and scalability model to an ordinary database right so what we're doing is we're making two independent decisions because we separated process and perception we can make a highly available highly redundant decision about reads and make a broom or traditional availability done via you know
hot standby decision for rights but the notable thing is that that right server has nothing else to do it doesn't process any queries it's not getting any query load it's only getting right load in this diagram indexing is done on the same box because for a lot of people that box has got plenty of time to do that but you could move that on to a different box so that's the general model and we can dig in a little bit the memory index is precisely what I showed you before it is a traditional persistent sorted
set it's a tree it has large internal nodes it can store any java data has pluggable comparators there are two sorts that are always maintained one is primary by entity and the other is primary by attribute which means it has both the characteristics of a document store and a column store via those two things and then optionally you can get attribute value indexing or and automatically if you're building reference to reference relationships you'll get value to attribute mapping so that's what's happens in memory oh wait forgot one critical thing the other thing that trans actor
does this line here probably blinding Brian over there it is okay the other thing trans actor does is it receives transactions it logs them to storage and also reflects then back to peers so all the peers are maintaining the same live index and all the peers will drop their live index every time in indexing job runs so that's the memory index there are two things kept in storage one is a log right so everything it's adorable system everything that comes in gets logged again that log is a low leverage thing it's just as it arrived
so it's not a lot of leverage in the log but it is there for durability purposes and then there are a bunch of indexes and and these indexes are fully covering indexes in other words they contain all of the data update UM's they're not pointers into other things right there so they're fully covering indices they have all four fields of a datum in every index and that's what's kept in storage so the I said before the day Tomic system treats storage the same way a database treats a file system so the storage doesn't have individual
facts stored in it it has big chunks of segments of a tree of an index tree and we store them in key value manner in any of those data storages so all we require is key value store we require to other capabilities of a key value store one is consistent read occasionally we need consistent read and finally we need conditional put for the log which is probably the highest difficulty constraint on the storage system obviously any sequel database can satisfy these requirements and so we run on top of any sequel database but dynamodb also satisfies
these requirements which is an example of a cloud database what's fully distributed and redundant and SSD backed and everything else in in-memory a database looks like looks like this and I don't really want to dig into this too much right you can see though there's the memory and then a path to the subsystem that's backed by storage it's a it's another tree with a bunch of local cache and then finally a lazy load from storage so if you if you get a local cache failure you'll try storage and actually we allow memcache to sit in
between there and automatically manage that so it's it's what I would call caching under instead of caching over which we typically do with databases so you can pervasively cache this stuff because it's all immutable and the final part of the memory model is there is some structure which is going to be updated every time you get a new set of process coming across but that just sits inside a closure atom in this case but you can think of that whole thing as Kaz Kaz is fast enough to do all the changes that happen to a
database you don't need anything fancier than that so there's actually one mutable thing and this whole thing on the client-side one it's all you need the rest of this is all immutable so that implements the model I showed before in storage it's it's what you expect right it's another tree there's it's actually a set of trees one for each sort and then there's you know just high branching factor large it's like a B tree except you're not saving space and you're not you know changing stuff in place but at the bottom at the leaf level
are just blocks of sorted datums that's the way that's the way the index works in storage as I said before process is represented by assertions and retractions but they're insufficient obviously you can't increment somebody's salary by asserting a new salary right that's racy you really want to be able to say this this change I want to make as a function of the current value and so we have something called transaction functions which you can send across to the trans actor and get run inside the transaction on the value of the database they actually get passed
the database as a value as an argument not a connection the database as a value the picture I just showed is passed as an argument to your transaction you can go look up whatever you want you know it's completely stable and you can think about as long as you want and figure out what you want to do and then the process happens through a technique we call expansion so a database function takes the database and maybe some arguments and it returns more transaction data so transaction data is assertions retractions and calls to database functions every
call to a database function is going to be called and it's going to yield assertions retractions and maybe more calls to database functions until there are no more calls to database functions and then we have assertions and retractions and that represents the factual change we're trying to encode by the transaction so we keep splicing until it bottoms out and that that looks like this alright so foo might turn into calls to to other database functions which might turn into sets of sources protections by the time we get to the bottom here that's what's going to
end up in the database those assertions and rejections and that's what's going to get reflected out to the peers so we have a couple of named pieces to this to transact as I said it accepts transactions it does that expansion process it applies the change to the database value it logs it to storage only then will it acknowledge it so it's a fully synchronously dorable system then it will broadcast that to connected peers an unconnected peer a pair that loses connectivity can always restore its live index from the log directly from storage because peers can
read storage currently the transact also does this indexing in the background index and creates garbage right we're see we see and the whole model is just so funny here you know the memory Java memory model being you know distributed now they the same GC problems and you know this acquisition of novelty of new memory right and then reuse by garbage collection it's the same model using memory we're gonna start using on disk and we should we have giant kantak discs now so index and creates garbage and we have GC to clean that up the critical
theory the critical thing here is all the peers have direct access to storage so they have their own quarry engine they can do this merge themselves and then there's this caching so there's a cache in memory with objects already fully expanded that's on heap and then there's this cache of segments which are highly compressed which are kept either off heap or in memcache and memcache is particularly cool because that's also distributed and shared so now against the language part of the talk there is programming aspect to this there the query language I chose for this
is another old thing called data log it was heavily researched and was a precursor to a lot of the work that ended up the optimizers inside the big relational databases in fact when those guys got hired suddenly no more papers were written about datalog it's quite stunning in the mid 90s it just vanished but they all got good jobs at Oracle and IBM and I still have them so we embed data log again this is going to run in every application server it's a subset of program Prolog has a logic oriented flavor but what's cool
about it is it has a set orientation unlike Prolog which is tupple at a time and it has guaranteed termination Prolog has all kinds of funkiness around clause order that way and so add a data log program consists of a database itself in our case more than one potentially a set of rules which you can not you can map to views in your mind and queries the data log that we we have in particular fully reifies everything so a lot like when you write a sequel query what's the database it's kind of implicit right you
never talk about it it's sort of an ambient property and and and it ends up that databases and table names and column names are not first class everything is first class in day Tomic dialog you pass databases specifically which means you can pass more than one database in you can write queries that operator on more than one database you can also pass because the query engines in memory you can pass collections that you have in memory to the data log and any combination thereof so you can do joins between an in-memory collection and your database
no problem some x been extended in that way you can call your own code with expression clauses as I'll show you in a second the other thing is the interface to data log is completely data-driven so you'll see some syntax but the fundamental interface is Java util lists and Java util map not strings and if we look at a little bit of data log the first line here is just what a call looks like from Java you say connection Q which is query you pass the query which is just a data structure and whatever data
sources you have could be databases could be stuff in memory could be scalars as you know just argument values or constraints and then a query looks like this latter block here which is is this is just the syntax that we have for the data structures but you can read this as a map where each key maps to a list and so we're saying finding the customer product and we're grabbing a shipping address in zip code and everything else you'll see there's no joins here all the joins are implicit so it's very pattern e if everybody
anyone's ever looked at sparkle the RDF quarry language that has this pattern thing or any kind of pattern matching in programming languages has the same kind of thing and unification unifies all the values of customer to be the same customer or product because same product I guess is the joint so the joins are implicit and the order doesn't matter and then this shipping estimate is a call to a Java static method so you can call your own code and you know this is now what I talked about bringing data driven programming completely up into the
application space so to wrap up I just want to talk a little bit about the experience dealing with JVM here this is a JVM app the trans actor is a JVM process the peers or JVM processes and the live the query engine and everything else is delivered as a Java library it's definitely a Java program it's written in closure but the api's are all Java and even the closure API is written in terms of the Java API there's fundamental API is Java there's really not a lot of news here right GC pauses our killers and
we desperately need pause list GC like everybody does but but the other thing that really hurts daytime dialogue for us is box numbers we really would like tag numbers it writing a database engine that uniform we can treat all data structures including you know strings and dates and numbers is really what you want but numbers are just have such overhead and of course we have to do the normal tricks people do because of lack of structural types I'm actually I don't care about this but we do do struck some arrays instead of a reason structs
and things like that on the Interop side I think we we've encountered a bunch of things I think matter for Java and matter for interaction we've talked a lot about how do we get jvm languages to talk to each other and call each other but so we hear what we've done is we've written an API in terms of fundamental Java collection types now I like the fact that there are fundamental collection types and enclosure in particular unifies all of its collection sites with the Java interfaces and I think we can derive a lot of value
for that but we've already seen some languages walk away from that and I think it's for a number of reasons but one of the reasons and one of the pain points I found was that the interfaces to these data structures is way too huge right if you want to say I have this new thing and it's associative there's no way to say that you have to take on all of Java util map which is enormous similarly if you just want to say this thing is indexed by integers you can't say that there are no interfaces
but you could easily superimpose them on top of what you've got and if you did you'd have a lot easier time making Java data structures a lingua franca for data amongst languages and I think we really need it the other thing is just as a consumer of an API written in Java in terms of Java's own collection types it's incredibly difficult to use them from Java there's no literals no accessors I mean it's so much easier to do this stuff from any other language any other JVM language than it is from Java another thing is
that there's no symbolic type but I'd have no hope for that so to wrap it up this model overall there's a lot more to it I'm not talking about today models that epical state I showed you before this coordination is used only for process it's only for novelty coordination you have stable basis same query same results if I see something funky I can convey to you with along the basis of the time at which I saw it and you can go back next week and see precisely what I saw and say oh yeah you're right
we had a bug in our program we should fix that or you know we need to issue a credit credit to that customer because that was that was wrong so we have stable basis the idea of a transaction is well-defined right a transaction is a function of a database to a new database these things matter tremendously to making programs that just are correct other things well I talked about the communicative basis you can freely change where your storage is or what your storage is you can in day Tomic you can start with you know an
embedded h2 thing you can say back up to an s3 bucket and you can restore in to DynamoDB you just don't care about that it's become completely orthogonal that kind of storage how you where you put it how you talk to it is completely orthogonal by taking it out another very cool thing which I didn't get to show you but sort of implied by keeping everything is the fact that you can go to the database as of any point in time you can say I have a database value you just gave me I want to
see that as of last month once you've done that you have another database value in hand you can pass to the same queries how many people maintain tying themselves in databases now yeah and when you want to ask question about last month what do you have to pass to every query and every sub query last month it's all over everywhere and it's a nightmare to do that plus those queries are they fast no they're terrible I mean they're really really difficult to make fast so it's very easy to do that you can issue a query
against the database as of right now and joining it to the database from last month no problem the other thing about reifying process is you know we saw that the process gets pushed from the trans actor to the peers that's Dec turns into an event stream it appears you can just say I want to listen to that and I want to bind a query to it and every time that query returns something I want something to happen all the stuff you'd have to build inverting control you don't have to do that anymore because processes become
first-class and process transmission has become first so I think that the the biggest value proposition in the end is is the making the database of value itself all right this just completely changes the integrity of the kinds of programs you're right against databases there are a lot less complex you have a lot more power in terms of doing more with less work the system is much more scalable because you've taken the reads completely out of it and the information model is a true information model it's an information model that incorporates time and incorporates backs and
that's what I've got for today