hello everyone my name is maxim fetiv i am presenting from bellevue washington i'm originally from russia where i started physics and later i received my computer science degree from rio de janeiro brazil today i'm going to talk about temporal um i want to give you some history um to kind of give you idea that it's not the project which we came up with overnight it took probably like at least 15 years to get where we are so i was tech lead over amazon messaging platform when amazon was transitioning into microservices and i i was uh
later that engine used for the amazon messaging platform was used by the simple queue service when aws appeared uh i also was technically simple workflow service uh which is kind of was the spaniard of ideas that we right now development is part of temporal my co-founder of the temporal samar worked with me on a simple workflow but later he came to microsoft and he was tech leader of the service bus azure service bus but as a side project he wanted to bring to the people the same ideas of the simple workflow so he developed durable
task framework which later uh became so popular that azure functions team adopted that and now you you can use it as azure durable functions uh later we both joined uh uber and there we built uh two open source projects one was jeremy which was a messaging system and later cadence which was implementation with simple workflow ideas using completely different software stack and as an open source project when cadence became very popular not only inside of uber where hundreds over 100 use cases were running on that within three years but also outside uh we uh quit
uber and started our own company called temporal uh and fourth uh candidate's project as temporal as an open source project i was told that if you kind of present in some technology you have to give some social proof and here's kind of some of it so these are early adopters of our technology and there are many more companies using that in production but these are companies which went public about their usage uh i give presentation into a multiple presentation if you google my name uh you will see a lot of videos about which present uh
cadence and later temporal uh and their programming model but today i want to talk about the back end about the how cluster is organized and why we have all these multiple roles of the class uh in our architecture and i don't expect you to understand that picture right now but i'm pretty sure by the end of the presentation you would be able to make more sense out of it but let's start from the beginning what is uh workflow and people have different definitions there are like even standards and so on uh my definition is that
buffalo is just a resilient program uh and resilient what it means that this program will continue execution uh independently of uh like in presence of failures and you can have new deployments you can have just infrastructure outages the data like availability is always going down and you wanted your program continued execution usually this program is organized as a like sequence of tasks and then it also has to react to external events and deal with time because time is important component of every business level process and timeouts i'm not going to spend time as i said
a lot on the programming model as i presented that multiple times but if you never thought about temporal locations uh here's the kind of basic idea basic idea that you just write code here's java sample we also support go and php sdk and working on other sdks in the future at least typescript and python will be probably developed this year and this looks like normal java code the give why is it workflow it is about flow because a full state of this program is fully preserved all the time it's fully durable so it means that
if your program restarts or even back-end service goes down and then back your program will be in exactly the same state with all local variables frets and stack traces in exactly the same state that's why you can go and write code we just call sleep block and sleep for 30 days and then block and do something after that and if you you don't need to care about process restarts and deal with databases and recovery of the state because it's all done automatically by the temporal and again as i said i'm not going to spend more
time on this if you're interested to watch my past presentation or just visit temple.org and go through our tutorials so on a high level any workflow engine and again it's not necessarily one which uses for flow s code but also engines which use yaml jiso and xml if you are like those languages they are kind of state machines so frankly workflow is a state machine which defines in which order to run tasks and also react to external event if necessary and if we represent the state machine kind of what workflow definition does as a sequence
diagram we can practice a to ask us to execute in certain order uh but uh if you want to implement the uh workflow engine we need to have this was like more logical uh representation uh but a physical representation will be some sort of code some engine which will drive those workflows so the way it's usually done is uh that you will go to workflow definition uh the state machine gives a state of the world to say what are the next uh tasks commands to run and the command can be run task one so it
will go and run command task 1 then it will enter 5 of the definition that task 1 is completed it will execute the state machine logic according to its definition and say okay now you need to run task 2. in practical systems you don't want to call tasks directly because these workers can be somewhere else they might be not available which run those tasks and processes they can be issues with flow control because just you don't you don't you uh and then can be unavailable for some time so you or they can can be slow
so you want to fit tasks in appropriate order so using cues to this page task to uh the implementation is a very common technique so you probably every almost every practical workflow engine uses queues to dispatch tasks to workers like our working processes which host those tasks and temporal at least we also run workflow code outside of the core engine so we also use tasks and queues to deliver uh the request to workflow definition and then get commands back what to be executed so practically all communication to workflow definition and task implementation is done for
queueing as i mentioned the time is important because every task in vacation should have a timeout a book log itself can have a timeout and then we need to be able to execute operations like sleep 30 days so we need some external uh timer service or timer queue which will durably store those timers and then dispatch those timers so workflow engine needs to have not only task use but also timer queues or timer engine and tasks there every time some state transition happens and then we also need to store state of the workflow itself so
every time we start a workflow uh every update practically we need to at least to create state in the state database then we need to create a timer and then we need to put task for the task queue for the offload definition to pick up and when workflow definition decides gives a list of commands we again need to update and create tasks and update the state uh what's important to understand and i think this is like a very important part of this presentation is unless these updates across this multiple data sources are transactional this engine
is not very practical you don't want to run on top of the engine which doesn't have transactions across all these components because you will run in all sort of phrase can you it's almost impossible to get rid of race conditions because if you update state and then try to put tasks in a task queue if update goes through and then update to task q fails so you will end up in the state which thinks there is a task outstanding but task is not if you put task and it asks you first and then you update
the state if update is slow task can be delivered and processed by the time update when uh like is it is going through so it will be inconsistent and it needs to deal with all these edge cases uh so uh if updates to all these components is transactional then uh all these trades can use just disappear uh your system becomes much simpler and the application level code doesn't see all these edge cases and it's actually simplified the programming model but it's not only applies to the state measure to the uh workflow engine implementations the reality
is that these days majority of the engineers don't use workflow like orchestrator or like workflow engines to write their uh services what they do they use queues they use databases use other data sources they create these hedgepodge architectures of like different components and there are no transactions across them usually and the point is that the like what i said about not having transactions all this race condition applies to majority of the system ad hoc system built by developers and i think this is important very important to understand that this is uh if you're building system
from these components you you're guaranteed to have edge cases in race conditions um so that's why having like a robust engine like temporal underlying architecture will simplify your life tremendously so if you want to remember one slide for the presentation remember this one that workflow engines are hard not because uh like mostly because they they need to deal with multiple things like skewing timers and state and updates across them should be transactional so when you implement a workflow engine because of those this transactionality requirements majority of the existing implementations just use one database and even
one process and then yes if you have one database and one process transaction requirements are easily solvable and you can provide pretty uh robust engine obviously this engine wouldn't be scalable because it will be limited to a single process to it or a single database so if you want a scalable solution first we need to decide what are the scale skills dimensions of scalability obviously one dimension would be creating these huge workflows and have a single workflow which spawns multiple machines for example map produces a technology which allows to write single kind of mapreduce pipeline
and this pipeline will be executed by thousands of machines and you can think of this pipeline being more as a single workflow uh for the use cases which we are targeting we actually decided to go with approach when we're not going to scale up a single workflow instance you cannot have single workflow which runs million uh tasks uh every workflow should be limited in size but then we can infinitely scale out the number of workflows so you can if you need to run mean and task you probably will have single workflow instance which will create
a thousand children each of the children can run thousand tasks you get your million tasks but each of the instances will be bounded so as soon as you say that uh workflow uh each workflow instance is a limit of limited size you can go and start distributing them across multiple machines and so scaling out a fleet of machines becomes practical because each each instance is guaranteed to fit within a single machine obviously if you want to have very very large system you need to scale out the database as well so uh having single database instance
will will be a bottleneck so if you have a partition database and you have partitioned a host which you can maintain the state and perform all the operations and workflows you can get to very uh scalable solution but as i said we need to maintain transactionality and as soon as you start breaking uh your resistance into multiple databases you wouldn't be able to unless you start doing uh complex things like two-phase commits and so on uh across multiple databases you wouldn't be able to provide transactional guarantees so uh the kind of simple way to solve
that would be to have one database per host right so if you have four hosts we can distribute this workflows across those hosts and every host will have its own database which contain the full like its own queues its own timers uh and so on and workflow state and this way uh we can guarantee transactions within each uh host and within each database and uh that will be pretty straightforward system to implement that system wouldn't be very practical because it would be very hard to add and remove hosts and i'm not going to spend much
time on that the standard way to solve this is just using charging so instead of physical hosts we use partitions within the database and and we can overlocate number of partitions then we can move those uh then we allocate those parties to specific physical hosts and we can move them around if necessary and the same applies to shards within our hosts so you can catch workflows to specific shard id and then use consistent hashing to uh put a special to to allocate chart to a specific host but to implement that you need to know membership
of your cluster because if you if you need to locate charts to host you need to know which hosts available in your cluster so as soon as you do that you need some mechanics for for a membership and then you need routing layer because if request comes in and usually you don't want to have flat client side libraries understand topology of your cluster so you need to have uh frontends which will uh know the membership of the cluster and route uh request to specif for example flow id one workflow id2 it will know it will
shut it it's allocated and it belongs to and then where this chart runs right now uh at we at which host the starting by workflow id works uh problem is that it doesn't work very well for uh for tasks in uh for task use for example if you have activities which listen on the task you named foo how do you get activity tasks waiting to be executed if the uh we store those activity tasks in every chart you probably will need to go to all charts and ask them do you have anything for full on
task you uh task you foo as we want to allocate a large number of shards because we want to spot very large scale and we will relocate them uh these type of queries become impractical even if you have them um like you cannot even aggregate them over host because uh each chart practically requires separate database query imagine you have 10 000 shards so every time you do pull you will need to go and like fan out for 10 10 000 database requests so practical solution is to move q into a separate component with its own
persistence and that solves the problem of routing because you know where to which host to route and which host right now owns that queue and you can have its own database with its own resistance for that but uh it obviously introduces other problems as i mentioned earlier is that we lose transactionality as soon as cues live outside of uh core charts of workflow state we don't have transactions across them anymore and we will have the risk conditions which i described earlier obviously one way to solve that would be uh to use some sort of two-phase
commit uh variant paxos or raft and so on but it would introduce pretty significant complexity to the system and uh also requires participating like every component of that interaction uh to implement pretty complex protocols uh the way we sold it in temporal is that we don't use uh any of those transactions we just align database how do we do that we use so-called transfer queues the idea is that every chart which stores workflow state also stores a queue so we will have if we have 10 000 charts we'll have 10 000 views and every time
we make update to this chart we also can update that queue transaction because it leaves in the same partition so if we need to start workflow we will create a state for the workflow and we need to create workflow tasks for the worker to pick up uh we will add the task to the local queue to the that chart and then uh this will be committed to the database atomically and then we will have a thread which will pull from that queue and then practically transfer that message to this uh queue in subsystem and this
way we we have transactional commit and then we can later transfer that to the queue subsystem obviously this transfer can fail we will retry so it can create duplicates but we have separate part of the system which will do the dupe in there so so that is kind of very simple mechanism but it allows to uh have transactionality at the same time don't rely on complex two-phase commit protocols other requirement that workflows any reasonable ecosystem should implement is ability to list workflows right because you want to go and say give me all workflows started by
this user in the last 24 hours all workflows which failed and so on uh going again to all charts and 10 000 charts and asking them for even if they have index each of them has its index for that information would be impractical so the way to solve that is to have a separate indexing component we use elasticsearch there is no it can any indexing technology could be used instead there is nothing special about elastic that but it's something which we just decided to use because it was open source and we use the same uh
uh transfer q approach to transfer uh to commit uh so-called visibility records into the h chart and then use transfer queue to transfer them to the elastic uh this mechanism uh has inherent delay so elastic index is always a few seconds maybe like at least some time behind the actual update so it's not it's kind of uh converges to the correct uh this delay but it's not uh um atomically updated uh with uh when they commit message uh to the core core to core chart uh partitioned but there is a guarantee that uh if a
commit happened that elastic will be updated and uh if uh cannot be it cannot happen that you will update the state and elastic wouldn't be updated because of the transfer key mechanism as soon as you can list workflows you uh your users will ask you to perform base operations for example i want to terminate all those flows which measure this criteria started by this user and you can program that from outside just to make a list get the ids save them to the file and then execute some script to terminate them but to execute that
your library this is actually sounds like a workflow and this is exactly what temporal is doing it implements this logic uh using a workflow and but this workflow is built into the core cluster so it's kind of system workflow uh user and it's implemented as normal workflow it could be implemented outside but we decided to bring it inside because we want to this kind of functionality be provided as part of the core cluster functionality so we have this walker role which performs system workflows like database scans and other like operations which can take long time
reliably uh using standard temporal abstract workflow abstractions i showed this slide at the beginning of the presentation and i i didn't expect you to make sense out of that but now i think you can understand why we have all these roles so we have so-called history component uh which uh is responsible for state transitions of individual workflows has transfer queues to be able to transactionally update uh update uh create tasks and uh then we have a matching component which is responsible for uh delivering uh tasks to uh like for queuing and mention uh pull requests
coming from uh external workers to add task requests and then we have front end because we need routing we have elastic because we have indexing and we have water component to implement background jobs and then we have workflows and activities which are implemented by application developers using temporal sdks this architecture is uh pretty uh as scalable and reliable because every single host in this architecture can fail and still will continue functioning you can uh because charts and history and service and mention will be redistributed automatically we use clustered database like cassandra which can sustain uh
not failures elastics also is full tolerant but um and front ends are stateless so you can add and remove them any time as well but this system still has a single points of failure in terms of like blast radios because from single bed schema deployment to the database can bring it down update of the front end with the bug and so on so if you want to provide very high availability we have multi-cluster deployment in this case we have a synchronous replication replication level and uh even the total meltdown of the cluster or even like
for the amazon region um unavailability of region will not stop your workflows because you would be able to fill over your execution to another cluster and continue execution uh this is for the next talk i have it's pretty complex uh part of the system uh the the whole multi-cluster cluster setup i can make multiple multiple conference talk about that as well just to recap temporal uh user technology uh besides this awesome programming model of writing a distributed code like a reliable code through like using the programming language it's a highly scalable and scalable number of
workflow execution and it's consistent uh which is very important to avoid all sorts of phrase conditions and it allows to run uh pluggable uh activities and even workflow implementations outside of the cluster which allows very very high like flexibility right you can practically write your workflows and activities and then you're like in the future and producing directly in the language this concludes my presentation uh i want to mention that we are actively hiring and you can reach our project on uh over temporal io or directly on github we also have kb forums for support and
you can reach me or our company on twitter thank you