[Music] okay thank you everyone so much for joining us today we're really excited to be here talking about um building Saga patterns with temporal and um I'm gonna just give a few words of introduction before we get started so my name is Megan I'm your host today I'm on the product marketing team here at temporal and I'm joined by Steve who's on our uh Solutions Team so Steve do you want to give a few words of intro hey um yeah thank you Megan um I'm Steve I'm a principal solution Architects here at temporal um I'm
based in Seattle uh originally from Australia awesome well thanks for joining us today Steve we're excited to hear from you so a little bit more about what to expect from this session so we're going to start out with a brief intro to Saga patterns then we're going to jump into some of the common Challenges that people face when they're developing ing Saga patterns um how temporal helps and we're going to finish up with a section on how to implement Saga patterns in temporal and Steve has a great demo queued up for you all um to
to dive into the weeds a little bit so with that I'm going to stop sharing I'm going to pass the screen over to you Steve and let you take it from here okay thank you here we go all right Megan uh you see the the Slides yeah we're all good okay perfect all right so what I want to do here is give a very brief intro uh to The Saga pattern um just to get everyone on the same page uh so what I'm going to do here is what is Saga all right um it's a
development pattern uh for business transactions all right uh it allows distributed system to ensure consistency um so the example I'm going to use throughout this this talk um is of an Autom management system but it really could be any business or non- business transaction any series of steps that you need to to happen um and succeed uh and so the thing that's unique about Saga is um even if these steps this happy path here in the in the top half the order service the payment service the inventory service or the delivery service even if uh
the steps don't succeed or particular steps fail um you want the system to end up consistent at The end and the way that happens is through a series of compensations or rolling back um steps in in the flow or in the transaction um and so why why Saga to achieve this well um it's a way of Designing for Success so you're coding really for your happy path up above you're coding for things to go right um but you want to make sure that any unrecoverable failures uh are compensated for In other words are taken care
of and the reason for that is you want your system to end up in a consistent State regardless of whether these operations succeed or fail all right um what I'm going to do next is let's look at a couple of common architectures for executing transactions across distributed systems so kind of microservices setups and why implementing sagas um using these common architectures um can Be a challenge right um and then the next thing I'm going to do is show you uh give you a brief intro to the temporal and then show you how sagas can be
done in temporal all right so the first architecture here is a choreography microservices architecture um and in a choreography architecture each service decides the next step in the transaction um so for this example uh there's a new order that comes in and The order service takes that the order service asks the payment service hey please process a payment for this order um the payment service is going to say please reserve inventory for this paid order it's going to ask the inventory service for that the inventory service is going to complete um that step and then
it's going to ask the delivery service hey please send this order out for delivery okay so each service is contacting uh another service in turn um The benefits of choreography is that you kind of get this loose coupling each service really is only looking after What It Takes the request it takes and the subsequent action right um it's only responsible for that and so it's easy to maintain each individual service and there's also no single point of failure if one of these Services go down the other services are are more or less unaffected right but
um there are inherent risks to choreography um so It's it's harder to create and update a choreograph service so let's imagine we add a step between payment and inventory here like an extra step in this um Auto order flow um now you have to edit the payment service to request from a different service you have to edit the inventory service potentially to accept a request from this new service sitting in the middle um and so it kind of inserts uh the need to create um to edit lots of different uh disperate services And make sure
they uh succeed and and on that on on success it's actually harder to monitor this kind of system because there's no one source of Truth for State because each uh uh service um is taking care of uh one particular task there's no way for you to sort of ask what is the state of this overall um and have that given to you by a central place and so it's harder to Monitor and then also debug when things go Wrong um another risk here is that uh let's say the delivery service is temporarily down um you
your inventory service should really be retrying that like let's assume it's just API downtime or something like that your inventory service should be retrying that request okay and um that means the inventory service needs retry functionality but then of course every service needs this fun functionality so what you end up doing is you end up building kind of Sophisticated retry mechanisms into disperate Services which again is additional work and that's another downside of choreography um how does saga look in choreography well just like the Happy path where you had the order service contacting payment payment
contacting inventory inventory contacting delivery um in a in a choreography based Saga each service decides on the next compensating step in a transaction so to take the same Example delivery service fails and let's say the delivery service fails unrecovered so we're not waiting for retries um let's say the delivery service gave some kind of junk response back something is wrong right we can't proceed with the order the delivery service needs to ask the inventory service can you restock the inventory right that's the first step in the compensation the inventory service goes can you refund the
payment it asks the Payment service and the payment service might say can you revert the order and then that may trigger an email to the customer saying sorry we couldn't process this something has gone deeply wrong right um that's your Saga so just like the Happy path um where each service contacts the Ser the subsequent service the uh compensation scenario or the roll back scenario um goes back the other way um a downside to that is that the Same risks apply um you can you can end up in in such a strange state so for
example delivery service is down let's say the delivery service uh wants to restock the inventory says okay let's start rolling this back let's say the inventory service is temporarily down okay and let's say we don't have retry mechanisms well what's the state of the system if I was to ask uh what is the state of this order well if I ask the delivery service the delivery service Thinks it's failed the inventory service doesn't know anything it's down the payment service and the order service think it succeeded because it took an order and uh a payment
succeeded at that point so we end up with this kind of weird System state and this is kind of what we're trying to avoid as developers and kind of operators of it systems the worst case scenario really is is um an inconsistent uh System state all right so moving on and this is The the the final kind of development pattern or architecture all show um is orchestration okay so this is an alternative to choreography it's very popular um an orchestration architecture um and in this you have a central orchestrator that controls the flow of transactions
and compensations in in The Saga case um so instead of each service contacting the the subsequent service we have this Central orchestrator and in this case it's the aut service Orchestrator and that auto service orchestrator has the job of reaching out to uh each service waiting for that to succeed and then contacting the next service in turn the great benefit of orchestration is that it's easier to code and maintain like that's one of the main benefits here um writing code writing control logic in the one place to control the entire transaction is much easier than
you know doing it in several places like the choreography Case um it's easier to debug and monitor right so the problem that we talked about with choreography where you know what is the state of the system is actually fairly easy to to work out in orchestration all you do is ask the order um order service orchestrator hey where is this order up to okay are you up to inventory uh um res reservation are you up to uh sending delivery out you you have one place to kind of ask because there is a single Central Orchestrator
um but you get additional RIS risks with orchestration um or the way it's traditionally implemented so um scaling is often limited by orchestrator uh capacity okay so now everything's going through the central service and that means that Central service has to be Rock Solid in it has to be reliable it has to scale uh for sale periods in in this order management case right this e-commerce kind of workflow um it has to survive Black Friday um and you also need to still code for retrice you're not going to get out of that you need to
make sure your orchestrator is equipped to um cover for the case when a a downstream service goes uh temporarily down and you also and the thing I haven't mentioned yet and this is true of choreography as well is you often have to deal with cues um Eventing and message cues and that kind of thing if you're looking for scalability people uh Start to implement cues and that makes coding harder overall it just makes it a little more abstract uh as well and so um so that's an additional layer of complexity and then when we're talking
about um Auto Service orchestrator here um if I was to ask the state of the system uh is that going to be in a database do we have a state database for that are we adding additional pieces here um and finally this is this is the the biggest risk of orchestration um to My mind you've got a single point of failure so if this order service orchestrator goes down or a workflow has a bug like one of the one of the orders has a bug um how does that how is that rescued I mean that
may bring the entire system down um you may not be taking any orders so with choreography you had this kind of disperate scenario where a service isn't going to bring down the other services um in this orchestration case if if your orchestrator goes down Your business might might stop and uh Saga uh the good news though about orchestration is that Saga are are fairly simple um just as laying out the happy path happens in one place in that orchestrator um laying out the Saga case is is also in a single place um in other words
if the delivery service fails in an orchestration scenario the orchestrator is going to take care of that that roll back or those compensations um it's going to say can You restock the inventory can you refund the payment it's going to do it in a way where you have one place to know how that went and and so it's good for a a system kind of consistency perspective um so given all that background choreography orchestration sagas I mean how how can temporal as a technology help um how can it help um you code sagas uh in
an easier way like create sagas in an easier way um how can it help you you do it more reliably and In a way uh that you get the insights you need to to know that you know these these transactions are succeeding um and so you get you so you get in insights and um the ability to to debug if things do go wrong um so I'm going to spend a moment explaining how temporal works okay so temporal is orchestration it's a new kind of workflow orchestration tool um temporal allows you to code uh reliable
um scalable business transactions without having to Worry about cues and events um it eliminates your requirement to add complex logic around retries you get that for free um or it allows you to avoid um complexity around roll backs or compensations or state you know managing State machines um I like to think about we have sort of an unofficial tagline it's not an official tagline but I always think about it which is a code like you're in college um you just have to think about you know the happy path And then the roll backs if things
go wrong you don't have to think about State machines you don't have to think about State databases you don't have to think about um cues or events um it AB temporal abstracts all all of that away and I'll show you how in a second and so you're going to get all the benefits of orchestration you get a single place to code your control logic you get a single place to monitor how that's going right that's those transactions are going um Without any of the drawbacks you you don't um lack uh scalability you don't have the
single point of failure issue and I'll also explain how that works in a second all right so how does temporal work um well the first thing you do is you you choose one of our sdks and there are many um and you create a workflow um a workflow is a central Concept in temporal and it defines your endtoend um process or control flow that you want to execute um on the right here I have some pseudo code for it's no particular SDK or language but I have um pseudo code showing this order management use case
and a workflow for the auto management use case at least the happy path would look like process order and then run process payment Reserve inventory deliver order it's that simple you just lay out the steps um using the temporal SDK again you you compose uh activities that will Be part of this workflow um and so activities are where your tasks uh are completed and and these activities can do anything you need um people often use activities to wrap uh API calls for example doesn't have to be but you know it's a very common case to
have activities wrapping API calls to your services um temporal activities have retry policies set for them um ensuring that they succeed when Services temporarily go down so that's the a Really big benefit of activities is that um you do have these smart retry policies for when systems are temporarily down and so you get that for free with temporal um and so to kind of summarize the the last couple of slides work FL control logic activity the steps you need to run and again uh that's my order example on the right um once that workflow and
those activities are sort of coded up um you deploy all of this in a temporal worker Um this worker is going to run your application um it's going to capture the state of execution of those workflows um and the critical thing about a worker is that it it runs in your environment and it requires like no change to how you build and deliver code um the only difference is you need to import a temporal SDK and so your workers these workers are really just lightweight processors that that run your code on your data in your
Environment and the really big piece about temporal here and this is probably the most critical technical slide about temporal um in this presentation is that um your workers that you've coded running your application running your workflows they take direction from a temporal Service uh for what to do next um the temporal Service uh including um temporal Cloud which is our software as a service product um it captures the state of your application and it's going To deliver tasks uh to your workers or make them available for your workers to consume um the temporal service is
is where um timers and retry policies and your application state is going to live okay so it's this kind of um this orchestrator um I like to think of the workers as constantly polling the temporal service asking what do I do next what do I do next so if you think of workers asking the temporal service what do I do next on masse that's kind Of what temporal is doing um if your app is running in a worker and it crashes for any reason um or let's say the infrastructure that your work is running on
goes down like the the cloud infrastructure or the compute behind that worker um another worker can actually pick up that work that exact work and resume EXE ution of a workflow without any without you doing anything um and the way that happens is uh that worker is going to ask the Temporal service hey um what was that where were we up to in that workflow and just continue carrying out that workflow like nothing ever went wrong um and this is this is a massive value proposition of temporal um it means that workflows are durable Beyond
um any individual process executing it so it can survive any worker crashing any worker dying any infrastructure under it going down um that's true of your workers and also the temporal service including temporal Cloud are both designed to be horizontally scalable and fault tolerant um there's also a temporal Cloud which is our pay as you go kind of uh um product uh as sass product is has a feature called multi- region namespaces as well and uh that means your workflows can actually survive an entire region going down um and pick up like an entire Cloud
region and um those workflows will be will pick up where they left off so again another massive bonus but um but You can do that of temporal Service as well the uh the self-managed temporal server um how temporal uh helps uh sort of insight and debugging um final slide about the temporal prop is uh we have an event history so temporal runs everything the workflow does um is is uh recorded in this event history and that enables easy insights for you to know what's going on it it enables debugging like you're going to get debug
level stack Trace level information about Every step of that workflow um and how it went um and this allows you to to Really trace the entire execution path so there's no more um like correlation tokens there's no more log diving it's all there in the the temporal history in a way that's just very very easy to interpret um the last thing I'll say about temporal is uh it's all MIT open source um that that goes for the temporal service and the sdks so this is just a straight up open source project That you can run
all right so with all that temporal background um I'm going to show you implementing sagas with temporal I'm going to show how it's easy um I'm going to show you how it's resilient um and I'll also show you a little demo that I did in the in the Java SDK all right so back to our order uh fulfillment workflow um we've now organized this order fulfillment into temporal um and so what you see see here Is we have this um temporal workflow um so we're taking an order and that actually kicks off a temporal workflow
so you can think of one order one workflow and now each step is an activity um and so the first activity is going to ask the payment service to process a payment so on and so forth um as you recall activities have retry policies as well so these um this workflow can now survive temporary Downtime in the payment inventory or delivery service and pick up where they left off um onto how a saga pattern looks inmoral and I'll show you the code level in a second but on an architecture kind of level um you the
state is positioned uh you have okay so the Saga is it's it's straight up orchestration the The Saga case really looks the same as the orchestrator case I showed you before but the benefits of temporal is um you Get the state persistent in an event history so instead of this single point of failure problem um this this workflow can uh persist Beyond um the worker that's executing it um to put that in another way if there's an issue with the temporal workflow on the worker the worker dies the worker crashes the infrastructure goes down this
workflow Will Survive another worker will pick up that workflow and carry it out um and and so it kind of gives you all the Benefits of that orchestration without the the the kind of single point of failure problem of orchestration um but for the Saga Point itself it's going to look the same as the orchestration case because what you're doing is you're catching something that went wrong in this case in this diagram is the delivery service that's failed again and you're essentially carrying out those compensations after that delivery service Fails all right into code a
little bit um so here's a snippet of a temporal workflow that implements a saga pattern um and I've done this in the simplest way I could think of which is um a try and a couch uh and so we're up to step three here and our delivery service has been failing a lot in this presentation and so I've shown you step three and which is delivering order and so this is literally temporal workflow code in Java SDK and so if you try activities deliver Order let's say that fails unrecoverable then we can catch that exception
that comes out of it and we can carry out those compensating actions so anyone who knows development will recognize a tri catch um and will recognize that um once this uh unrecoverable ER has been caught that those compensations are running and those compensations are running in reverse order by the way you get cancel delivery restock or a refund payment um And by the way these can be done in parallel using temporal that's fine but I'm choosing to do them sequentially to make sure each goes fine um and then finally what we're doing is we're wrapping
the workflow in an exception uh in that exception so we captured the information of what went wrong with the delivery order step and then we're going to fail the workflow um and so this will end up with the workflow failed again this is for the unrecoverable case if The order delivery was temporarily down um what the retry policy would kick in um but this is the case where there something catastrophically went wrong with the order delivery there was a bug there was an invalid response that came back and the workflow just doesn't know how to
recover we're just going to roll back these steps so that's the simplest case um but the temporals Java SDK actually contains a saga helper class and this allows um you to create these Compensations with with less code in that control logic um and for the other sdks we have a go sdk. net typescript so on and so forth um you can quite easily create a saga class what what this Saga class is doing is really just collecting a list of functions it'll need to run in case of failure so it collects that that list of
compensations it's going to need to run and so if we reorganize our code into this uh to use this um Saga helper class in Java um what you end up with is You have the compensation step that you add first um and then in this case it's step three again so the compensation we're adding is cancel delivery it's a delivery fails right um and then we try deliver order and let's assume that deliver order fails again delivery service is is really bad today we catch that exception in the same way we did in the last
slide um but we run compensate and all compensate does is roll is is is iterate Through a list of compensations in reverse order so the last compensation we added it's going to execute that first and so in this case it's going to run the compensation for cancer delivery and it's going to run the compensations for restocking the inventory refunding the payment so on and so forth but this is just a simpler oneline way to do it and then finally again because this is an unrecoverable situation we are going to fail the work Flu last code
slide um to put it all together I just wanted to show you the complete control Logic for a temporal workflow that includes a saga pattern you'll notice it's it's small enough like this whole use case is small enough I can fit the entire control Logic on a single slide um pretty comfortably I'm not going to spend too much time on this slide at all uh because I'm actually going to run this control logic I I I really I created this and um pasted it In in TJ and run it and um it worked so I'm
just going to uh run a quick demo where um I'll show you a saga in temporal and show you how that looks um in the temporal UI okay time all right so this is you should recognize this because this is literally the code that was on the slide I think I have an extra logging statement here but aside from that it's exactly what what you saw on the slide you have our Compensations and you have uh an order so this is a workflow here this workflow uh method and then we have three activities process payment
Reserve inventory deliver order with the corresponding compensations that we're just going to add in case any of these steps go wrong and so what I'm going to do first is I'm going to run the workflow and I do that just by calling a caller that I've set up it's just a method that calls the temporal API I'm doing this in temporal Cloud which is our our s product but you could do this on the self-managed sort of Open Source um temporal okay so this workflow is running and if I click on it um it's actually
not going to go anywhere um I wanted to show you this uh there are no workers running um actually consuming tasks so like I said workers are something that you run in your infrastructure that executes your code Um temporal Cloud doesn't know anything about the code you need to run it just knows that there's a workflow that needs running so the first thing I really need to do is start a worker that's going to run uh this code so let me do that and then go back to that interface and I'll show you it running
so I we go back to intell J I'm just going to start a single worker right we go back to Cloud if I refresh This these tasks should be coming in okay process payment it has started this is all marked out you'll see that in a second Reserve inventory is happening deliver order is happening and that should finish soon great if I go back up here you can see that the order is completed this is a successful workflow execution it's just a happy path to begin with the uh workflow is complete which is an order
processing uh example if I look at my Workflow execution you're going to see this kind of dead time originally because I wasn't running a worker we started the workflow here there was no there were no workers to consum it um and now we have these three steps that ran once I started the worker okay so those three steps and this again for a saga webinar this isn't a very exciting um demo I'm going to start messing things up in a second and show You those sagas kicking in but there those are the three happy path
steps and so that's why the workflow completed without a problem so the next thing I'm going to do is I'm going to go into one of these activities process payment Reserve inventory or deliver order I'm actually going to make one of them fail in an unrecoverable way um and that should trigger The Saga compensation um and so maybe the most interesting thing to do would be to make Deliver order that activity fail because if I make this like the final step of the auto fail it's going to have to go back and roll everything else
back so let's do that so going over to my activities uh class where my where my activities live um you'll immediately see that it's all just mocked out code I'm I'm just putting like thread sleeps in and logs in like it's we're not doing any real API calls or anything um but if I go into deliver order um I'm just Going to make the the um this activity fail and the way I'm going to do that is I'm going to throw a runtime exception so this is just a classic Java runtime exception um there's nothing
special about it I'm just uh making essentially making this method this deliver order activity crash um and like I said before if I made it crash the default thing for temporal to do is for the retry policies to kick in um temporal wants workflows to succeed And if there are temporary errors or if there are errors temporal is going to assume they're temporary like API is being down or some other issue that will write itself um but in this case we want an unrecoverable failure uh because we want to trigger The Saga so what I'm
going to do is uh I'm going to put that runtime exception in a temporal concept here which is a a new non- rtire new non- retriable failure I'm going to feed this to that and that basically tells Temporal hey this particular exception that came through we can't recover from this something something is deeply wrong with the delivery system we're just going to need to fail this this workflow um we're probably going to need to go back to the customer and say hey that order you put in just sorry we can't do it like it's a
really unhappy case but let's do it so that's that's in now I'm going to save that I'm going to restart my worker with that new code so kill the Old worker run the worker again so that loads this new version of the code it's ready to go and I'm going to run that workflow again and so this time it's going to run the workflow and it's going to get to this deliver order step and it's going to encounter that runtime exception and it's going to throw sort of un un retriable failure and that's going to
Trigger these compensations so let's do that all right things happen quickly so I need to go back to the interface wait a second there it is all right it's processing the payment it's reserving the inventory at least it's trying to reserve inventory um that should succeed and then now it's trying to deliver the order okay uh that's our runtime exception right there delivery failed Which is my little runtime exception message and you can see immediately now that deliver order has failed um it those those compensations have kicked in cancer delivery restock inventory refund payment and
so just like that you get um you get that Saga pattern you get that uh those roll backs or compensations and to put that into the timeline view temporals kind of timeline view here we now have a workflow and it's big and angry and red because it failed um If I go up here it says it failed to right um because again this is unrecoverable we we process the payment fine we reserve the inventory fine deliver order had that nasty runtime exception in it that I that you can't recover from because of that the workflow
triggered Saga compensate and Saga compensate canceled delivery restocked the inventory refunded the payment so it it it rolled back each of these steps so This inventory has now been restocked this payment has now been refunded the workflow has failed of course you could have a step here that says notify customer of of the the failure um but that's that's all I wanted to show you um in in sort of the temporal demo I just wanted to show you how lightweight and and simple um something as as complex as a saga pattern can be um if
you use temporal it's all plain old code if you're a Java developer particularly This is going to look very uh idiomatic to you hopefully um but we have sdks in tons of languages as well uh but that's the end of that demo I have one more slide just to kind of wrap it up all right so I was thinking about what is it like to code suers when you use choreography or when you use kind of traditional what I consider traditional orchestration Technologies you know Q's State machines um State databases that kind of thing versus
temporal um so when It comes to coding choreography is instantly difficult when you're trying to do sagas because your Saga Saga code's going to be all over the place um orchestration makes life better because you have a single place for that solo code and all those roll backs are in a single like all your control logic is in a single place um but scaling really and and and developers will know this um doing orchestration in a in a Scalable way and a fault tolerant way these days requires you to think about message cues buses State
databases State machines that kind of thing so life gets very complex as a developer um whereas temporal you've got this you saw it just then you just have this single place paga code um no dealing with q's of course q's and database are happening in the back end like that stuff we have that stuff but you don't have to think about it um for the scaling side a Traditional orchestrator can have scaling challenges right um you all orders are going through or being orchestrated and all state is being captured by this orchestrator but temporal is
horizontally scalable out of the box I could have run a thousand workers and then just started throwing those order that order demo like tons of orders at my demo and temporal would have handled it just fine um the same is true for the uh temporal service So the temporal service is architected to be horizontally scalable and fault tolerant as well in many ways particularly our um temporal Cloud our SAS product we've done we've done a bunch of work around making that resilient fa tolerant um for debugging and and insight uh troubleshooting choreography you know it's
difficult uh problems in a service you have to really debug that service so You have to use correlation token tokens and log dive to know what's going on um I'd say with orchestration it it's easy um you probably do have to have a state database or something that that um persists Beyond a process running um to understand what's going on um but in temporal you get the luxury of this event history you get this um temporal runs on this event history in that UI which uh you can also access by um command line or you
can access by API so You don't have to use the a API to know what's going uh the UI to know what's going on there's very different ways to see the event history um you can go there though to know what what is the state of that order what happened in that order in in great detail um you saw that when I made the delivery service fail that it said uh delivery failed but if I was to click on that there's literally a Java stack Trace in there that you can look at um so you
get Extreme Insight I would say on a on a per workflow or per transaction level um so kind of to wrap it up hopefully that gives a good overview of of the challenges people usually face um implementing sagas and uh I was able to show you what that looks like in temporal and that it's just I just think it's a lot easier um and and so if you're coding sagas and you're coding complex business transactions um I I think uh you should consider temporal um As a simpler more reliable option um and with that um
happy to take ton of questions about uh temporal whether it be sagas or anything um temporal related awesome Steve thank you so much um so we did have a lot of questions come in um we'll try to get to them all if we don't get to them all we can follow up over email um so don't worry about that um you'll you'll get your question answered one way or another um so just to start out there was a Question will this recording be available yes it will be we'll be sending this out for everyone to
watch afterwards um I'm going to just pull out a few of the common themes of some of the questions to get to them and then let's dive into specific questions um so there are a few questions about the sdks that are available um and so do you want to just talk a little bit about that yeah definitely um this is going to be a test of my memory um the sdks that We have are Java go they were the first two sdks by the way typescript python net and I believe that's it for now um
and the thing the thing about our sdks is that you can do the same temporal things in all of them so they should the feature-wise they should be very very uh similar like almost a complete parody If not complete parity um for for tempor What you can do temporal ways um but the the most important thing about um the sdks to me is that they're idiomatic so if you're a go developer you're going to be dealing with go routines and channels um if you're using typescripts you're going to deal with a weight you know you're
G to you do promises um so on and so forth and and so that's what I like about it Donnet you're going to use tasks um it's it's very idiomatic it's very uh comfortable for developers who Know those particular languages awesome thank you um okay so I'm going to go and and we can start addressing some of the individual questions here um so Alec wants to know if a single workflow can have activities that run on different microservices whether a single workflow can have activities that run on different microservices um yeah definitely you can if
I have your question right you Can split out your activities to run on different infrastructure um for example so let's say you had and the way you do that is is you can have your worker talk to different cues so let's say you have one workflow and you have a worker that essentially just pulls that workflow for the next thing to do any of those activities can actually run on different workers in different infrastructure um one a common case I think about or a case that's becoming extreme Increasingly more common is an AI type use
case where uh you'll have this data processing step maybe that runs on standard CPU based workers um so you those activities will run on compute infrastructure that's sort of plain old and then you'll have like a GPU infint step and that you know that inference needs to happen on a on a big fat GPU machine um and so what you're going to use is you're going to have workers that uh run purely to execute that activity On that machine but it's within the context of that entire workflow so what you end up with this workflow
that essentially executes across a disperate set of machines but in a way where it's it's all durable and you have a single event history all that good temporal stuff but it's definitely optimized to run in many different places all right thank you um Dave wants to know if an activity is the same thing as a virtual Actor um to be completely honest I would have to know more about what a virtual actor is um so maybe we can have a follow-up conversation um a virtual actor to me can mean a couple of different things um
so rather than deep diving it here let's have a follow-up conversation um I would say an activity is it it really is a wrapper to a set of arbitrary code so your the what happens in an activity is completely up to you it could be processing data it could be Running AI inference it could be making API calls it really temporal doesn't care at all what happens inside activities what we do care about is that we wrap them in a way that we can schedule them and we can retry them and we can collect an
event history on them so they become durable they durably run and and I think uh maybe that that does correspond to a virtual actor when I think about it but it's definitely worth a follow-up Conversation awesome um okay so we kind of covered this in the presentation but um someone wants to note can temporal handle backward recovery across multiple services and then do separate teams need to share code in order to compensate across application boundaries it's so great two great questions um the first the first answer is yes you can you can essentially compensate over
like multiple disparate services For sure and I hope I I showed that um just assume those activities are calling are making API calls to disparate Services owned by different teams um in My Demo um as far as multiple teams working on workflows and that kind of thing uh the I think the great news for temporal is normal code collaboration and organization constructs apply so just how you would in the same way that you would organize Different uh code into different repositories and run them in different environments right and um and collaborate on them in different
ways um and also run them in different environments when you're operating them you can do the exact same with temporal because temporal is just code you can slice and dice workflows and have Parts owned by different teams you can have activities that are owned by different teams and you just import them as Libraries or however you want um I would say think about from from uh that perspective think about this not in really temporal terms but think about what would you usually do uh in in code to organize your concerns right um how would you
organize those transactions across multiple teams um and the same should apply to temporal because at the end of the day you're just using the temporal SDK you should be able to organize things the way you usually Do great thank you um so Bradley's wondering about he's saying the big drawback for orchestration is the orchestrator goes down wouldn't relying on temporal suffer from the same issue you know you address this a little bit but maybe you can get a little bit more into the the temporal service yeah I'm I'm glad I'm glad you asked that question
um and I was hoping someone would ask that question let me go just a little bit I know we're getting probably To time but let me go just a little bit into sorry we can go over we can go over oh okay cool cool cool so people have to jump they can jump but we have a lot of great questions so we'll give it a few more minutes past the the top that all right well aome I think this question deserves like a proper answer um yeah the temporal service because the workers are taking tasks
from the temporal service um it is it is a single point right it is a single point but I would Say that it's a much more fault tolerant single point than any or other orchestrator that I know so for example the temporal service um is architected in in an extremely horizontally scalable way it's a series of services people often run it in kubernetes but they don't have to temporal Cloud runs in a certain way that's our SAS product um where it it stretches across availability zones um there are multiple redundancies um including on the Database
level um and so for example I'll talk about temporal Cloud um because that's that's what I know quite well um it can can survive an entire availability Zone loss in in an AWS Cloud region um and without any downtime uh without any data loss right zero um and that's a testament to the resiliency of the temporal service so the next question you might ask and I kind of talked about this in one of the slides um is okay well what if a whole region Goes down and and I say okay good point um that that
is generally a catastrophic thing my question for you is my answer for you is twofold the first thing is if if an entire region goes down you probably have more problems than just temporal um I I would say frankly and in my experience that tends to be true um think about all the services you have but let's say you have a truly multi- region redundant setup which is which is true and people should strive for that Um we we do have a um feature of temporal Cloud called multi- region namespaces and there is an upcoming
webinar about this actually so that's it's probably good timing um where uh temporal uh has asynchronous replication on a very tight timeline um where it can tolerate an entire region going down and seamlessly switch over to another region to complete those workflows um with very very minimal disruption and that that there is your um resiliency case I would Say um that is your way around a your ultimate way around a a single point of failure for for really for applications that are extreme Mission critical perfect thank you and we also touch on this very briefly
at the end but um M has a question about how does temporal scale yeah um great question there's two there's two kind of um aspects to that there's the the worker side and I think that's easy to to understand for a lot Of developers the more workers that you add to to consume workflows the more scalable it's going to be so I'll probably end that one there and then the other question is how do the how does the temporal service itself scale um if you're self-managing temporal that's going to be on the strength of the
amount of compute you add to it like I said it's a horizontally scalable service so you can actually uh add um database capacity you can add um more if It's a kubernetes setup you can add more history pods matching pods front-end pods like our kind of um the services that can compose the temporal cluster or the temporal Services you can add them on the Fly and so that's how you kind of get your your scalability out of the box and you'd have to monitor that yourself and then um respond or set up autoscaling policies for
those Services um in temporal Cloud we run it as a um run that kind of scaling transparent uh To you so in other words we we give you a namespace you connect to that namespace and we run it in an elastic way behind the scenes so instead of thinking about temporal Cloud as as a single server that we run for you what we're really doing is running a set of of servers that we interchange um constantly it's we do this a very elastic thing behind the scenes to make sure that if you get a spike
we uh we can respond to that Spike um extremely Quickly um in fact uh who have been running um workflows in temporal Cloud on mass um may not know this but we're actually constantly changing infrastructure around upgrading things doing all kinds of stuff with zero Interruption to your running workflows because that's how it's been architected so I think scaling is different depending on whether you're self-managing or whether you're using our Cloud product which is a bit more Sophisticated in that that scaling way but either way we're set up for it all right thank you um
Dave had a question about um this is one that's a little bit complic complex can an activity inside a workflow within a worker initiate a child workflow within another worker um maybe maybe we'll just follow up um with this one because there's yeah I I love that one in writing I'd hate to get the um I'm sure it's in writing in the chat But I I'd love to think about it in deeper I don't want to give an answer that's wrong um because I misunderstood the question um from what I heard I I'm G I
think the answer is yes but let's just talk about the implications of of what you're asking um that's definitely a followup yeah yeah okay um and then we're we do have a question about some Alternatives compared to temporal um so is there anything you can talk about the differences or advantages of temporal Over something like mule saw yeah um I don't want to talk about specific kind of Alternatives or anything like that that that may be it may feel mean or or wrong but um I'm going to say there's a couple of categories of Alternatives
um one would be like a a bpmn sort of visual workflow Builder type solution um and people tend to like them because you can see what you're building you can structure your workflow in a very visual way um I think The drawback to that is you you're not running that diagram directly there's always going to be some translation between what you're putting together visually and what actually runs and so from a debugging and a complexity standpoint um firstly you're probably not going to be able to capture the true complexity of what you want to do
for any worklow that's reasonably complex and secondly um I don't feel too comfortable about sort of putting Something together in a diagram and knowing it's that's exactly what's going to run I feel much better about temporal because I can unit test it I can integration test it I can throw test workflow um histories at it or past workflow histories at it to know that um if that succeeds then the production case is definitely going to succeed so that's one category it's sort of your visual workl designers and the second category I can think about is
sort of Your dsls or your um your kind of Json usually Json or yaml based kind of specs like there's a lot of work Solutions out there that really translate to a bunch of Json like a a dag or like a a graph a task graph um or a yaml kind of structure for this is the stuff that needs to run um and I've actually have quite a bit of experience in some of those and again it's the same reason it's just not very comfortable it's not very um as a coder I want to write
code And I want to read code I don't want to read Json um I don't want to debug Json I don't want to I I can't like unit testing Json is kind of annoying you have to sort of pull it in and then pass it and like it's it's just very uncomfortable um the reason why I like temporal and join temporal is is it allows me to deal with code AS code um and sort of as an a past engineer that's just way more comfortable so I think that's yeah BPM and visual workflow Designers and
then sort of your your um transpiling cases or your um your DSL or your Json yaml type cases are sort of our main uh the main Alternatives I would say but I think I think for developers they tend to prefer temporal and that's what we're seeing with they tend to prefer workflows as code and that's what we're seeing yep exactly so we'll take just um maybe two or three more questions and then we'll wrap things up and follow up Afterwards with the rest of the questions awesome okay so um William had a question about um
he's wondering in the order example can the workflow run asynchronously um I.E when a new order request comes in does the caller have to wait for all the activities to finish before a response can be sent back uh um no not at all um temporal is uh insanely asynchronous friendly I would say um so there's multiple levels uh for the demo I gave you I could have started A thousand orders at the same time and that single worker could have just processed through those um pretty gracefully sure that worker would be busy if it was
a thousand but the great news is because there's an aventry that prevents uh that presents the next task to do even if that one worker was overloaded and even if I didn't want to scale it um that that all that work is going to do is say hey I'm not ready for this yet and the temporal service is Going to sit there with the next task ready to go um so in other words uh it might get slower to execute but um if you don't scale it but um you can absolutely uh throw a bunch
of orders at at async while it's busy and um temporal service is going to make sure all of that succeeds um the other case is and you didn't exactly ask this but I wanted to mention it um asynchronous code is ridiculously easy to write in in temporal because you don't have to think About cues and state machines and databases um you can just write async code as you would in a single script you know using promises or awaits or tasks or go routines or whatever you're used to um and uh you can also interact with
workflows while they're running so let's say you had an order that was running and you needed to like there's a big red button like at Amazon you had the andon cord right if something really went wrong and everything stopped so let's Say you had that scenario you had like a thousand orders and you needed to tell them all to pause or stop immediately um you can send a to those running workflows a signal is a temporal concept that allows you to um to to take new information while workflow is running and handle that information so
in this case the information would be hey pause all of these orders please and you have a signal Handler in each of those workflows that order workflow that goes Oh okay um I just received this signal I was doing something but I just received this signal um Let me let me wait or let me do something else um because I received this signal um and so async on many levels is is something that's quite easy to do it's probably the topic of a follow-up webinar to be honest but um yeah the answer is an emphatic
yes perfect and then we'll do the last question here um someone wants to know if the whole Saga is operating in a Single workflow versus multiple workflows under a single worker ah um in my case in my example it was a single workflow um the relation I wouldn't think about a tight relation between workers and workflows um multiple workers can execute one workflow um in certain scenarios like I wouldn't I wouldn't even think about it that stuff stuff sort of um happens transparently to you um you can run Sagas across a number of workflows I
would probably recommend against it because uh a workflow is really where your control logic lives so if you want uh a series of compensations you probably should put that control logic in one place um an exception to that would be if you had let's say you had a top level workflow and then you had control logic owned by other teams for example you might want to have child workflows owned by those Teams and so you have this kind of parent workflow that does the overall order and then maybe the payment thing is its own workflow
that's a child workflow that's a temporal concept um and then so if a saga would be uh refund that payment might be a child workflow that kind of thing so you can get this kind of hierarchy of workflows and that can be owned by different teams and different workers then um but all of that's possible um that's again probably A deeper best practices conversation um but generally I would say if you want the at its simplest if you want um compensations to happen and roll backs to happen um it's best if you think about putting
them in the one place in the one at least directing them from the one parent workflow because then you have a single durable place to to view how that overall business transaction is going and we we do a lot of as a Solutions architect we do a lot of work Workflow design sessions with people and we spend a lot of time helping them uh conceptualize how their business transactions are going to fit into workflows and it's usually pretty natural um but we we help them weigh pros and cons and that kind of thing great perfect
thank you so much Steve um so if you just want to go to the the final slides we can wrap up here uh yes um great questions by the way just a lot of great questions yeah Great questions thank you so much for all the questions um and thank you Steve thank you for your time that was awesome than you so much to everyone who joined and um who who's been really engaged if you likeed this session we actually do have another session coming up on June 20th about sagas so one of our customers will
Bank they're going to be talking about how they have implemented sagas using temporal um so that'll be a really great followup just to learn a little Bit more you can register on our website or you can just scan the QR code there um and go ahead and register now so um that's it thank you everyone so much for joining thank you Steve um yeah we'll see youall later thank you Megan bye bye