What are AWS Step Functions? (and why you should love them)

alright folks welcome back to AWS simplified the place where I teach you everything there is to know about AWS and today we're going to be talking about AWS step functions and this is going to be an overview of why they're useful and why you should probably be using them just before I continue onto the video I've had to suggest to take a look at my previous video where I give you a walkthrough of how to setup a step function with a lambda and I'll give you a link in the description and as a thumbnail on

this video right now so here's a quick summary of why step functions are useful and why you may want to use them so step functions allow you to create workflows and that follow a fixed or dynamic sequence and that's where the steps name comes from so for instance in a traditional application you may want to get a API request you may want to write to some kind of database and depending what was already there you may decide to proceed to the next step you may decide to just return and if your database call fails then

you return right away step functions allow you to model this as independent units and these independent units or these tiny tasks can reach in and directly interact with AWS resources such as dynamo SNS lambda to go and perform some job and return the response back to the step function so it's almost as if step functions are an orchestration for an application something that's really great about them that you don't really have to worry about if you use step functions is that they have built-in retry functionality and you can set this up however you want you

can set this up so you retry three times say if you're trying to write to dynamo or SNS or something you can have no retry policy and you can also define attributes of your retry policy so if you want to do exponential back-off or linear you can define those settings as part of your retry policy and perhaps the most important or the greatest benefit of step functions is that they natively integrate with AWS services such as lambda SNS sqs dynamo and many others that they're constantly adding so instead of doing like work where in a

traditional application you need to you know write your access or classes you need to get your I am role setup and make sure that everything is set up correctly in your code this these integrations are defined in what's called an ADSL language which is a step functions kind of JSON like language where you define steps and you chain those steps together and one of those tasks within the chain could be interacting with an AWS service such as lambda or SNS in this case and from a user perspective it's truly great in terms of auditing where

you are in your application so if you're progressing through your step function say you write to dynamo then you write to us an S and then you write to s3 it gives you a very feature-rich GUI or graphical user interface that helps you identify where and in the application flow you are and kind of keep track of it so it's very easy to identify what the input and output was from one step to another and also quickly identify if one step failed and where the where the gaps are if your step function goes down or

if your step function starts failing and finally another obviously great feature that we come to expect of AWS highly scalable and very low cost so it's two and a half cents for every 1000 state transitions so to put this in a more practical perspective say you define a state machine as part of your step function that has five transitions and you have a hundred thousand invocations of those per month that costs you $12.50 that's a pretty good deal in my mind so I figured the best way to explain how this actually works was to give

you a practical example and in this example I'm going to tell you about this hypothetical credit card processing application that I like to use and how you may model this in the traditional sense so without using step functions and having a server that's processing all this logic so let's assume for a moment you have something called a transaction processor service and this can be on ec2 could be lambda could be whatever you want and its role is to basically orchestrate interactions with multiple services there could be you know pin authorization comes to mind to make

sure that you're the actual person or the pin is valid before it kind of commits the transaction there's the stuff where you may want to commit the transaction to the credit card provider like Visa MasterCard whatever and then maybe there's a step at the end depending on if the prior two succeeded maybe you want to tell the world about it you want to broadcast that to an SNS topic so other customers or other kind of Co teams can find out about it right so let's start with this so we have a credit card purchase request

that comes in and hits this ec2 server maybe it's always seen a REST API and this is what a request may look like so we have a customer ID it's this random number credit card number provided pin and an amount so the first step that you may want to do is store an audit log so you want to keep track of who when who and when people try to submit transaction requests to your service ok so what would this require all you need to setup DynamoDB client and then you need to write to the table

from there provided the the step two succeeds or the the right to dynamodb succeeds you probably want to proceed to the next step which is to authorize the pin so we may reach out to this pin authorization service could be on-site or could be some kind of external visa endpoint that you're hitting and let's assume that it can reply yes or no like accept or reject if it accepts then you want to move to the next step if it rejects then you want to return a response all the way back to the other end right

and then the fourth step is provided the authorization step is successful you actually want to commit the transaction to some kind of credit card authority service that I'm calling down here and assuming this goes right and you were successful in winning the transaction maybe you want to tell the world so you want to broadcast that to it to an SMS topic that you own okay so this is kind of the hypothetical application that we're dealing with so let's do like an exercise here let's think about this from a logical perspective of how you would implement

this so if I were implementing this what would I do so I'd have this ec2 machine that's hosting a rest endpoint right and I probably read my code such that first thing I do is try to write to a dynamodb table right maybe that succeeds okay and then the second thing I do is I want to authorize the pin so I call authorization service now two things could happen well actually three things they can accept they can reject or they can throw an exception so the accept and reject aren't very interesting cases because if they

accept we proceed if we if they reject then we just come back all the way out the other end but what happens if they throw that throw an exception their hard down for a little while right this happens services go down sometimes so what does that mean for you well it means that this credit card purchase request like what's gonna happen to it it's gonna fail right because you failed at this step and then at some point later whoever made this request is gonna have to come along and resubmit the request and then when they

resubmit it what's gonna happen you're gonna create a new request log right and then when they submit the request you're gonna repeat the same process you're gonna try to authorize the pin again and call the service maybe it's still down maybe it's not but basically you need to reset your context you need to retry at the entire suite of events or the entire sequence of events that led up to this point right and that's not a very scalable thing because if you think about this if you need to retry these steps that means that someone

needs to come along make a new request and then you need to write a new row to your DynamoDB right so there's it's waste on that level so that's the kind of use case that step functions help solve so they help you orchestrate the sequence of events that you kind of code into your step functions so that provided this fails you can have some kind of retry policy you can have a wait time that says okay don't don't fail right away wait maybe 30 or 40 seconds or a minute or five minutes whatever you want

so you can build in these retry policies and these rich functionalities to drive your workflow as opposed to doing it all through specific sequential code so that's the the use case that step functions are trying to solve okay so now let's actually look at how this can be done using step functions and so I actually wrote the adsl which is the language that you use to write your step functions in AWS done functions so this is the output of it and I've actually just kind of superimposed the resources that I'm interacting with because it's not

obvious unless you are actually looking at the ADSL code so so what happens here so we start off with processing and transaction right and the input is the same we have the customer ID we have the cost or the purchase amount and then we have the credit card number in the pin right so the first thing we want to do is store the history and this is extremely easy to do in step functions as opposed to writing the code yourself where you ask we need to call dynamodb do all this extra stuff it's brain dead

simple so here's an example of how it works so you define a task it's a store history sorry you define a state and this state is a pipe task and the resource that you're going to be active interacting with is Dino and DB and specifically you're going to try to put an item there and we're gonna specify some parameters right and the parameters are as such the table name is this table name and this is the content of what I want to write to my dynamo dB I want the transaction ID which is the primary

key of the table and this notation is basically saying take the input of what was fed into the state machine originally so transaction ID and save that to my dynamo DB table you can obviously include much more context here if you want to include the pin actually probably not everyone included details like that but if you want to include stuff like timestamps things like that you can do that very very easily here using this adsl notation and then from there you can define a retry policy so if this fails the first time on any error

so here's the error that we're kind of catching I want to attempt three times and my intervals between the attempts are going to be 3 so 3 max attempts and 1 second intervals in between each and then provided this actually works then we want to move on to the pin authorizer which is this next step right and then the pin authorizer as i described in the example previously you may want to reach out to some service to determine whether or not the pin that was provided is actually valid or not and that service is going

to return one of three things like I said before so it's going to return and accept or reject or an exception if it throws an exception you can potentially move this to end immediately or you can do something more dynamic like you can define a rejection step that exists over here and the case that it's rejected you can do something or you can return that response directly back to your client so let's assume that you the pin was either accepted or rejected right so that inputs or that output from the pin authorizer gets fed to

the inputs of the pin decider so if this returns accept then accept is piped into the pin decider and this can take two branches one is success if it moves if it was accepted then you move to the success step if it failed then you move to the rejection step and if it failed then you if it was rejected then you end you're done right there's nothing left to do however have you succeeded then we want to move on to the next steps right so as I was talking about before now we know that this

user is authorized now we want to call this authority service to actually try and commit this transaction and we can call that service through a lambda function so we can type the amount all the details that are are used in this transaction into this lambda and it can actually call that service and there's two kind of situations here you can either succeed or you can fail and if you fail then the transaction is failed and then you're done however if you were successful then you move on to the success step and as we said before

if we're successful we want to tell the world about it so we broadcast that SMS message so that other clients can consume it so the the emphasis here is to show how easy it is to set this up and I hope I made it clear by showing you instead of having to write all this complex code that hands that handles these partial failures it's very very easy to integrate with all these AWS resources using the ADSL language and I also have an example here for how it works with lambdas so you can see with lambdas

is even more brain-dead simple you just provide the state that you want to map to this task and it's a type of task you provide the AR end of the resource which is the pin authorizer and then you give it the next step so depending on what this kind of returns you can pipe that output into the input of the pin decider and then it can branch off and perform some logic depending on whether or not it was accepted or rejected so hopefully you found these videos useful in my next one I'm gonna actually show

you how to set all of this up using all of these resources so we're going to be creating the dynamo table the lambdas the SNS topics and really creating this example using the real resources that go along with it so I'm gonna it's going to be an end-to-end tutorial and it'll be in my next video so that about wraps this video up thank you so much for watching and if you have any topics you'd like me to cover please feel free to drop me a message below and don't forget to Like and subscribe thanks so

much and I'll see you next time