A Simple Introduction to AWS Step Functions
Step Functions is a managed service by AWS that implements the Finite-state Machine (FSM) model.
You coordinate multiple AWS services into serverless workflows so you can build and update apps quickly. Using Step Functions, you can design and run workflows that stitch together services such as AWS Lambda and Amazon ECS into feature-rich applications.
You can read Wikipedia’s definition of a Finite-state Machine, but I think you’ll like the next section more. Keep on reading.
By example, we learn
Have you noticed that intersection traffic lights never turn green simultaneously for crossing directions? Not even once in a Million times?
It never bugs down! How come?
Thank FSM Gods next time you drive through it safely.
The FSM model ensures that, when a traffic light is about to go green for one direction, all the others turned red before.
It does that by managing
transitions. Mark those two words. They’re pivotal.
Right, let’s clarify.
For one light to
transition into the green
state, all other lights must have
transitioned to the red
state before. Simple, right?
FSM is a robust model for that type of scenario: multiple states with transitional rules.
It’s kind of a design pattern. Not quite, but not too strechy.
Wanna see more?
Ever wondered how vending machines work? Yup, FSM is there too!
It guarantees you won’t get a snack until you feed it up with at least those 3 bucks. Ouch, expensive!
FSM will also manage the transition to delivering the snack after payment, and also secure your change if any is owed.
Yes, some times vending machines fail, but it’s not due to the FSM model
FSM is a mature and proven model that can be trusted. Implementing correctly isn’t hard. When done right, rest assured it is going to fulfill its promises.
Not recommending these libraries, just exemplifying. Do your research.
Or… even better… wait for it:
Wouldn’t there be a way to use FSM without having to implement anything programming-wise?
YES! Enters AWS Step Functions.
Step Functions: a managed FSM service
Tasks are also part of the package, but more on that later.
Straight to the point
We’ll start with some examples and go from there.
Consider a huge number of entries in a database. They all need to move to another storage location.
Too many entries, can’t do in a single shot. Will need to loop.
Step Functions implementation – example diagram – adapted from AWS Docs
This outlines how Step Functions would handle that data migration process for us.
The initial state is data seeded from the source database. It goes through a loop, then reading next entry, sending it to another location, until it finally succeeded.
It won’t magically guess what we need. An FSM is coded in Step Functions using the Amazon States Language. It’s just a JSON representation of your States, Transitions and Tasks.
Here’s the JSON snippet corresponding the diagram above.
Tasks. It’s what executes what you need once data transitions across states.
In our example, when a DB entry comes from the loop, it gotta go to the new storage location. A Task would involve, in simple terms:
- Getting the original data
- Formatting it for the new location standards
- Establishing a connection with the target database
- Inserting the data
- Waiting for confirmation
- Returning a
200 | Successresponse from the transfer request
In the AWS example, this
Task is accomplished by a Lambda function.
Advantages of Step Functions
In case the migration for a given entry fails for any reason, the Step Function can automatically retry it for us. Could even have it notifying ourselves in case of errors.
It will make sure an entry is not duplicated in the target storage by tracking its state and whether a retry is needed.
Once successfully migrated, an entry cannot possibly go back to “pending migration” state. That’s guaranteed by the FSM model.
Everything beautiful about the FSM model comes bundled by default:
- Maturity and trustworthiness: resilient and fault-tolerant
- Quickly move from an abstract, conceptual process to code and execution
- Little processing overhead
Adapted from: elprocus.com
As we discussed previously, we can avoid the hassle of implementing an FSM library in our own environment.
AWS will manage everything. Make sure servers are provisioned to run our state machines. Scale the infrastructure when needed. Up and down. Horizontally. In the fourth dimension. Kidding.
No, but really, scalability is difficult to get done right, this is a big plus.
Step Functions will seamlessly integrate with various other AWS services.
It’s possible to connect your Step Functions workflow with other AWS services or even other workflows. That’s right, you can compose workflows to create higher-level processes. Here are some examples of what you can do:
- Invoke a Lambda function
- Run Amazon Fargate task or Amazon ECS
- Submit an AWS batch job
- Publish a message to an SNS Topic
- Interact with SageMaker machine learning model training, inference and classification
- Send a message to an SQS queue
- Put or get items from a DynamoDB table
Monitoring & Debugging
You must be asking: ok, this all too good, but how do I monitor my workflows, make sure they’re running as expected or debug issues?
Step Functions publishes events and metrics to CloudTrail and CloudWatch, and is also monitored by Dashbird’s Serverless Insights platform which contributes to better architectural practices. With Dashbird you can set alarms and monitor the health of workflows throught various indicators.