A Simple Introduction to AWS Step Functions

Thank FSM Gods next time you drive through it safely.

The FSM model ensures that, when a traffic light is about to go green for one direction, all the others turned red before.vIt does that by managing states and transitions. Mark those two words. They’re pivotal.

Right, let’s clarify.

For one light to transition into the green state, all other lights must have transitioned to the red state before. Simple, right?

FSM is a robust model for that type of scenario: multiple states with transitional rules. It’s kind of a design pattern. Not quite, but not too strechy.

Wanna see more?

Ever wondered how vending machines work? Yup, FSM is there too! It guarantees you won’t get a snack until you feed it up with at least those 3 bucks. Ouch, expensive! FSM will also manage the transition to delivering the snack after payment, and also secure your change if any is owed.

Yes, some times vending machines fail, but it’s not due to the FSM model

FSM is a mature and proven model that can be trusted. Implementing correctly isn’t hard. When done right, rest assured it is going to fulfill its promises.

FSM implementations

As a mature, battle-tested model, there are implementations in many programming languages, such as Python, Javascript, Java, etc.

Please do your own research when deciding on libraries as these are just examples.

Or… even better… wait for it:

Wouldn’t there be a way to use FSM without having to implement anything programming-wise?

YES! Enter AWS Step Functions.

Step Functions: a managed FSM service

Following FSM concepts, AWS Step Functions also has states and transitions. Tasks are also part of the package, but more on that later.

Straight to the point

We’ll start with some examples and go from there. Consider a huge number of entries in a database. They all need to move to another storage location. There are too many entries, we can’t do in a single shot. Will need to loop.

Step Functions implementation – example diagram – adapted from AWS Docs

This outlines how Step Functions would handle that data migration process for us.

The initial state is data seeded from the source database. It goes through a loop, then reading the next entry, sending it to another location, until it finally succeeded.

It won’t magically guess what we need. An FSM is coded in Step Functions using the Amazon States Language. It’s just a JSON representation of your States, Transitions, and Tasks.

Here’s the JSON snippet corresponding to the diagram above.

Now Tasks. It’s what executes what you need once data transitions across states.

In our example, when a DB entry comes from the loop, it gotta go to the new storage location. A Task would involve, in simple terms:

1. Getting the original data
2. Formatting it for the new location standards
3. Establishing a connection with the target database
4. Inserting the data
5. Waiting for confirmation
6. Returning a 200 | Success response from the transfer request

In the AWS example, this Task is accomplished by a Lambda function.

Advantages of Step Functions

Auto-retry failures

In case the migration for a given entry fails for any reason, the Step Function can automatically retry it for us. Could even have it notifying ourselves in case of errors.

It will make sure an entry is not duplicated in the target storage by tracking its state and whether a retry is needed.

Once successfully migrated, an entry cannot possibly go back to “pending migration” state. That’s guaranteed by the FSM model.

FSM Benefits

Everything beautiful about the FSM model comes bundled by default:

• Maturity and trustworthiness: resilient and fault-tolerant
• Flexibility
• Quickly move from an abstract, conceptual process to code and execution
• Little processing overhead

Adapted from: elprocus.com

Fully-managed

As we discussed previously, we can avoid the hassle of implementing an FSM library in our own environment.

AWS will manage everything. Make sure servers are provisioned to run our state machines. Scale the infrastructure when needed. Up and down. Horizontally. In the fourth dimension. Kidding.

No, but really, scalability is difficult to get done right, this is a big plus.

Integrations

Step Functions will seamlessly integrate with various other AWS services.

It’s possible to connect your Step Functions workflow with other AWS services or even other workflows. That’s right, you can compose workflows to create higher-level processes. Here are some examples of what you can do:

• Invoke a Lambda function
• Run Amazon Fargate task or Amazon ECS
• Submit an AWS batch job
• Publish a message to an SNS Topic
• Interact with SageMaker machine learning model training, inference, and classification
• Send a message to an SQS queue
• Put or get items from a DynamoDB table

Monitoring & Debugging

You must be asking: ok, this all too good, but how do I monitor my workflows, make sure they’re running as expected, or debug issues?

Step Functions publishes events and metrics to CloudTrail and CloudWatch, and is also monitored by Dashbird’s Step Functions monitoring platform which contributes to better architectural practices. With Dashbird you can set alarms and monitor the health of workflows throughout various indicators.