The Lambda Circuit Breaker Pattern

In this blog post, I will explain a Serverless pattern called “Circuit Breaker”, particularly powerful when using Amazon Lambda with 3rd party API calls. The first time I read about it was on the Jeremy Daly Blog.

Problem To Solve

When we integrate an external service into our function, the total execution time is dependent on the service response time. Let’s make an example: if our function takes 400ms to execute but it calls an external API which takes 2 seconds to reply back, the total execution time is 400ms+2000ms=2.4s.We will pay for the entire 2.4s. Now think if the external service is down or if it’s taking ten seconds to send the answer because of a timeout.

Since Lambda pricing is proportional to the total execution time, we need a way to know if an external service is acting as a bottleneck. The idea behind the Circuit Breaker pattern is to be able to track errors, and, if we reach a certain threshold, we avoid calling the 3rd party service and we return an error. Then, after a certain timeout, we can try again and check if the service is up and functional.

If you prefer watching a video explaining the pattern, please click on the Youtube Video below:

Architecture

Let’s assume we have a web application where we have implemented the Circuit Breaker pattern. The first step is our customers calling the API through Amazon API Gateway (if you want an introduction to APIGw, check out this video).

Amazon API Gateway will call the Lambda function where the 3rd party service is integrated. The Lambda function, before calling the external service, queries a DynamoDB table to check if there are recent errors for the external service. If the number of errors is less than the threshold, Lambda will call the external service. At this point, there are two possible scenarios:

1. 200 OK: the external service works and sends back the response to the Lambda function which will forward it back to the client 🙂
2. Failure: the external service sends back an error. The Lambda function will catch the error and push an event to Amazon Eventbridge.

Let’s dig into Scenario 2: Amazon Eventbridge will take the event pushed by the Lambda function and trigger a second Lambda function (refer to the architecture diagram above). The Lambda function triggered by Eventbridge will take the event’s payload and put an item inside DynamoDB. The DynamoDB’s item is composed of:

• PK (Partition Key): contains the external service name.
• SK (Sort Key): contains the expiration time
• Event Type: contains metadata about the error type
• TTL: contains the expiration time for the error. On DynamoDB is possible to specify an item’s attribute as the TimeToLive attribute. The attribute must be of Number type and expressed in Seconds. DynamoDB engine will delete the item automatically after the time has expired.

Let’s assume now there is a second API call. The function will check again on DynamoDB if there are any recent errors by querying DynamoDB like so:

var dynamoParams = {    
  ExpressionAttributeValues: {
    ":serviceName":  {"S": serviceName},
    ":now": {"N": now.toString() }},
    KeyConditionExpression: "PK = :serviceName and SK >= :now",
    TableName: config.DYNAMODB_TABLE,   
};   
const errors = await dynamo.query(dynamoParams).promise();

If errors.Count > Threshold , the function will avoid calling the external service and will send immediately an error back to the client. I’ve done a live demo on my Youtube channel:

Conclusion

We have broken down the Circuit Breaker pattern and analysed the advantages. The pattern is available in my Github repo: https://github.com/enricop89/aws-serverless-samples.

Feel free to use it and let me know if you have any questions or feedback.

Follow me on Twitter and Youtube for more!