Introduction

In modern distributed systems, failures are inevitable. Whether it’s a third-party API or a database connection, a single failure can cascade, affecting multiple components of your system. One solution to prevent this is the circuit breaker pattern.

This blog post will cover:
✔️ What is a circuit breaker pattern?
✔️ Why use it in Ruby on Rails applications?
✔️ How to implement a circuit breaker in Rails
✔️ Handling errors and managing fallback strategies

1. What is the Circuit Breaker Pattern?

The circuit breaker is a design pattern that prevents a failure in one part of your system from spreading to others. It works by monitoring the system’s interactions and “tripping” (breaking the circuit) when a failure threshold is reached, thus halting further calls to the failing service until it recovers.

States of a Circuit Breaker
  • Closed: The circuit is closed, and requests are allowed to go through.
  • Open: The circuit is open, and requests are blocked from going through.
  • Half-Open: The circuit is in a testing phase where some requests are allowed to pass to check if the system has recovered.

2. Why Use the Circuit Breaker Pattern in Rails?

In a Rails application, particularly with microservices or services interacting with external APIs, the failure of one service can propagate and affect the entire system. Here’s why implementing the circuit breaker pattern can be highly beneficial:

  • Fault Isolation: Prevents failure in one component from affecting the entire application.
  • Graceful Degradation: If a service fails, it doesn’t crash the whole system, and you can provide an alternative response.
  • Improved Resilience: Automatically reopens the circuit when the service recovers, reducing the need for manual intervention.
  • API Reliability: In cases of external API failures, you can avoid retry storms and further overload the service.

3. Setting Up Circuit Breaker in Rails

Using the circuitbox Gem

One of the best ways to implement the circuit breaker pattern in Rails is by using the circuitbox gem, which integrates seamlessly with Ruby on Rails.

Installation

Add the gem to your Gemfile:

gem "circuitbox"

Run:

bundle install
Basic Configuration

In Rails, you can configure circuit breakers globally or on a per-service basis. Here’s a basic setup for an external API call:

class ExternalApiService
def self.call
circuit.run do
# The API call that might fail
response = Faraday.get("https://api.example.com/data")
JSON.parse(response.body)
end
end

def self.circuit
@circuit ||= Circuitbox.circuit(:external_api) do |circuit|
circuit.failure_threshold = 5        # Number of failed requests before opening the circuit
circuit.retry_timeout = 30            # Time in seconds before retrying
circuit.timeout = 10                  # Timeout before considering the request a failure
circuit.time_window = 60              # Time window to track failures
end
end
end

In this example, the ExternalApiService.call method makes an API call that may fail. The circuit breaker will open if it detects five consecutive failures within a 60-second window, and the retry_timeout will ensure the system waits for 30 seconds before trying again.

4. Circuit Breaker States and Management

Closing and Opening the Circuit

When a service fails repeatedly, the circuit will open, stopping further calls from being made. You can easily check if the circuit is open or closed by using:

if circuit.open?
# Handle the failure gracefully (return fallback data or a predefined response)
else
# Proceed with the request
end
Handling Failures with Fallbacks

When the circuit breaker is open, it’s crucial to provide a fallback mechanism. For example, you might want to return a cached response or a default value instead of trying to call the external service again.

def self.call
circuit.run do
response = Faraday.get("https://api.example.com/data")
JSON.parse(response.body)
end
rescue Circuitbox::OpenCircuitError => e
# Return fallback data when the circuit is open
{ error: "Service is temporarily unavailable. Please try again later." }
end

By catching the Circuitbox::OpenCircuitError, you can handle the error gracefully and provide a fallback response, thus reducing downtime for your users.

5. Monitoring and Metrics

To ensure that the circuit breaker is performing as expected, it’s important to keep track of its state and behavior. circuitbox provides built-in metrics for monitoring the number of failures, retries, and the state of the circuit. You can integrate it with tools like Prometheus, New Relic, or Datadog to keep track of circuit state in real time.

Example of logging circuit status:

logger.info("Circuit state: #{circuit.state}")

6. Advanced Usage: Combining Circuit Breaker with Retry Logic

You can combine the circuit breaker pattern with retry logic for more sophisticated handling. For instance, you may want to retry a failed request a few times before opening the circuit.

def self.call
retries = 3
begin
circuit.run { external_api_call }
rescue Faraday::TimeoutError, Faraday::ConnectionFailed => e
retries -= 1
retry if retries > 0
raise "External API is down. Could not retrieve data."
end
end

This allows you to retry failed requests a certain number of times before ultimately opening the circuit.

7. Conclusion

Implementing the circuit breaker pattern in Rails applications enhances resilience and protects your system from cascading failures. With gems like circuitbox, integrating this pattern into your application is straightforward and effective.

Incorporating the circuit breaker pattern leads to:
✔️ Better fault tolerance
✔️ Automatic recovery after service failures
✔️ Improved system stability and user experience

By implementing this pattern, you can significantly reduce downtime, increase the reliability of your application, and protect your resources from excessive load during failure scenarios.

Next Steps

To further improve your Rails application’s fault tolerance, consider implementing additional patterns like retry patterns, rate limiting, or bulkheads.