Circuit breaker design pattern

Circuit breaker is a design pattern used in software development. It is used to detect failures and encapsulates the logic of preventing a failure from constantly recurring, during maintenance, temporary external system failure or unexpected system difficulties. Circuit breaker pattern prevents cascading failures particularly in distributed systems.

According to Marc Brooker, circuit breakers can misinterpret a partial failure as total system failure and inadvertently bring down the entire system. In particular, sharded systems and cell-based architectures are vulnerable to this issue. A workaround is that the server indicates to the client which specific part is overloaded and the client uses a corresponding mini circuit breaker. However, this workaround can be complex and expensive.

Circuit breaker pattern should be used along with other patterns such as retry, fallback and timeout pattern. This helps the system to be more fault tolerant.

Common uses

Assume that an application connects to a database 100 times per second and the database fails. The application designer does not want to have the same error reoccur constantly. They also want to handle the error quickly and gracefully without waiting for TCP connection timeout.

Generally Circuit Breaker can be used to check the availability of an external service. An external service can be a database server or a web service used by the application.

Circuit breaker detects failures and prevents the application from trying to perform the action that is doomed to fail (until it's safe to retry).

Performance implication

While it's safe to say that the benefits outweigh the consequences, implementing Circuit Breaker will negatively affect storage space, application complexity, and computational cost to the executing application. This is because it adds additional code into the execution path to check for the state of the circuit. This can be seen in the PHP example below, where checking APC for the database status costs a few extra cycles. Also, running the circuit breaker code itself consumes resources on the system where it is running, thus leaving less execution power for "real" applications.

By how much depends on the storage layer used and generally available resources. The largest factors in this regard are the type of cache, for example, disk-based vs. memory-based and local vs. network.

Different states of circuit breaker

• Closed
• Open
• Half-open

Closed state

When everything is normal, the circuit breakers remained closed, and all the request passes through to the services as shown below. If the number of failures increases beyond the threshold, the circuit breaker trips and goes into an open state.

Circuit Breaker Closed State

Open state

In this state circuit breaker returns an error immediately without even invoking the services. The Circuit breakers move into the half-open state after a timeout period elapses. Usually, it will have a monitoring system where the timeout will be specified.

Circuit Breaker Open State

Half-open state

In this state, the circuit breaker allows a limited number of requests from the service to pass through and invoke the operation. If the requests are successful, then the circuit breaker will go to the closed state. However, if the requests continue to fail, then it goes back to Open state.

Circuit Breaker Half Open State

External links

• Example of PHP implementation with diagrams
• Example of Retry Pattern with Polly using C#
• Example of C# implementation from Anders Lybeckers using Polly
• Polly NuGet package
• Example of C# implementation from Alexandr Nikitin
• Implementation in Python
• Stability patterns applied in a RESTful architecture
• Martin Fowler Bliki