Introduction to Circuit Breakers
In the world of microservices, where multiple services collaborate to handle requests, the risk of cascading failures is ever-present. Imagine a scenario where one service is down or responding slowly, causing a chain reaction that brings down the entire system. This is where the Circuit Breaker pattern comes into play, acting as a guardian that prevents such catastrophic failures.
What is a Circuit Breaker?
A Circuit Breaker is a design pattern that prevents a network or service failure from cascading to other services. It works similarly to an electrical circuit breaker, which trips and breaks the circuit when the current exceeds a certain threshold, preventing further damage.
States of a Circuit Breaker
A Circuit Breaker has three primary states:
Closed
In the Closed state, the circuit breaker allows all requests to pass through to the service. If a certain number of consecutive requests fail (reaching a failure threshold), the circuit breaker switches to the Open state.
Open
In the Open state, all requests are immediately blocked, and an error is returned to the caller without attempting to contact the failing service. This gives the service time to recover and prevents it from being overwhelmed.
Half-Open
After a predefined recovery period, the circuit breaker transitions to the Half-Open state. Here, it allows a limited number of test requests to see if the service has recovered. If these requests succeed, the circuit breaker transitions back to the Closed state. If any of these requests fail, it goes back to the Open state.
Implementing a Circuit Breaker in Go
To implement a Circuit Breaker in Go, you can either use an existing library like gobreaker or create your own from scratch.
Using the gobreaker Library
The gobreaker library is a popular choice for implementing Circuit Breakers in Go. Here’s how you can use it:
package main
import (
"fmt"
"log"
"net/http"
"time"
"github.com/sony/gobreaker"
)
func main() {
var st gobreaker.Settings
st.Name = "Circuit Breaker PoC"
st.Timeout = time.Second * 5
st.MaxRequests = 2
st.ReadyToTrip = func(counts gobreaker.Counts) bool {
return counts.ConsecutiveFailures >= 1
}
cb := gobreaker.NewCircuitBreaker(st)
url := "http://localhost:8080/success"
cb.Execute(func() (int, error) { return Get(url) })
fmt.Println("Circuit Breaker state:", cb.State()) // closed!
url = "http://localhost:8080/failure"
cb.Execute(func() (int, error) { return Get(url) })
fmt.Println("Circuit Breaker state:", cb.State()) // open!
time.Sleep(time.Second * 6)
url = "http://localhost:8080/success"
cb.Execute(func() (int, error) { return Get(url) })
fmt.Println("Circuit Breaker state:", cb.State()) // half-open!
url = "http://localhost:8080/success"
cb.Execute(func() (int, error) { return Get(url) })
fmt.Println("Circuit Breaker state:", cb.State()) // closed!
}
func Get(url string) (int, error) {
r, _ := http.Get(url)
if r.StatusCode != http.StatusOK {
return r.StatusCode, fmt.Errorf("failed to get %s", url)
}
return r.StatusCode, nil
}
Creating a Circuit Breaker from Scratch
If you prefer to create your own Circuit Breaker, here’s a simple example:
package main
import (
"fmt"
"log"
"net/http"
"sync"
"time"
)
const (
Closed = "closed"
Open = "open"
HalfOpen = "half-open"
)
type circuitBreaker struct {
mu sync.Mutex
state string
failureCount int
lastFailureTime time.Time
halfOpenSuccessCount int
failureThreshold int
recoveryTime time.Duration
halfOpenMaxRequests int
timeout time.Duration
}
func NewCircuitBreaker(failureThreshold int, recoveryTime time.Duration, halfOpenMaxRequests int, timeout time.Duration) *circuitBreaker {
return &circuitBreaker{
state: Closed,
failureThreshold: failureThreshold,
recoveryTime: recoveryTime,
halfOpenMaxRequests: halfOpenMaxRequests,
timeout: timeout,
}
}
func (cb *circuitBreaker) Call(fn func() (any, error)) (any, error) {
cb.mu.Lock()
defer cb.mu.Unlock()
switch cb.state {
case Closed:
return cb.handleClosedState(fn)
case Open:
return cb.handleOpenState()
case HalfOpen:
return cb.handleHalfOpenState(fn)
default:
return nil, fmt.Errorf("unknown circuit state")
}
}
func (cb *circuitBreaker) handleClosedState(fn func() (any, error)) (any, error) {
result, err := fn()
if err != nil {
cb.failureCount++
cb.lastFailureTime = time.Now()
if cb.failureCount >= cb.failureThreshold {
cb.state = Open
}
return nil, err
}
cb.failureCount = 0
return result, nil
}
func (cb *circuitBreaker) handleOpenState() (any, error) {
if time.Since(cb.lastFailureTime) > cb.recoveryTime {
cb.state = HalfOpen
}
return nil, fmt.Errorf("circuit is open")
}
func (cb *circuitBreaker) handleHalfOpenState(fn func() (any, error)) (any, error) {
result, err := fn()
if err != nil {
cb.state = Open
return nil, err
}
cb.halfOpenSuccessCount++
if cb.halfOpenSuccessCount >= cb.halfOpenMaxRequests {
cb.state = Closed
}
return result, nil
}
func main() {
cb := NewCircuitBreaker(3, time.Second*5, 2, time.Second*1)
url := "http://localhost:8080/success"
result, err := cb.Call(func() (any, error) { return Get(url) })
fmt.Println("Circuit Breaker state:", cb.state)
url = "http://localhost:8080/failure"
result, err = cb.Call(func() (any, error) { return Get(url) })
fmt.Println("Circuit Breaker state:", cb.state)
time.Sleep(time.Second * 6)
url = "http://localhost:8080/success"
result, err = cb.Call(func() (any, error) { return Get(url) })
fmt.Println("Circuit Breaker state:", cb.state)
}
func Get(url string) (int, error) {
r, _ := http.Get(url)
if r.StatusCode != http.StatusOK {
return r.StatusCode, fmt.Errorf("failed to get %s", url)
}
return r.StatusCode, nil
}
Benefits and Challenges
Benefits
- Prevents Cascading Failures: By tripping the circuit when a service is failing, you prevent other services from being overwhelmed with requests.
- Reduces Latency: Immediate failure in the Open state reduces the latency caused by waiting for a failing service to respond.
- Improves Resilience: Allows the system to recover more quickly by giving the failing service time to recover without additional load.
Challenges
- Choosing Thresholds: It can be challenging to choose the right failure thresholds and recovery times without introducing false positives or excessive latency.
- Complexity: Implementing a Circuit Breaker can add complexity to your codebase, especially if you are creating it from scratch.
Real-World Example
Let’s consider a real-world example where we have a microservice architecture with multiple services interacting with each other. One of these services, UserService, depends on another service, PaymentService, to process payments.
In this scenario, if PaymentService starts failing, the Circuit Breaker will trip and prevent further requests from reaching PaymentService, thus preventing cascading failures.
Conclusion
The Circuit Breaker pattern is a powerful tool in the arsenal of any microservice architect. By preventing cascading failures and reducing latency, it significantly improves the resilience of your system. Whether you choose to use an existing library like gobreaker or implement your own Circuit Breaker from scratch, the benefits are well worth the effort.
So the next time your services start to feel like they’re in a game of dominoes, just remember: a Circuit Breaker can be the safety net that keeps everything standing tall. Happy coding