Why Resilient Microservices Require Circuit Breakers
Under peak loads, backend microservices dispatching email notifications can spawn tens of thousands of concurrent goroutines. If an upstream network bottleneck, DNS resolution stall, or provider rate limit occurs, unbounded goroutines will stack up waiting on hung network sockets.
Within minutes, the host container exhausts Linux file descriptors (too many open files), leading to catastrophic cascading failure across the entire backend cluster. By wrapping outbound dispatches in a Sony GoBreaker circuit breaker, the service fails fast as soon as error thresholds are breached, protecting upstream systems and shedding load gracefully.
1. Production Go Client with GoBreaker (client.go)
This production-grade Go client configures a shared http.Client with custom transport connection pools, context timeouts, and structured logging using Go 1.21+ slog.
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"log/slog"
"net/http"
"time"
"github.com/sony/gobreaker"
)
type EmailClient struct {
apiKey string
httpClient *http.Client
cb *gobreaker.CircuitBreaker
logger *slog.Logger
}
type SendRequest struct {
To string `json:"to"`
Subject string `json:"subject"`
Text string `json:"text"`
HTML string `json:"html,omitempty"`
}
func NewEmailClient(apiKey string, logger *slog.Logger) *EmailClient {
// Configure Circuit Breaker: Tripped when 5 consecutive failures occur within 30s
st := gobreaker.Settings{
Name: "SadaSendOutboundBreaker",
MaxRequests: 3,
Interval: 30 * time.Second,
Timeout: 10 * time.Second,
ReadyToTrip: func(counts gobreaker.Counts) bool {
failureRatio := float64(counts.TotalFailures) / float64(counts.Requests)
return counts.Requests >= 5 && failureRatio >= 0.6
},
OnStateChange: func(name string, from gobreaker.State, to gobreaker.State) {
logger.Warn("circuit breaker state transition", "name", name, "from", from.String(), "to", to.String())
},
}
transport := &http.Transport{
MaxIdleConns: 100,
MaxIdleConnsPerHost: 20,
IdleConnTimeout: 90 * time.Second,
}
return &EmailClient{
apiKey: apiKey,
httpClient: &http.Client{
Transport: transport,
Timeout: 5 * time.Second,
},
cb: gobreaker.NewCircuitBreaker(st),
logger: logger,
}
}
func (c *EmailClient) Send(ctx context.Context, req SendRequest) error {
_, err := c.cb.Execute(func() (interface{}, error) {
payload, err := json.Marshal(req)
if err != nil {
return nil, fmt.Errorf("marshal payload: %w", err)
}
httpReq, err := http.NewRequestWithContext(ctx, "POST", "https://api.sadasend.com/emails", bytes.NewBuffer(payload))
if err != nil {
return nil, fmt.Errorf("create request: %w", err)
}
httpReq.Header.Set("Authorization", "Bearer "+c.apiKey)
httpReq.Header.Set("Content-Type", "application/json")
resp, err := c.httpClient.Do(httpReq)
if err != nil {
c.logger.Error("outbound network error", "err", err)
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode >= 500 {
return nil, fmt.Errorf("upstream server error: status %d", resp.StatusCode)
}
if resp.StatusCode >= 400 {
// Client errors (4xx) should not trip the circuit breaker
c.logger.Warn("client rejection from sadasend", "status", resp.StatusCode)
return nil, nil
}
return nil, nil
})
return err
}Circuit Breaker State Machine & Failure Classification
| HTTP Response | Circuit Breaker Action | Microservice Behavior | Engineering Remedy |
|---|---|---|---|
| 200 OK / 202 Accepted | Count as Success | Acknowledge caller immediately | Normal operations. |
| 400 Bad Request | Do NOT count as Failure | Return 400 to caller | Fix invalid email syntax or payload schema. |
| 403 Forbidden | Do NOT count as Failure | Alert on allowlist breach | Register domain in SadaSend dashboard. |
| 429 Too Many Requests | Count as Partial Failure | Apply exponential jitter backoff | Upgrade tier or throttle sender concurrency. |
| 500 / 503 Gateway Error | Count as Breaker Failure | Trip to OPEN after threshold | Shed load; retry pending jobs from DB. |
Building AI agents that send email?
Scoped API keys, per-key recipient allowlists, approval mode and a hosted MCP server with ten tools — on the free plan, without a card.