5.3.6 熔断

熔断，是创建弹性微服务应用程序的重要模式。熔断能够使您的应用程序具备应对来自故障、潜在峰值和其他未知网络因素影响的能力。

这个任务中，你将配置熔断规则，然后通过有意的使熔断器“跳闸”来测试配置。httpbin 作为此任务的后端服务。

1.前置条件

• 安装istio

• 启动httpbin服务

  kubectl apply -f samples/httpbin/httpbin.yaml
  serviceaccount/httpbin created
  service/httpbin created
  deployment.apps/httpbin created
  

2.配置熔断器

如果您的Istio启用了双向TLS身份验证，则必须在应用目标规则之前将TLS流量策略mode：ISTIO_MUTUAL添加到DestinationRule 。否则请求将产生503错误

创建一个 DestinationRule，在调用 httpbin 服务时应用熔断设置

kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: httpbin
spec:
  host: httpbin
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 1
      http:
        http1MaxPendingRequests: 1
        maxRequestsPerConnection: 1
    outlierDetection:
      consecutive5xxErrors: 1
      interval: 1s
      baseEjectionTime: 3m
      maxEjectionPercent: 100
EOF
destinationrule.networking.istio.io/httpbin created

2.验证目标规则是否已经正确创建

kubectl get destinationrule httpbin -o yaml

3.添加一个客户端

创建客户端程序以发送流量到httpbin服务。这是一个名为Fortio的负载测试客户的，其可以控制连接数、并发数及发送HTTP请求的延迟。通过Fortio能够有效的触发前面在DestinationRule中设置的熔断策略。

1.向客户端注入 Istio Sidecar 代理，以便 Istio 对其网络交互进行管理

kubectl apply -f samples/httpbin/sample-client/fortio-deploy.yaml
service/fortio created
deployment.apps/fortio-deploy created

kubectl get po
NAME                             READY   STATUS    RESTARTS   AGE
fortio-deploy-5b4656987b-b58bn   2/2     Running   0          24s
httpbin-779c54bf49-fq7gg         2/2     Running   0          5m23s

2.登入客户端 Pod 并使用 Fortio 工具调用 httpbin 服务

FORTIO_POD=$(kubectl get pod | grep fortio | awk '{ print $1 }')
kubectl exec -it $FORTIO_POD  -c fortio -- /usr/bin/fortio load -curl  http://httpbin:8000/get
HTTP/1.1 200 OK
server: envoy
date: Sat, 21 Aug 2021 08:12:34 GMT
content-type: application/json
content-length: 594
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 97

{
  "args": {},
  "headers": {
    "Host": "httpbin:8000",
    "User-Agent": "fortio.org/fortio-1.11.3",
    "X-B3-Parentspanid": "9b943656befa52a8",
    "X-B3-Sampled": "1",
    "X-B3-Spanid": "fdd35d9a9896ae10",
    "X-B3-Traceid": "f6d81ba2c71f637e9b943656befa52a8",
    "X-Envoy-Attempt-Count": "1",
    "X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/default/sa/httpbin;Hash=9004f620fef03d4b69798ee69f7270fb9380771d145b45df6e9dc1e0903772af;Subject=\"\";URI=spiffe://cluster.local/ns/default/sa/default"
  },
  "origin": "127.0.0.6",
  "url": "http://httpbin:8000/get"
}

看到调用后端服务的请求已经成功！接下来，可以测试熔断。

4.触发熔断器

在DestinationRule配置中，您定义了maxConnections: 1和http1MaxPendingRequests: 1。这些规则意味着，如果并发的连接和请求数超过一个，在istio-proxy进行进一步的请求和连接时，后续请求或连接将被阻止。

1.发送并发数为2的连接（-c 2），请求20次（-n 20）

kubectl exec -it $FORTIO_POD  -c fortio -- /usr/bin/fortio load -c 2 -qps 0 -n 20 -loglevel Warning http://httpbin:8000/get

输出如下

08:15:07 I logger.go:127> Log level is now 3 Warning (was 2 Info)
Fortio 1.11.3 running at 0 queries per second, 4->4 procs, for 20 calls: http://httpbin:8000/get
Starting at max qps with 2 thread(s) [gomax 4] for exactly 20 calls (10 per thread + 0)
08:15:07 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:15:07 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:15:07 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:15:07 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:15:07 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:15:07 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
Ended after 144.883411ms : 20 calls. qps=138.04
Aggregated Function Time : count 20 avg 0.013612914 +/- 0.01866 min 0.000777306 max 0.07671694 sum 0.27225829
# range, mid point, percentile, count
>= 0.000777306 <= 0.001 , 0.000888653 , 15.00, 3
> 0.001 <= 0.002 , 0.0015 , 20.00, 1
> 0.005 <= 0.006 , 0.0055 , 30.00, 2
> 0.006 <= 0.007 , 0.0065 , 55.00, 5
> 0.007 <= 0.008 , 0.0075 , 65.00, 2
> 0.009 <= 0.01 , 0.0095 , 70.00, 1
> 0.01 <= 0.011 , 0.0105 , 75.00, 1
> 0.012 <= 0.014 , 0.013 , 80.00, 1
> 0.014 <= 0.016 , 0.015 , 85.00, 1
> 0.03 <= 0.035 , 0.0325 , 90.00, 1
> 0.05 <= 0.06 , 0.055 , 95.00, 1
> 0.07 <= 0.0767169 , 0.0733585 , 100.00, 1
# target 50% 0.0068
# target 75% 0.011
# target 90% 0.035
# target 99% 0.0753736
# target 99.9% 0.0765826
Sockets used: 8 (for perfect keepalive, would be 2)
Jitter: false
Code 200 : 14 (70.0 %)
Code 503 : 6 (30.0 %)
Response Header Sizes : count 20 avg 161.15 +/- 105.5 min 0 max 231 sum 3223
Response Body/Total Sizes : count 20 avg 649.25 +/- 267.3 min 241 max 825 sum 12985
All done 20 calls (plus 0 warmup) 13.613 ms avg, 138.0 qps

2.将并发连接数提高到3个

kubectl exec -it $FORTIO_POD  -c fortio -- /usr/bin/fortio load -c 3 -qps 0 -n 30 -loglevel Warning http://httpbin:8000/get

输出如下

08:16:44 I logger.go:127> Log level is now 3 Warning (was 2 Info)
Fortio 1.11.3 running at 0 queries per second, 4->4 procs, for 30 calls: http://httpbin:8000/get
Starting at max qps with 3 thread(s) [gomax 4] for exactly 30 calls (10 per thread + 0)
08:16:44 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:16:44 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:16:44 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:16:44 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:16:44 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:16:44 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:16:44 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:16:44 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:16:44 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:16:44 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:16:44 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:16:44 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:16:44 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:16:44 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:16:44 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:16:44 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:16:44 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:16:44 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:16:44 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:16:44 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
08:16:44 W http_client.go:693> Parsed non ok code 503 (HTTP/1.1 503)
Ended after 54.153348ms : 30 calls. qps=553.98
Aggregated Function Time : count 30 avg 0.0041682143 +/- 0.005383 min 0.000507205 max 0.018732702 sum 0.12504643
# range, mid point, percentile, count
>= 0.000507205 <= 0.001 , 0.000753603 , 40.00, 12
> 0.001 <= 0.002 , 0.0015 , 63.33, 7
> 0.002 <= 0.003 , 0.0025 , 70.00, 2
> 0.005 <= 0.006 , 0.0055 , 73.33, 1
> 0.006 <= 0.007 , 0.0065 , 76.67, 1
> 0.007 <= 0.008 , 0.0075 , 80.00, 1
> 0.008 <= 0.009 , 0.0085 , 83.33, 1
> 0.009 <= 0.01 , 0.0095 , 86.67, 1
> 0.01 <= 0.011 , 0.0105 , 90.00, 1
> 0.016 <= 0.018 , 0.017 , 93.33, 1
> 0.018 <= 0.0187327 , 0.0183664 , 100.00, 2
# target 50% 0.00142857
# target 75% 0.0065
# target 90% 0.011
# target 99% 0.0186228
# target 99.9% 0.0187217
Sockets used: 23 (for perfect keepalive, would be 3)
Jitter: false
Code 200 : 9 (30.0 %)
Code 503 : 21 (70.0 %)
Response Header Sizes : count 30 avg 69.1 +/- 105.6 min 0 max 231 sum 2073
Response Body/Total Sizes : count 30 avg 416 +/- 267.3 min 241 max 825 sum 12480
All done 30 calls (plus 0 warmup) 4.168 ms avg, 554.0 qps

现在，您将开始看到预期的熔断行为，只有30%的请求成功，其余的均被熔断器拦截

Code 200 : 9 (30.0 %) Code 503 : 21 (70.0 %)

3.查询istio-proxy状态以了解更多熔断详情

kubectl exec $FORTIO_POD -c istio-proxy -- pilot-agent request GET stats | grep httpbin | grep pending

cluster.outbound|8000||httpbin.default.svc.cluster.local.circuit_breakers.default.remaining_pending: 1
cluster.outbound|8000||httpbin.default.svc.cluster.local.circuit_breakers.default.rq_pending_open: 0
cluster.outbound|8000||httpbin.default.svc.cluster.local.circuit_breakers.high.rq_pending_open: 0
cluster.outbound|8000||httpbin.default.svc.cluster.local.upstream_rq_pending_active: 0
cluster.outbound|8000||httpbin.default.svc.cluster.local.upstream_rq_pending_failure_eject: 0
cluster.outbound|8000||httpbin.default.svc.cluster.local.upstream_rq_pending_overflow: 27
cluster.outbound|8000||httpbin.default.svc.cluster.local.upstream_rq_pending_total: 24

看到upstream_rq_pending_overflow的值为27，代表27个已经标识为熔断

5.清理

1.清理规则

kubectl delete destinationrule httpbin

2.下线httpbin服务和客户端

kubectl delete deploy httpbin fortio-deploy
kubectl delete svc httpbin