Envoy Module Troubleshooting Guide
Overview
Envoy Issues
ERROR: The v2 xDS major version is deprecated and disabled by default. Support for v2 will be removed from Envoy at the start of Q1 2021.
This means that the V2 API was used. Most likely you forgot a transport_api_version: V3 in a filter or are using the old V2 config.
Envoy is returning 403 with SigSci enabled
Typically this means that the external authorization filter was not configured to fail open. Make sure you are setting failure_mode_allow: true.
See:
- The note on failure_mode_allow
- HTTP filter v3 API reference
Signal Sciences Issues
There are no response headers in the request details and all status codes are 200
You probably do not have the HTTP gRPC Access Log Service configured in Envoy. Or, you do not have the agent configured with envoy-expect-response-data enabled.
Only anomalies seem to be detected - no attacks
You probably only have the HTTP gRPC Access Log Service configured (inspects the response) without the gRPC External Authorization HTTP Filter configuration (inspects the request).
See: install guides here
Debugging
SigSci Agent
The primary way to debug in the agent is to raise the log level via the SigSci Agent (envoy-grpc-log-level) config option.
- Range: 0-9
- Typical value 3 or 4
- Good initial debug value is 6 (logs gRPC calls, but without data)
- To also get data in the gRPC call set it to 7
Jaeger Tracing
TBD: Description
Enable Jaeger tracing in the SigSci Agent config.
Enable Jaeger tracing in Envoy
And make sure to propagate the tracing headers in the gRPC access log service config by including the additional headers in the lists:
EX:
static_resources:
listeners:
...
- address:
...
access_log:
- name: envoy.access_loggers.http_grpc
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.grpc.v3.HttpGrpcAccessLogConfig
common_config:
log_name: "sigsci-agent-grpc"
transport_api_version: V3
grpc_service:
envoy_grpc:
### NOTE: Use your own cluster name here
cluster_name: sigsci-agent-grpc
timeout: 0.02s
additional_request_headers_to_log:
# These are required:
- "x-sigsci-request-id"
- "x-sigsci-waf-response"
# These are additional you want recorded:
- "accept"
- "content-type"
- "content-length"
######## Required for tracing
- "x-request-id"
- "x-b3-traceid"
- "x-b3-spanid"
- "x-b3-parentspanid"
- "x-b3-sampled"
- "b3"
additional_response_headers_to_log:
# These are additional you want recorded:
- "date"
- "server"
- "content-type"
- "content-length"
######## Required for tracing
- "x-request-id"
- "x-b3-traceid"
- "x-b3-spanid"
- "x-b3-parentspanid"
- "x-b3-sampled"
- "b3"
Istio
TODO: This needs more work
Common commands
Get all the objects that are running. Here you can see the pods and you will need the identifiers there.
kubectl get all
Command to get the envoy config. Here you can check that filters are applied
istioctl proxy-config listeners -o json {pod} > envoy-config.json
Command to get logs for envoy or agent
kubectl logs {pod} {envoy-proxy|sigsci-agent}
Istio v1.7+
Istio v1.7+ will need v3 config. These are the changes needed:
--- sigsci-helloworld_istio-filters.yaml 2021-04-13 10:25:15.000000000 -0400
+++ sigsci-helloworld_istio-filters_v3.yaml 2021-08-19 14:28:05.000000000 -0400
@@ -26,8 +26,10 @@
operation: INSERT_BEFORE
value:
# Configure the envoy.ext_authz here:
- name: envoy.ext_authz
- config:
+ name: envoy.filters.http.ext_authz
+ typed_config:
+ "@type": "type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz"
+ transport_api_version: "V3"
grpc_service:
# NOTE: *SHOULD* use envoy_grpc as ext_authz can use dynamic clusters and has connection pooling
envoy_grpc:
@@ -52,13 +54,15 @@
value:
name: "envoy.http_connection_manager"
typed_config:
- "@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager"
+ "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
access_log:
# Configure the envoy.http_grpc_access_log here:
- name: "envoy.http_grpc_access_log"
- config:
+ typed_config:
+ "@type": "type.googleapis.com/envoy.extensions.access_loggers.grpc.v3.HttpGrpcAccessLogConfig"
common_config:
log_name: "sigsci-agent-grpc"
+ transport_api_version: "V3"
grpc_service:
# NOTE: *MUST* use google_grpc as envoy_grpc cannot handle a dynamic cluster for ALS (yet)
google_grpc:
Istio Gateway
TODO: verify this
The VirtualService can show if the hosts are registered to the gateway correctly.
kubectl get virtualservice -o=yaml
To dump the underlying envoy config you need to find the pod and then fetch the proxy config:
# find istio ingress gateway pod
kubectl get pods -n istio-system -l app=istio-ingressgateway
#
istioctl proxy-config listeners -n istio-system -o=json {pod-name}
To enable envoy debug logging:
# enable debugging on envoy
kubectl exec --namespace=istio-system {pod-name} -c istio-proxy -- curl -X POST 'http://localhost:15000/logging?level=debug'
Logs can then be fetched with the normal kubectl log command.
Other useful commands (TODO: validate and document):
istioctl proxy-config log {ingress-pod}.istio-system --level debug
istioctl dashboard envoy {ingress-pod}.istio-system