Skip to content

Envoy Module Architecture and Configuration

Overview

The Envoy Proxy is a proxy and communication bus designed for large modern service oriented architectures. It can be used as a frontend proxy and load balancer to manage multiple upstream services and can also be used to build a mesh network for tunneling all network communication to provide consistent visibility (metrics, tracing, etc.). Envoy supports adding services via gRPC. In the case of the agent support, the agent extends Envoy by implementing two services: The gRPC External Authorization (ext_authz) for visibility into ingress traffic and the HTTP gRPC Access Log Service (http_grpc_access_log) for visibility into egress traffic.

See: - About Envoy - Starting point for configuration

API Versioning

As of Envoy v1.18 the V2 API has been removed and is no longer supported. In later versions quite a few filter names and types have changed and these will be noted. In general, you should be able to use the V3 API in most cases (even if V2 is still supported in that version of Envoy). You may need to explicitly configure the V3 API to do this (See here)

See here

Architecture and Configuration

See here

SigSci Agent Configuration

For Envoy to communicate with the SigSci Agent, the agent needs to be running in Envoy gRPC proxy mode. To do this, the SigSci Agent envoy-grpc-address (or SIGSCI_ENVOY_GRPC_ADDRESS env var) must be configured. In addition, you should enable the envoy-expect-response-data (or SIGSCI_ENVOY_EXPECT_RESPONSE_DATA env var) configuration option.

Example Docker Compose configuration:

services:
  sigsci-agent-grpc:
    image: "${SIGSCI_AGENT_IMAGE:-signalsciences/sigsci-agent}"
    deploy:
      resources:
        limits:
          cpus: "${SIGSCI_AGENT_CPUS:-2}"
    environment:
    - "SIGSCI_ACCESSKEYID=${SIGSCI_ACCESSKEYID:-}"
    - "SIGSCI_SECRETACCESSKEY=${SIGSCI_SECRETACCESSKEY:-}"
    - "SIGSCI_ENVOY_GRPC_ADDRESS=${SIGSCI_ENVOY_GRPC_ADDRESS:-sigsci-agent-grpc:8085}"
#    - "SIGSCI_ENVOY_GRPC_CERT=${SIGSCI_ENVOY_GRPC_CERT:-/local/signalsciences-dev.net.crt.pem}"
#    - "SIGSCI_ENVOY_GRPC_KEY=${SIGSCI_ENVOY_GRPC_KEY:-/local/signalsciences-dev.net.key.pem}"
    - "SIGSCI_ENVOY_GRPC_LOG_LEVEL=${SIGSCI_ENVOY_GRPC_LOG_LEVEL:-3}"
    - "SIGSCI_ENVOY_EXPECT_RESPONSE_DATA=${SIGSCI_ENVOY_EXPECT_RESPONSE_DATA:-0}"
#    - "SIGSCI_MAX_CONNECTIONS=${SIGSCI_MAX_CONNECTIONS:-0}"
    - "SIGSCI_MAX_PROCS=${SIGSCI_MAX_PROCS:-100%}"
#    - "SIGSCI_JAEGER_TRACING=${SIGSCI_JAEGER_TRACING:-false}"
#    # Jaeger config
#    - "JAEGER_AGENT_HOST=${JAEGER_AGENT_HOST:-jaeger}"
#    - "JAEGER_SAMPLER_TYPE=${JAEGER_SAMPLER_TYPE:-probabilistic}"
#    - "JAEGER_SAMPLER_PARAM=${JAEGER_SAMPLER_PARAM:-1}"
#    #- "JAEGER_ENDPOINT=${JAEGER_ENDPOINT:-http://jaeger:9411/api/v2/spans}"
    expose:
    - "8085"
    volumes:
    - /var/tmp:/sigsci/tmp
    - ./local:/local
    - ./logs:/logs

Envoy Configuration

Generally three items will need to be configured for full functionality.

SigSci Agent Cluster

Envoy abstracts external communication via clusters. The cluster links a name with addressing information. This cluster name is then used in the rest of the configuration. In this case the cluster is only a single SigSci Agent. The specific address and port are not important, but Envoy will need to be able to resolve these to a SigSci Agent running with the same values.

  • Using Docker Compose or similar, the address will typically be the configured service name (e.g., "sigsci-agent-grpc")
  • Using Kubernetes and running the SigSci Agent as a sidecar in the same pod, the address will typically be 127.0.0.1 or localhost which is the shared inter-pod address
  • Using something outside of containers this may be a DNS resolvable hostname or IP address (or UNIX Domain socket file)

Here is an example typical in Docker Compose where you have defined the SigSci agent service sigsci-agent-grpc:

static_resources:
  ...

  clusters:
  - name: sigsci-agent-grpc
    type: STRICT_DNS
    http2_protocol_options: {}
    connect_timeout: 0.25s
    load_assignment:
      cluster_name: sigsci-agent-grpc
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              ### Use a UNIX Domain socket (pipe) OR TCP socket address
              #pipe:
              #  path: /sigsci/tmp/sigsci-grpc.sock
              socket_address:
                address: sigsci-agent-grpc
                port_value: 8085

gRPC External Authorization HTTP Filter

To inspect the HTTP request, Envoy will need to call out to the SigSci Agent running as a gRPC server (in Envoy gRPC mode). To do this, you will need to configure an HTTP filter BEFORE the HTTP router filter.

Using the V3 API this would be something like this under the appropriate HTTP Connection Manager:

static_resources:
  listeners:

  ...

  - address:

    ...

    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          http_filters:
          - name: envoy.filters.http.ext_authz
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
              transport_api_version: V3
              grpc_service:
                envoy_grpc:
                  ### NOTE: Use your own cluster name here
                  cluster_name: sigsci-agent-grpc
                timeout: 0.2s
              filter_enabled:
                default_value:
                  numerator: 100
                  denominator: HUNDRED
                runtime_key: "sigsci-agent-grpc.sampling"
              failure_mode_allow: true
              with_request_body:
                max_request_bytes: 8388608
                allow_partial_message: true
          - name: envoy.filters.http.router
            typed_config: {}

HTTP gRPC Access Log Service

Similar to inspecting the HTTP request, you should also configure inspecting the HTTP response. While this is not required it will limit the functionality of the product if you do not configure this as you will not see the HTTP response status code or headers recorded. If this is used (and again you should), then the SigSci Agent also needs to be configured to expect the response data (e.g., set env SIGSCI_ENVOY_EXPECT_RESPONSE_DATA=1 in the SigSci Agent). To configure Envoy, you need to add a gRPC access log service that points to the SigSci Agent (cluster). Note that the access log service will need to be configured to send some additional headers as all are not sent by default.

Using the V3 API this would be something like this under the appropriate HTTP Connection Manager:

static_resources:
  listeners:

  ...

  - address:

    ...

          access_log:
          - name: envoy.access_loggers.http_grpc
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.grpc.v3.HttpGrpcAccessLogConfig
              common_config:
                log_name: "sigsci-agent-grpc"
                transport_api_version: V3
                grpc_service:
                  envoy_grpc:
                    ### NOTE: Use your own cluster name here
                    cluster_name: sigsci-agent-grpc
                  timeout: 0.02s
              additional_request_headers_to_log:
              # These are required:
              - "x-sigsci-request-id"
              - "x-sigsci-waf-response"
              # These are additional you want recorded:
              - "accept"
              - "content-type"
              - "content-length"
              additional_response_headers_to_log:
              # These are additional you want recorded:
              - "date"
              - "server"
              - "content-type"
              - "content-length"