Skip to content

Sigsci Agent Troubleshooting and Diagnostics Guide

Common Problems and Solutions

Unable to start: accesskeyid is required

The agent MUST be configured with an accesskeyid and secretaccesskey to function normally with uploads and downloads. If you are testing an agent and do not want this, then you can start with the debug-standalone = 3 option that disables uploads and downloads.

Unable to start: secretaccesskey is required

The agent MUST be configured with an accesskeyid and secretaccesskey to function normally with uploads and downloads. If you are testing an agent and do not want this, then you can start with the debug-standalone = 3 option that disables uploads and downloads.

Started legacy RPC listener on "unix:/tmp/sigsci-lua"

The new default location of the UNIX domain socket file was moved from /tmp/sigsci-lua to /var/run/sigsci.sock. In order to handle older modules where this change was not updated, the agent now starts two UNIX domain socket files IF the agent configuration does not specify a rpc-address configuration. If the module uses the new location, then you can get rid of this log by configuring rpc-address = "unix:/var/run/sigsci.sock" or similar.

Failed to start RPC listener on "unix:/var/run/sigsci.sock": listen unix /var/run/sigsci.sock: bind: permission denied

The location of the RPC UNIX domain socket file is not writable by the user running the agent. Normally the agent is run as root, but if it is not, then it probably does not have permissions to write to /var/run or similar. To get around this, you must start the agent as root OR configure the module with another RPC socket location AND configure the agent with a corresponding rpc-address option.

RPC server shutting down followed by Starting RPC server with RPC-vN

You might see something like this in the agent logs where the RPC server restarts when there is a configuration update:

2017/10/23 19:47:05.862162 1 config updated
2017/10/23 19:47:05.862228 RPC server shutting down
2017/10/23 19:47:05.865060 Starting RPC server with RPC-v0
2017/10/23 19:47:05.865199 Started RPC listener on "unix:/tmp/sigsci.sock"

This is normal - just a bit verbose. The RPC service in the agent will restart to reload the config when it is updated. As we move to full reloadable configs this may go away (or if we add detection for when configs change for a service).

ERROR initial upload: upload failure: Post https://c.signalsciences.net/0/push:

The agent failed to do its initial upload (startup log entries, etc.) to the collector. The collector may be down, or the network is unavailable. Make sure the host's network can connect externally to the given URL.

ERROR periodic upload: upload failure: Post https://c.signalsciences.net/0/push:

Same as the "initial upload" error above, but this upload may also contain signal data and not just startup data. Typically this is a transient error or some other network issue.

Config update failed try : unable to update rules: unable to process request: Get https://sigsci-agent-wafconf.s3.amazonaws.com//.conf:

The agent could not download the given configuration update from Amazon S3. Amazon S3 may be down or there is a network issue on the agent side.

ERROR loading configuration: While parsing config: (, ):

This happens when there is a typo in the agent config file. The error was noticed on the given line and column in the config file. Often the actual typo is before this point, though. Take a look at the config file at the given line and fix any typos.

Hints: * All text should be in quotes * Windows paths generally should be in single quotes (see following error in this doc for details) * Numeric values should generally not have quotes (exceptions are time durations like "100ms") * Lines starting with a hash (#) are comments

ERROR loading configuration: While parsing config: (5, 12): invalid escape sequence: \P

This happens often on Windows when a path is enclosed in double quotes, but does not have the Windows \ characters escaped as \. In this particular case it does not like the \P in c:\Program Files\ and is a product of the agent being cross-platform and the Windows path separator being an escape character on all other OSes. This should be able to be resolved by one of the following:

  • Switch to using single quotes around paths with Windows path separators () instead of double quotes
  • Switch to using universal path separators (change \ to /)
  • Escape the Windows path separators (change \ to \)

[INFO] Could not parse config format: unable to decrypt rules: square/go-jose: error in cryptographic primitive

This means that the accesskeyid and/or secretaccesskey were not correct for the config update that was downloaded by the agent. Probably what this means is that the id/key was rotated for that site and the new config (encrypted with the new creds) has not yet propagated to S3 where the agent downloaded the file. OR, the agent was not yet updated with the new id/key. See also the next error regarding the cached config for another cause.

[WARN] Ignoring cached config from /var/tmp/sigsci-agent_latest.conf: unable to decrypt rules: square/go-jose: error in cryptographic primitive

Similar to the previous error, this just means that the cached file cannot be decrypted with the current keys. Probably the id/key was rotated and the cached file was downloaded with the old id/key. Once the new id/key is in place the old cached file will be updated with the new file. If this persists, then you may need to manually delete the cached file. The agent will not delete the cached file.

Windows Specific Issues

ERROR loading configuration: While parsing config: (1, 1): unexpected token

This typically happens when the agent config was edited with a UTF-8 text editor and saved the file as UTF-8 with a BOM (Byte Order Mark). This is a special first byte (hence the line 1 column 1 location) that sets the UTF-8 byte order used in the file. We do not support this and instead the customer should save as ASCII or UTF-8 without the BOM.

Windows filesystem paths

On windows filesystem paths can use the \ or / path separator (golang allows using /). However, there is one caveat to using \ on Windows in the config file. The config file allows escape characters when using double quotes in values, but does not when using single quotes, so specify file paths with the correct quotes on Windows. All these are essentially the same (note quotes):

tls-cert = "E:/path/with spaces/to/cert.pem"
tls-cert = "E:\\path\\with spaces\\to\\cert.pem"
tls-cert = 'E:/path/with spaces/to/cert.pem'
tls-cert = 'E:\path\with spaces\to\cert.pem'

Windows Tips and Information

The following items might help you debug agent issues easier on Windows.

When you need to stop/start/restart the sigsci agent service, the following Powershell commands will come in handy:

pwsh> Stop-Service sigsci-agent

pwsh> Start-Service sigsci-agent

pwsh> Restart-Service sigsci-agent

In DOS:
C:\> net <stop|start> sigsci-agent

Agent Reverse Proxy

Configuration

Common configuration is as follows:

accesskeyid="*****************************"
secretaccesskey="*****************************"

[revproxy-listener.https]
listener  = "https://0.0.0.0:8446"
upstreams = "https://internal.hostname:8446"
tls-cert  = "/etc/pki/reverse-proxy/hostname.cert.pem"
tls-key   = "/etc/pki/reverse-proxy/hostname.key.pem"

See the full configuration options here: https://docs.fastly.com/signalsciences/install-guides/agent-config/#agentcfg_revproxy-listener

Common Problems and Solutions

revproxy upstream: tls: x509: certificate signed by unknown authority

This means that the Certificate Authority (CA) used to sign the certificate is not known and trusted. Two things to check here:

  1. If it is a self signed cert, then the following will need added to the revproxy-listener block as the validation is not possible: tls-insecure-skip-verify = true NOTE: Use this only as a last resort as it disables all TLS validation

  2. If this is not a self signed certificate, then:

  3. The CA cert(s) may not be in the system trusted CA store for the OS (current list as of this writting)
    • Debian/Ubuntu/Gentoo: /etc/ssl/certs/ca-certificates.crt
    • Fedora/RHEL 6: /etc/pki/tls/certs/ca-bundle.crt
    • OpenSUSE: /etc/ssl/ca-bundle.pem
    • OpenELEC: /etc/pki/tls/cacert.pem
    • CentOS/RHEL 7: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
  4. The CA cert(s) may need to be added to the revproxy-listener configuration if they are non-standard via (sigsci-agent v3.4+): tls-ca-roots = "/path/to/trusted-ca-roots.pem"

revproxy upstream: tls: x509: certificate is valid for , not

This means that the upstream certificate is not valid for the internal hostname in the upstreams URL. This may require the addition of the following configuration directive added to the revproxy-listener block in order to force the hostname used to validate the certificate during the TLS handshake when using Server Name Indication (SNI) (sigsci-agent v3.4+): tls-verify-servername = "valid.cert.hostname"

revproxy upstream: tls: x509: cannot validate certificate for because it doesn't contain any IP SANs

Similar to the hostname validation above and may require: tls-verify-servername = "valid.cert.hostname"

Zero downtime service restarts unavailable: OR Disabling zero downtime restarts for reverse proxy (not supported on this platform)

The platform that the agent is running on does not support zero downtime restarts. At this time it is only supported on BSDs and Linux kernel >= 3.9. Windows is not supported. Without zero downtime restarts, the reverse proxy may go down for a short period (typically < 500ms) while the listener reloads its configuration and restarts. During this restart period any load balancer may get a 502/503/504 status code if it does not support rebalancing.

If a load balancer in front of the reverse proxy agent is returning 504 Gateway Timeout, then it may help to reduce the shutdown-timeout and/or conn-timeout time from the default 30s. This may help the load balancer understand that that node is down sooner and rebalance to another node.

Failed to shutdown reverse proxy () gracefully: context deadline exceeded

The reverse proxy tried to reload its configuration by starting up a new listener (using the new config) to replace the old listener (using the old config). In doing so it will wait up to the shutdown-timeout time (default 30s) for pending connections/transactions to complete before shutting down the old listener. After this point, this log is generated and any existing connections are terminated for the old listener as it is forced to shut down.

It may be that there are long-running connections that do not complete and shutdown-timeout may need to be adjusted accordingly.

Other Reverse Proxy issues, like 503/503 status codes

These typically indicate a problem with the upstream(s). If TLS (https URLs) are used for the upstream, then this typically indicates TLS handshake of validation issues. Sometimes a TLS handshake issue is not logged. The common problems are now logged, but loud errors may be suppressed and require enabling full logging via (sigsci-agent v3.4+): log-all-errors = true

Depending on the error, these should be able to be solved by one of the solutions above, with tls-insecure-skip-verify only being used as a last resort (it is the sledgehammer approach).

Inspection Options

Similar to other modules, the following inspection customizations can be configured with the reverse proxy:

  • inspection-anomaly-duration = time-duration Duration after which the request will be considered an anomaly and the response will be inspected even if nothing else was found in the request during inspection [dynamic] (default "1s")

  • inspection-anomaly-size = int64 Response size limit which the request will be considered an anomaly and the response will be inspected even if nothing else was found in the request during inspection [dynamic] (default 524288)

  • inspection-debug = bool Enable/disable inspection debug logging [dynamic] (default false)

  • inspection-max-content-length = int64 Max request content length that is allowed to be inspected [dynamic] (default 307200)

  • inspection-timeout = time-duration Inspection timeout after which the system will fail open [dynamic] (default "100ms")

Reverse Proxy Side Effects and Limitations

There are a few items to watch out for that may cause issues.

HTTP Header Names are Normalized

The reverse proxy will automatically normalize all header names by capitalizing the first letter in each word (separated by a dash). For example: my-custom-header: some data would become My-Custom-Header: some data

HTTP Header Order May Not Be Maintained to Upstream

The HTTP stack in golang does not allow for tracking and maintaining header order, so the order of headers may change when sent to the upstream server. That said, the order of the same header does seem to be maintained, though these will be grouped together.

For example:

GET /test HTTP/1.1
Host: foo.com
X-Test: test1
X-Blah: some data
X-Test: test2
Accept: */*

Would end up getting sent as something like this (note grouping of X-Test headers):

GET /test HTTP/1.1
Host: foo.com
Accept: */*
X-Blah: some data
X-Test: test1
X-Test: test2

HTTP Proxy Headers are added, but can be configured to not do this

The following proxy headers are added to the upstream request by default:

X-Forwarded-For
X-Forwarded-Host
X-Forwarded-Proto
X-Forwarded-Server

As of agent 3.7, each listener can be configured with minimal-header-rewriting = true, however, and these additional headers will NOT be added/modified (but may be passed through if they exist in the request and trust-proxy-headers = true, the default). In addition, configuring a listener to not trust the proxy headers with trust-proxy-headers = false will allow for stripping these headers before sending to the upstream.

SigSci Headers are Added

X-Sigsci-Agentresponse
X-Sigsci-Tags (only if there were tags added)

HTTP/0.9 is not supported

Golang does not support HTTP prior to HTTP/1.0 and any requests in the 0.9 format will result in a 400 Bad Request error response. This may affect some simple monitoring from poorly written monitors or older load balancers, etc.

HTTP/0.9 is HTTP before it had headers and protocol versioning. These requests are just a method and a URI: GET / All requests like this to Go based web servers will result in a response similar to:

HTTP/1.1 400 Bad Request
Content-Type: text/plain; charset=utf-8
Connection: close

HTTP to Upstream is upgraded to HTTP/1.1

Any HTTP/1.0 requests will be upgraded to HTTP/1.1 to the upstream. This means: * HTTP Keepalives are enabled by default * HTTP/1.1 version is used in the request line * The HTTP Host header is added (required) * The Accept-Encoding: gzip header is added (Go client adds this)

Websockets

Lifespan

At the time of writing there are no limitations within the agent reverse proxy which prevent long lived WebSocket connections.

Standard HTTP connections are subject to timeouts that don't apply to WebSocket connections. There have been customer questions regarding the viability long lived WebSocket connections (24hr+). Testing with Agent 4.21.1 reverse proxy with inspect-websocket = true the agent had no problem maintaining the connection for 3.5+ days (test was manually halted after 3.5 days). This was tested using a basic WebSocket echo server proxying through the agent on AWS EC2, the client sent a timestamp message every 60 seconds to the server which was echoed back to the client. The client was running on a Macbook Pro outside the AWS network. Another test opened a connection using the Javascript API in Google Chrome and left the connection idle, only sending a message manually with a long gap (24hr+) between messages without issue. Though it's possible the browser may have implemented a ping-pong keepalive method as defined in RFC-6455 - more testing required to confirm.

Agent Response Codes

Overview

Agent Response Meaning
-2 An UpdateRequest RPC call was made with a Request ID that is invalid or expired.
-1 An error or timeout occurred in the agent.
0 The module RPC PreRequest did not get a response (timed out?), so there was no known agent response code when the module sent the RPC PostRequest.
200 The module should NOT block this traffic. This does NOT mean that there was nothing found, but rather that conditions were not yet met to block the request.
>=300 Conditions were met to block this request. The agent response code is now configurable (default is 406).

Code: -2

When signals are reported (or when a rule forces analysis of the response) the agent will temporarily store data from the request while waiting for the response to be processed. This data is called the “request context” and is referred to by a Request ID. The Request ID is stored in a special request header named X-Sigsci-Request-Id. The module will use this Request ID when making the UpdateRequest RPC call (which asks the agent to process the response). A few things can happen to cause the -2 code:

  • The module took too long to send the response data and the context stored in the agent expired and was purged causing the Request ID to be invalidated. This timeout is set via the context-expiration agent configuration option.
  • The module “lost” the Request ID. This can happen if the X-Sigsci-Request-Id request header was removed or otherwise not kept intact. See: Dealing with NGINX return directives (agent response -2 errors)
  • The agent restarted in between request and response processing and the context was lost.
  • There are multiple agents and the module sent the response data to the wrong agent (e.g., load balancing agents without sticky sessions enabled).
Troubleshooting Code -2
  • Are there performance issues, causing the module to take too long to send response data?
  • Make sure there are enough resources for the agent.
  • Make sure the timeout is reasonable given traffic and resources.
  • Check rules to see if there are issues there (e.g., pentest rules enabled?)
  • Traffic is too high? Maybe sampling is required?
  • Was the agent restarted around the time the -2 response code was seen in the console?
  • Are there multiple agents being used with the module?
  • Is the NGINX Lua module in use? Is switching to the C Binary (native) NGINX module an option? If not, has the Dealing with NGINX return directives (agent response -2 errors) solution been attempted?
Smuggling -2
  • The agent is seeing the response from the smuggled request
  • https://github.com/signalsciences/sigsci-module-nginx/blob/master/SignalSciences.lua#L465-L484

Code: -1

Typically the agent just timed out or there was some other error while processing the RPC request. This could indicate too much load on the agent or that the agent needs to be tuned or given more resources. The agent metrics and logs should be consulted.

Code: 0

Similar to a -2 code, the module will normally receive the agent response code (WAF Response) in a special X-Sigsci-Waf-Response request header. In this case, the initial PreRequest RPC call timed out from the modules perspective (module timeout), but due to some anomaly in the response (i.e., 4xx, 5xx response, etc.), the PostRequest RPC call was made, but the initial agent response code was not known. In this case it is recorded as a 0 (zero - unknown).

Agent Config Options for Diagnostics

Details on the available configuration options can be found on the Agent Configuration documentation page: https://docs.fastly.com/signalsciences/install-guides/agent-config/

There is an enormous amount of debug data that the agent can log. By habit, many will resort to the sledge hammer that is debug-log-all-the-things, which will pretty much log everything, which is loud (VERY LOUD). Here are a few other options that you might want to use.

debug-log-all-the-things

This is the sledge hammer option that is the same as specifying all of these configuration options, which you might want to use individually for a bit quieter logs:

debug-log-proxy-requests = true
debug-log-blocked-requests = true
debug-log-connection-errors = 1
debug-log-uploads = 2
debug-log-config-updates = 2
debug-log-web-inputs = 2
debug-log-web-outputs = 2

debug-log-web-inputs

This option will log the RPC data sent from the module to the agent in a JSON format. Setting this to 1 will log on a single line vs setting this to 2 to have a more human readable, multi-line version.

Example set to 2 with an XSS attack:

2017/08/18 11:41:45 RPC.PreRequest Request: {
    "RequestID": "",
    "ModuleVersion": "sigsci-agent-revproxy 1.22.0",
    "ServerVersion": "SigSciAgent/1.22.0 revproxy go1.8.3",
    "ServerFlavor": "native",
    "ServerName": "localhost:8800",
    "Timestamp": 1503070905,
    "NowMillis": 1503070905913,
    "RemoteAddr": "127.0.0.1",
    "Method": "GET",
    "Scheme": "http",
    "URI": "/test?q=\u003cscript\u003ealert('xss')\u003c/script\u003e",
    "Protocol": "HTTP/1.1",
    "ResponseCode": -1,
    "ResponseMillis": -1,
    "ResponseSize": -1,
    "HeadersIn": [
        [
            "User-Agent",
            "curl/7.54.0"
        ],
        [
            "Accept",
            "*/*"
        ]
    ],
    "PostBody": ""
}
2017/08/18 11:41:45 RPC.UpdateRequest Request: {
    "RequestID": "59970ab931490c0f4469eb82",
    "ResponseCode": 200,
    "ResponseMillis": 4,
    "ResponseSize": 7,
    "HeadersOut": [
        [
            "Content-Length",
            "7"
        ],
        [
            "Content-Type",
            "text/plain; charset=utf-8"
        ],
        [
            "Date",
            "Fri, 18 Aug 2017 15:41:45 GMT"
        ]
    ]
}

debug-log-web-outputs

Similar to debug-log-web-inputs, this option logs the RPC response data generated by the agent and sent to the module. Again, a 1 means single line and a 2 means human readable.

Example set to 2 from an XSS attack:

2017/08/18 11:41:45 RPC.PreRequest Response: {
    "WAFResponse": 200,
    "RequestID": "59970ab931490c0f4469eb82",
    "RequestHeaders": [
        [
            "X-SigSci-Tags",
            "XSS"
        ]
    ]
}

debug-log-uploads

This option is pretty heavy weight and will log everything sent from the agent to the collector in JSON format. Setting to 1 for single line and 2 for human readable.

Example set to 2 from an XSS attack:

2017/08/18 11:47:21 uploading to https://collector.signalsciences-dev.net:8082/0/push: {
"Meta": {
    "ServerHostname": "test-revproxy",
    "ServerOS": "darwin/darwin//10.12.6",
    "ServerNumCPU": 4,
    "AgentEnabled": 1,
    "ServerArch": "amd64",
    "ServerCPUPercent": 19.1904047976012,
    "HostName": "test-revproxy",
    "HostUptime": 2064671,
    "HostOS": "darwin",
    "HostPlatform": "darwin",
    "HostPlatformVersion": "10.12.6",
    "AgentAddress": "unix:/var/run/sigsci.sock",
    "AgentVersion": "1.22.0",
    "AgentBuildID": "3b85372b8bd6056c26922dc7a020c9795b48995d",
    "AgentArgs": "sigsci-agent -config=/etc/sigsci/agent.conf.revproxy -debug-log-uploads=2 ",
    "AgentTimezone": "EDT",
    "AgentCGroupName": "",
    "AgentTimezoneOffset": -14400,
    "AgentMaxProcs": 2,
    "AgentPID": 69962,
    "HostCPUMhz": 2700,
    "AgentCPUPercent": 0,
    "ModuleType": "sigsci-agent-revproxy",
    "ModuleVersion": "1.22.0",
    "ModuleServer": "SigSciAgent/1.22.0 revproxy go1.8.3",
    "ModuleLastInitDate": "2017-08-18T15:46:25.624156994Z",
    "ModuleUptimeSeconds": 56,
    "AgentTimestampUTC": 1503071241,
    "AgentWAFConfigDate": "2017-08-18T15:46:25.913694274Z",
    "AgentWAFConfigCounter": 1,
    "AgentCurrentRequests": 1,
    "AgentNumGoroutines": 18,
    "ConnectionsTotal": 0,
    "ConnectionsOpen": 0,
    "ConnectionsDropped": 0,
    "ConnReadBytes": 0,
    "ConnWriteBytes": 0,
    "AgentTotalRequests": 1,
    "RequestsUpdate": 1,
    "RequestsPost": 0,
    "AgentUptimeSeconds": 56,
    "AgentMemAlloc": 92725640,
    "AgentNumGC": 4,
    "AgentTotalGCPauseMilli": 26.538097,
    "UploadMetadataFailures": 0,
    "AgentDecisionTimes": {
        "50th": 1.335392,
        "95th": 1.335392,
        "99th": 1.335392,
        "latency50": 0.4541015625,
        "latency95": 0.4541015625,
        "latency99": 0.4541015625
    }
},
"Metrics": {
    "Agent": {
        "GaugeMap": {
            "agent.revproxy.downstream.conn_active": 0,
            "agent.revproxy.downstream.conn_open": 0,
            "agent.revproxy.upstream.conn_active": 0,
            "agent.revproxy.upstream.conn_open": 0,
            "agent.tx.inspecting": 0
        },
        "CounterMap": {
            "agent.revproxy.downstream.conn_new": 1,
            "agent.revproxy.downstream.conn_req_bytes": 114,
            "agent.revproxy.downstream.conn_resp_bytes": 123,
            "agent.revproxy.downstream.conn_total": 1,
            "agent.revproxy.downstream.tx_errors": 0,
            "agent.revproxy.downstream.tx_success": 1,
            "agent.revproxy.downstream.tx_total": 1,
            "agent.revproxy.upstream.conn_new": 1,
            "agent.revproxy.upstream.conn_req_bytes": 382,
            "agent.revproxy.upstream.conn_resp_bytes": 123,
            "agent.revproxy.upstream.conn_total": 1,
            "agent.revproxy.upstream.tx_errors": 0,
            "agent.revproxy.upstream.tx_success": 1,
            "agent.revproxy.upstream.tx_total": 1,
            "agent.tx.bypass": 0
        }
    },
    "Site": {
        "GaugeMap": {},
        "CounterMap": {}
    }
},
"Signals": [
    {
        "RequestIDStr": "59970c0631490c114a0c2f66",
        "ServerHostname": "test-revproxy",
        "Timestamp": 1503071238,
        "ServerName": "localhost:8800",
        "Scheme": "http",
        "URI": "/test",
        "Path": "/test",
        "UserAgent": "curl/7.54.0",
        "RemoteAddr": "127.0.0.1",
        "RemoteHostname": "",
        "RemoteCountryCode": "",
        "Method": "GET",
        "Protocol": "HTTP/1.1",
        "HeadersIn": [
            [
                "User-Agent",
                "curl/7.54.0"
            ],
            [
                "Accept",
                "*/*"
            ]
        ],
        "WAFResponse": 200,
        "ResponseCode": 200,
        "ResponseSize": 7,
        "ResponseMillis": 2,
        "HeadersOut": [
            [
                "Date",
                "Fri, 18 Aug 2017 15:47:18 GMT"
            ],
            [
                "Content-Length",
                "7"
            ],
            [
                "Content-Type",
                "text/plain; charset=utf-8"
            ]
        ],
        "AgentLatencyMillis": 0,
        "AgentDecisionMillis": 0,
        "Summation": {
            "Attrs": null,
            "Attacks": null
        },
        "SignalList": [
            {
                "Type": "XSS",
                "Location": "QUERYSTRING",
                "Value": "q=\u003cscript\u003ealert('xss')\u003c/script\u003e",
                "Detector": "LIBINJECTION",
                "Redaction": 0
            }
        ]
    }
],
"Errors": null
}

debug-profile

This allows you to profile the agent when there are severe performance issues. It will produce a profile output file that engineering can then analyze. It really should only be used after discussions from engineering.

The option should be one of the following (cpu or memory). It will begin profiling when the agent starts and end when the agent stops, so should only be run for a limited timeframe. It does need to run for a few minutes to capture enough data, though (e.g., during a load test, etc):

  • -debug-profile=cpu,/tmp/sigsci-agent-profiling This will profile the CPU and dump data to: /tmp/sigsci-agent-profiling/cpu.pprof

  • -debug-profile=mem,/tmp/sigsci-agent-profiling This will profile the memory usage and dump data to: /tmp/sigsci-agent-profiling/mem.pprof

debug-listener

This option runs a local web server that allows gathering internal data from the running agent.

Run with something like: -debug-listener=127.0.0.1:8881 Then read: https://golang.org/pkg/net/http/pprof/

debug-always-reply

This allow you to test what happens if the agent responds with the given AgentResponse.

Run with something like: -debug-always-reply=406 Obviously you probably do not want to run this in a production setting.

debug-delay

This option allows you to simulate the agent taking too long to respond.

Run with something like: -debug-delay=150ms Obviously you probably do not want to run this in a production setting.

Tracing Issues Example

If you suspect or want to verify RPC timing data or RPC request/response data, then you should turn on debug-log-web-inputs=1 and debug-log-web-outputs=1

This will log data for the RPC requests and responses one-per-line:

2017/09/06 09:25:09.309505 RPC.PreRequest Request: {"RequestID":"","ModuleVersion":"sigsci-sdk-golang 1.0","ServerVersion":"go1.8.3","ServerFlavor":"","ServerName":"127.0.0.1:8085","Timestamp":1504704309,"NowMillis":1504704309306,"RemoteAddr":"127.0.0.1","Method":"GET","Scheme":"http","URI":"/200_all_methods?q=\u003cscript\u003ealert('xss')\u003c/script\u003e","Protocol":"HTTP/1.1","ResponseCode":-1,"ResponseMillis":-1,"ResponseSize":-1,"HeadersIn":[["User-Agent","curl/7.54.0"],["Accept","*/*"]],"PostBody":""}
2017/09/06 09:25:09.310380 RPC.PreRequest Response: {"WAFResponse":200,"RequestID":"59aff73531490ce8af1dfe9b","RequestHeaders":[["X-SigSci-Tags","XSS"]]}
2017/09/06 09:25:09.313927 RPC.UpdateRequest Request: {"RequestID":"59aff73531490ce8af1dfe9b","ResponseCode":200,"ResponseMillis":5,"ResponseSize":3,"HeadersOut":[["X-Request-Id","59aff73531490c5ee1de46c2"],["X-Sigsci-Requestid","59aff73531490ce8af1dfe9b"],["X-Sigsci-Agentresponse","200"],["X-Sigsci-Tags","XSS"]]}

As of agent 1.23.0 the time format is in microsecond resolution, so you should be able to better see timing data.

Getting the Status of a Running Agent

To get details on the current running state of the agent, you can send a SIGUSR1 signal (on *NIX, not Windows) and the agent will log additional status data. To do this, just determine the agent process ID (PID) and issue kill -USR1 PID to the PID. This will trigger the agent to run diagnostics and log the results.

From the log, you can see the PID (or via ps command, etc):

Signal Sciences Agent 4.22.1 running as user root with PID 14201 Issue a SIGUSR1 signal to the PID:

kill -USR1 14201 You will see something like this in the agent log:

2021/10/07 16:23:53.528440 Received diagnostics signal. Running diagnostics
2021/10/07 16:23:53.528514 Service downloader Up 23.53657391s
2021/10/07 16:23:53.528521 Service cleaner Up 23.536578047s
2021/10/07 16:23:53.528524 Service uploader Up 23.536664711s
2021/10/07 16:23:53.528526 Service metrics Up 23.536622916s
2021/10/07 16:23:53.528528 Service rpc Up 23.990073188s
2021/10/07 16:23:53.528545 Running agent diagnostics: pid=14201
2021/10/07 16:23:53.528549 Signal Sciences Agent 4.22.1 running as user root with PID 14201
2021/10/07 16:23:53.529806 Configured Options: {"accesskeyid":"fa7ada4b-7838-4f46-bdfa-5f819487bf95","config":"/Users/alexm/agent-module.conf","max-backlog":800,"max-connections":800,"max-records":100000,"rpc-workers":4,"server-hostname":"C02DD3R3ML85"}
2021/10/07 16:23:53.529828 External data item name=datacenters type=ipv4ranges version=2019.9.19+001731
2021/10/07 16:23:53.529830 External data item name=geolite2-country type=mmdb-data version=2021.9.15-12.03.14
2021/10/07 16:23:53.529833 Tracked Contexts: 0
2021/10/07 16:23:53.529868 Memory Available: 6.234G / 16.000G RAM available
2021/10/07 16:23:53.529950 Memory Allocated: heap=55.758M stack=0.969M
2021/10/07 16:23:53.529998 Profiling disabled.

Note the Service uptimes, running version, current config, and the Tracked Contexts count.