Skip to content

Signal Sciences Java Module Troubleshooting Guide

General Tips

  • Ensure the customer has the latest version of the module installed
  • Ensure the customer is running on JDK 1.8+
  • Ensure the SigSciFilter is of highest priority in the filter chain
  • Turn on debug and record server logs

JDK version issues

The JDK version should be over 1.8. A Java version mismatch will cause a runtime error like the following:

Unsupported major.minor version 52.0

UNIX domain socket for RPC address is not supported on Mac when the shaded jar is used because the shaded jar doesn't carry native libraries for the Mac. Please request the customer use TCP for the RPC address in this instance.

Caused by: java.lang.UnsatisfiedLinkError: could not get native definition for type `POINTER`, original error message follows: java.lang.UnsatisfiedLinkError: could not locate stub library in jar file. Tried [jni/Darwin/libjffisigsci-1.2.dylib, /jni/Darwin/libjffisigsci-1.2.dylib]

Spring boot

The Java module should support all version of spring boot through 2.x.x

Web Server issues

Only certain web servers are supported, at a minimum the web server must support JDK 1.8. The code base contains examples integrating with Tomcat, Netty, Jetty and Spring boot. It has also been tested with certain versions of Weblogic. If the web server isn’t compatible it won’t start when you try to integrate with the filter.

Basic Installation issues

Check the logs to see if the filter is actually installed correctly. When traffic is passed through the filter there should be some additional messages in the logs to show that requests are being processed through the SigSciFilter. If it’s being processed but the response code seems incorrect from the agent, then it’s more likely an agent issue than a module issue.

Ensure the SigSciFilter is of highest priority in the filter chain

Because we change properties in the HTTP request/response it’s important that the SigSciFilter is able to process the request and response prior to being intercepted by another filter. The server will start normally but the agent may be constantly returning an incorrect error code.

Programmatically with Spring you would need to use the command:

FilterRegistrationBean<SigSciFilter> registration = new FilterRegistrationBean<>();
registration.setOrder(Ordered.HIGHEST_PRECEDENCE);

In web.xml you would ensure that the filter mapping comes first.

Agent Connection issues

If the module is unable to connect with the agent you should receive the following error: sigsci module and agent communication, failing open. java.net.ConnectException: Connection refused In this case we should check the rpcServerURI setting which is specified either in code or in the web.xml. The rpcServerURI setting by default is unix:/var/run/sigsci/sigsci.sock, if there are issues with connecting with the agent we should check that the agent is listening on that socket. Otherwise, you are able to connect the module with the agent via TCP, in this case the rpcServerURI should look like tcp://agent:9090 and we should check that the agent is indeed running on that TCP port.

Classpath issues

This is generally manifested as a java.lang.NoClassDefFoundError, java.lang.NoSuchMethodError, java.lang.IncompatibleClassChangeError or java.lang.LinkageError.

The first thing to try in any of these cases is ensuring the customer has installed the latest version of the Java module. Otherwise this error can occur at runtime in the following cases. Firstly, if we do just a partial recompilation of our code. Secondly, if there is version incompatibility with the dependencies in our application, such as the external jars. Generally it’s the latter that’s the cause of the issue if we see this. In this case, we'll first check the order of the jars in the build path pulled by the classpath loader. And we'll trace and update the inconsistent jar. We should also make sure that there are no duplicate classes in two or more jars. Once we have identified the offending dependency we should exclude one from the build.

Servlet specification issues

This is a subset of the classpath issue, usually when the servlet jar being used is missing or incompatible with SigSciFilter code. It looks like the following exception java.lang.NoClassDefFoundError: javax/servlet/http/HttpServlet. The first thing to check is that the customer has the most recent version of the java module. After checking that the solution would be to include the appropriate servlet jar in the classpath.

Debugging

It is a good idea to turn on debugging if there are issues with the SigSciFilter. We usually will ask for a log with debug turned on if it’s not immediately clear what the issue is or if there’s a brief error message in the logs but it’s not detailed. Turning on debugging will allow you to see full stack traces instead of a short error description. Using log4j or slf4j, you would need to edit your log4j.properties (or log4j.xml) to include the following:

# turn on debug for all stacktraces
log4j.rootLogger=debug
# turn on debug for the SigSciFilter class where most of the problems would
# most likely be
log4j.logger.com.signalsciences.servlet.filter=debug

Escalating to engineering

Please include any server logs with stacktraces (with debug logging enabled), your web.xml or Spring configuration file and your pom.xml if you are using maven.

Thread Pool Sizing and Downstream Latency Issues

Under high traffic or backend slowness, worker threads in the Java module may accumulate and cause thread pool or server-level resource exhaustion.

Downstream Latency and Thread Accumulation

Thread accumulation is primarily caused by downstream application latency rather than WAF decision delays. The SigSciFilter executes using a try / finally block:

RPCMsgOut rpcResponse = filterService.getPreRequest(httpRequest, msgIn);

try {
    // WAF passes control downstream to your application code
    chain.doFilter(httpRequest, response);
} finally {
    // Executed ONLY after application code completely finishes
    filterService.postRequest(httpRequest, httpResponse);
}

Because postRequest sits inside the finally block, the WAF holds the outer thread stack open until the underlying application code finishes processing. If downstream dependencies stall (such as database or remote socket connect timeouts), worker threads pile up on chain.doFilter until the web server's overall thread pool is exhausted.

Controlling Thread Pooling (rpcPoolEnabled)

The rpcPoolEnabled configuration toggles between the legacy execution client and an optimized pooled client: - rpcPoolEnabled = false (Legacy Client): Thread pool capacity scales up to $10\times$ available CPU cores (e.g., up to 960 threads on a 96-core system). Under downstream latency, this high ceiling causes massive thread accumulation and high Garbage Collection (GC) overhead. - rpcPoolEnabled = true (Optimized Client): Drops the ceiling to $1\times$ available CPU cores (e.g., 96 threads on a 96-core system) and reuses thread-local buffers to significantly reduce GC pressure.

Thread Pool Lifecycle and Bounding

Worker threads process three core RPC tasks: preRequest, updateRequest, and postRequest. Threads are returned to the pool immediately upon completing any of these tasks, and the pool is destroyed when the web server shuts down.

To fix the maximum capacity of worker threads, configure rpcThreadCount explicitly (e.g., rpcThreadCount = 64).

Fail-Open Behavior and Timeouts

The filter is hardcoded to fail open to prioritize application availability over security enforcement. This behavior is governed by rpcTimeout (default: 300ms), which dictates how long the module waits for an agent response. If the agent does not respond within this window, a warning is logged and the request passes through.

Note: If threads appear blocked for longer than one minute, check downstream application connection timeouts rather than rpcTimeout.

To prevent thread exhaustion and lower system overhead, configure the following settings (exposed via environment variables or servlet configuration depending on your deployment setup):

# Enables the optimized client (1x core count ceiling)
rpcPoolEnabled = true

# Caps maximum worker thread count (tune based on server capacity)
rpcThreadCount = 64

# Reduces agent response timeout for faster fail-open (in ms)
rpcTimeout = 100