Skip to content

Troubleshooting

The Customer Tech team has excellent experience troubleshooting nginx lua, see the following in the Customer Tech Team Knowledge Base

Table of Contents * Gather Basic Details * TCP Socket Timeout Errors * Nginx crash when SElinux enabled

Gather Basic Details

The version of nginx or openresty is very important.

$ nginx -V

Log file typically found at: * /var/log/nginx/error.log * /usr/local/openresty/nginx/logs/error.log

Gather important config information related to the issue. Often there is a location involved with the problem.

Somewhat complex example of locations with an auth directive:

    # Authentication Based on Subrequest Result
    location /private/ {
        auth_request     /auth;
        auth_request_set $auth_status $upstream_status;
        proxy_pass      http://127.0.0.1:80/private/;
    }

    # proxy_pass_request_body turns off passing the post body
    # request body is discarded for authentication subrequests
    location = /auth {
        internal;
        proxy_pass      http://127.0.0.1:80/inspect/;
        proxy_pass_request_body off;
        proxy_set_header        Content-Length "";
        proxy_set_header        X-Original-URI $request_uri;
    }

TCP Socket Timeout Errors

Unusually high volume of "lua tcp socket timeout" errors in nginx logs.

Causes

If there aren't enough CPU resources available, customer may see a large volume of "lua tcp socket timeout" errors. External factors can create spikes in traffic, which can also result in these errors. TAMs/Engineering can validate whether CPU resources are limited by checking the agent decision time graph and what rules are enabled. This may result when the nginx server gets too busy to to respond to new I/O events due to CPU spike.

Another possible issue concerns Advanced Rules. This may be indicated by viewing the dashboard for an agent. That may reveal decision times that are too slow.

Possibly the application receives lots of very large POST requests. This may affect both decision time and in particular latency. Latency may be viewed in the dashboard for an agent as well.

Solution

The best solution at this point is to recommend allocating additional CPU cores to the server or increasing the socket timeout value (assuming customer is ok with the additional latency).

It is highly recommended by openresty to set the lua sockets log config to off in the nginx.conf: Ex:

lua_socket_log_errors off;

An option that may help alleviate the issue is to increase the timeout used by our module. The config variable sigsci.timeout has a default of 100, and may be set in the sigsci.conf Ex:

sigsci.timeout = 200

If the issue is a poorly written Advanced Rule, then the rule would need to be rewritten or even disabled.

If there are alot of large POST requests then one can tune the module parameter sigsci.maxpost which defaults to 100000. Lower this value to an acceptable range.

Nginx crash when SElinux enabled

Nginx crashes with message "nginx: lua atpanic: Lua VM crashed, reason: runtime code generation failed, restricted kernel?" in the nginx eror log.

Causes

SELinux is enabled by default on modern RHEL and CentOS servers. With SElinux, there are policies to prevent perceived dangerous behavior by processes. JIT is flagged as a perceived dangerous behavior.

Solution

Customers running into this issue should do one of the following:

  • Set SELinux to permissive mode - https://docs.signalsciences.net/troubleshooting/selinux/#set-se-linux-to-permissive-mode-or-disable-se-linux-completely
  • Disable SELinux for nginx - https://www.nginx.com/blog/using-nginx-plus-with-selinux/
  • Disable the jit by adding a line in the nginx.conf next to where the load_module lines are added:
env SIGSCI_NGINX_DISABLE_JIT=true;