Skip to content

SigSci Nginx Lua module troubleshooting guide

Introduction

This troubleshooting guide is helpful to support engineers for diagnosing Nginx Lua module issues reported by the customers.

Nginx versions supported

  • 1.10.0 and above

Review Nginx error logs

Agent online but module isn't registered or undetected

  • Verify the module configuration
  • Verify that agent is running
    systemctl status sigsci-agent

Request data uploaded to the cloud have agentresponse -2

This occurs when the agent sees a response without seeing the original request. Agent not being able to pair up the SigSci PreRequest and PostRequest of a given request. This means that phases that normally run are skipped, such as the rewrite or access phase. This also means that later phases that are run regardless, e.g. log_by_lua, will not have access to information that is normally set in those phases.

  • Nginx is handling/erroring the request before it gets to the Lua module.

    Requests are terminated early within customer's nginx.conf, i.e. with 4xx codes. This can cause NGINX to respond to the request before the Lua 'access_by_lua' phase runs, which contains 'sigsci.prerequest()' statement.

    Nginx may terminate a request early with:

    code message
    400 Bad Request
    405 Not Allowed
    408 Request Timeout
    413 Request Entity Too Large
    414 Request URI Too Large
    494 Request Headers Too Large
    499 Client Closed Request
    500 Internal Server Error
    501 Not Implemented

    Reference

  • Nginx configuration contained {access,rewrite}_by_lua in server or location block which override the SigSci {access,rewrite}_by_lua in http block and didn't call the method 'sigsci.prerequest()'.

    Add method 'sigsci.prerequest()' as first statement to 'access_by_lua' of customer's existing {access,rewrite}_by_lua script to resolve the issue.

  • Nginx configuration contained a return or deny all statment in location block which returned response before calling SigSci PreRequest in {access,rewrite}_by_lua

Example with return statement

server {
  listen 80;
  server_name ironmq.dactbc.com;
  location / {
    # Calling the rewrite_by_lua_block directly allows us
    # to force our module to run first and then do the
    # return statement for nginx.
    rewrite_by_lua_block {
      sigsci.prerequest()
      return ngx.exit(200)
    }
    # The return statements will stop all processing so
    # normally with our access_by_lua 'sigsci.prerequest()',
    # which is in the include /opt/sigsci/nginx/sigsci.conf,
    # these requests are missed by us.
    #return 200;
  }

  location /redirect {
    # Same thing here but this includes a redirect to another
    # page and/or server. All of the functionality of the return
    # statement is available in the lua block
    rewrite_by_lua_block {
      sigsci.prerequest()
      return ngx.redirect("https://cloud.dactbc.com")
    }
    #return 302 https://cloud.dactbc.com;
  }

  ## Adding Additional Data to a Return statement
  location /ping {
    default_type application/json;
    rewrite_by_lua_block {
      sigsci.prerequest()
      ngx.say('{"ping":"successful"}')
      return ngx.exit(200)
    }
  }
}

Additional examples

# Return a 503 error if the maintenance page exists.
if (-f /var/www/ring-site/shared/public/system/maintenance.html) {
  # Start of SigSci lua script
  rewrite_by_lua_block {
    sigsci.prerequest()
    return ngx.exit(503)
  }
  # End of SigSci lua script
  # return 503; # old return statement
}

#returning a 301 with a few variables
location ~ ^/oauth/authorize$ {
  # Start of SigSci lua script
  rewrite_by_lua_block {
    sigsci.prerequest()
    # this is calling nginx variables into lua variables
    return ngx.redirect(ngx.var.oauth_upstream .. ngx.var.request_uri, 301);
  }
  # End of SigSci lua script
  # return 301 $oauth_upstream$request_uri; # old return statement
}

# Listen to HTTP and redirect to HTTPS for additional server names
# Returning a 301 with a string and a few variables
server {
  listen 81;
  server_name site-stg.dev.ring.com site-stg.ring.com;
  # Start of SigSci lua script
  rewrite_by_lua_block {
    sigsci.prerequest()
    return ngx.redirect("https://" .. ngx.var.host .. ngx.var.request_uri, 301)
  }
  # End of SigSci lua script
  # return 301 https://$host$request_uri;
}

In another case, the deny all directive could be used, which responds with a 403.

# Block access to "hidden" files and directories whose names begin with a
# period. This includes directories used by version control systems such
# as Subversion or Git to store control files.
location ~ /\.(.*)/ {
  # Calling the rewrite_by_lua_block directly allows us to force our
  # module to run first and then do the return statement for nginx.
  # In this example, instead of calling "deny all" after the
  # sigsci.prerequest(), we call "return ngx.exit(403)". It is possible
  # that "deny all" could be used again, but not tested as of posting this example.
  rewrite_by_lua_block {
    sigsci.prerequest()
    return ngx.exit(403)
  }
  #deny all;
}

After installing 'nginx110/111-lua-module' from the SigSci repo and trying to load into the nginx conf, I get the error "Module is binary incompatible" when trying to start nginx

The dynamic Lua modules we provide for nginx versions > 1.10 example are built for specific distributions of nginx, generally the distros provided directly by NGINX.org. Even when using dynamic modules, nginx tends to be very finicky about the compile options needing to match between the dynamic module and the nginx build itself, and when they are built with different compile flags this error can occur. Where applicable, we've also provided Lua packages for Amazon linux and EPEL distros of nginx.

If you run into this error you will likely need to explore either having the customer download a different distro of nginx (either NGINX.org to support our dynamic Lua modules, or a distro such as OpenResty that already has Lua pre-baked) or recompiling nginx themselves to include Lua support.

Reference

Request data uploaded to the cloud have agentresponse 406 but server response isn't 406

This occurs when the Nginx configuration mapped error code to non-existing custom error page.

# example
server {
  error_page 406 /406.html

  location / {
    redirect /index.htm permanent;
  }

  #location /406.html {
  # root /usr/share/nginx/html;
  # break;
  #}
}

Ability to exclude certain paths from being seen/processed by module

Reference

Update the existing access_by_lua and log_by_lua directives in /opt/sigsci/nginx/sigsci.conf file It is safe to edit the sigsci.conf file as it is outside the module code.

Exclude module based on path using regexp

access_by_lua 'if not ngx.re.match(ngx.var.uri, [[^/mailbox.*]]) then
  sigsci.prerequest()
end';
log_by_lua 'if not ngx.re.match(ngx.var.uri, [[^/mailbox.*]]) then
  sigsci.postrequest()
end';

Exclude module based on IP

access_by_lua 'if not (( ngx.var.remote_addr == "132.5.72.3") or (ngx.var.remote_addr == "132.5.72.3")) then
  sigsci.prerequest()
end';
log_by_lua 'if not (( ngx.var.remote_addr == "132.5.72.3") or (ngx.var.remote_addr == "132.5.72.3")) then
  sigsci.postrequest()
end';

Module latency time can be way off

The nginx module's latency graphs can show very high, but the queue time is very low. This makes it look like there is a lot of module/network overhead, but it is probably a Lua feature (bug) with a cached ngx.now() value.

The latency time is measured from NowMillis until inspection is about to start. The queue time is a sub-set of the latency time - how long the call spent in the queue before a worker handled it. A high latency, low queue means there was a lot of overhead in the RPC call (module or network). The nginx module uses ngx.now() to calculate NowMillis, but it seems that this value is cached.

Reference

Use source IP in proxy protocol as RemoteAddress

AWS load balancer and Haproxy support PROXY protocol and can send the client remote address before HTTP request. Nginx variable ngx.var.remote_addr contains downstream proxy IP address and not the client remote address when the Nginx configued to use PROXY protocol. SigSci module sends the proxy IP address and not the real client remote address.

This issue can be resolved by replacing one line in SigSci NGINX Lua module file SignalSciences.lua

# replace 
RemoteAddr = ngx.var.remote_addr,
# with
RemoteAddr = (ngx.var.proxy_protocol_addr ~= "" and ngx.var.proxy_protocol_addr or ngx.var.remote_addr)

This issue can also be resolved by adding these statements to nginx configuration

real_ip_header proxy_protocol;
# Replace subnet 172.31.19.0/24 or IP with the correct AWS Load balancer
# NGINX facing IP or subnet.
set_real_ip_from 172.31.19.0/24;

Make sure NGINX built with --with-http_realip_module, otherwise the above configuration may not work.

Following command should print 1

nginx -V 2>&1|grep -c realip_module

Reference

Reference

Conflicting with 'satisfy any' directive

HTTP basic auth along with IP restriction, the directive satisfy any won't behave as expected if sigsci nginx module is loaded. All traffic allowed when an auth dialogue should be shown for those who didn't satisfy IP rules. SigSci module bypassing the auth.

The directive satisfy all grants access if a client satisfies all conditions. The directive satisfy any grants access if a client satisfies at least one condition.

Lua hook access_by_lua satisfies any condition, so basic auth by-passed. This is why Lua hook rewrite_by_lua recommended to resolve the issue.

Example:

server {
  satisfy any;
  allow 1.1.1.1/32;
  deny all;
  auth_basic "login required";
  auth_basic_user_file conf/htpasswd;
}

Reference

Escalating issues to engineering

Add the output of the following commands to the JIRA ticket

nginx -v
nginx -T > nginx.config
uname -a
cat /etc/os-release