Skip to content

Module Behaviour Testing

Herein lies the module behaviour testing suite. The goal is to have a set of tests that can be fired at a module implementation to validate that it conforms to our "module specification [1]".

The tests focus on the RPC behaviour between module and agent. The tests utilize an RPC test harness built into the production sigsci agent and is enabled with the flag -rpc-test-harness. This exercises the production RPC code paths however bypasses collector upload and decisioning.

Quickstart

Tests run inside a docker container and rely on mounting your local checkout of the various code repos into the container for testing. Using the nginx module as an example:

  1. Checkout the nginx module source somewhere on your machine, in this example $HOME/src/sigsci/sigsci-module/nginx

  2. Export the following environment variable:

    export SIGSCI_MODULE_NGINX_SRC=$HOME/src/sigsci/sigsci-module-nginx

  3. Run the tests:

    make nginx

The first run will be slow as we need to build the initial testing container. Subsequent runs will use the existing cached container. (See note at the bottom of this section)

To run the other tests you'll need to checkout the code, and set the following env vars. The second column is the corresponding make target to run for tests.

env var make target
SIGSCI_MODULE_NGINX_SRC=<path_to_checked_out_nginx_module_source> make nginx
SIGSCI_MODULE_APACHE_SRC=<path_to_checked_out_apache_module_source> make apache
SIGSCI_SDK_PHP_SRC=<path_to_checked_out_php_source> make php, make mod_php, make php_fpm
uses local punching bag source make golang

If you are working on module code and or tests regularly, it may help to add all the relevant env vars to your shell startup scripts.

NOTE: if you see 404 package errors running any tests you may need to force a rebuild of the test container by running make rebuild

Testing with a local Agent

If you want to use a local agent instead of the released agent, then just set BUILD_LOCAL_AGENT=1 and the local agent will be built to run the tests.

make BUILD_LOCAL_AGENT=1 nginx

Hacking on Tests

Most modules have a test script test.sh that runs the module-testing tests against the module. To run the local version of the module-tests 1. Comment out the line in test.sh that pulls the module-testing image from AWS. It will be something like $DOCKERCOMPOSE pull --ignore-pull-failures 2. Build the module-testing image. Unless you're on Windows, this will be make docker-build. 3. Run the suite in the module via test.sh.

If you want to run a single test you can use the RUN environment variable. For example to run JUST the TestServerResponseWithoutSignal test against nginx you can run the test suite as follows:

RUN=TestServerResponseWithoutSignal make nginx

NOTE: The RUN env var is used by the golang -test.run parameter so can be a regular expression if you have a range of tests you might want to run.

Manual Testing

It is possible to just skip the default tests to run in manual test mode. With this there are two possibilities.

Test Command Mode

This mode allow you to run an arbitrary command in the container instead of the normal go test. Just set RUN_TESTS_CMD to some shell command.

For example, this would get you an interactive shell in the container to do what you want:

make RUN_TESTS_CMD=bash nginx

Daemon Mode

In this mode, you will have a container running in daemon mode with port 8885 mapped so that you can hit the container from a local browser or other tool.

Just specify the make target as [target]-daemon. For example, to run the nginx target container environment in daemon mode with a locally built agent:

make BUILD_LOCAL_AGENT=1 nginx-daemon

Then you can hit http://$(docker-machine ip sigsci-dev):8885/ with any tools you want.

NOTE: This just daemonizes the container with a RUN_TESTS_CMD=wait.

Use daemon-stop and daemon-console commands to stop the daemon mode and get a shell in the running daemon container for diagnostics, etc.

You can start a target in daemon mode, get a shell into it, and then run tests manually:

$ make nginx-daemon
  ...
Running in nginx daemon mode: http://192.168.99.100:8885/

$ curl http://192.168.99.100:8885/
<pre>
<a href="404.html">404.html</a>
<a href="html_content.html">html_content.html</a>
<a href="text_content.txt">text_content.txt</a>
</pre>

$ make daemon-console

root@a5f44cb3a185:/go/src/github.com/signalsciences/module-testing# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 22:01 ?        00:00:00 /bin/bash ./module_test.sh
nginx
root        36     1  0 22:01 ?        00:00:00 nginx: master process
/usr/sbin/nginx
www-data    37    36  0 22:01 ?        00:00:00 nginx: worker process
www-data    38    36  0 22:01 ?        00:00:00 nginx: worker process
www-data    39    36  0 22:01 ?        00:00:00 nginx: worker process
www-data    40    36  0 22:01 ?        00:00:00 nginx: worker process
root        81     1  0 22:01 ?        00:00:00 httpsim -docroot=/go/src/github.com/signalsciences/module-testing/docroot
root        90     1  0 22:01 ?        00:00:00 sigsci-agent -debug-rpc-test-harness
root       118     0  1 22:03 ?        00:00:00 bash
root       132   118  0 22:03 ?        00:00:00 ps -ef root@a5f44cb3a185:/go/src/github.com/signalsciences/module-testing# godep go test

root@a5f44cb3a185:/go/src/github.com/signalsciences/module-testing# godep go test -test.v
=== RUN   TestAgentResponseCodesFileExists
  ...

Component Flow


  +====== WEBSERVER ======+
  | +-------------------+ |                            +--------------------------+
  | | module under test | <--- unix domain socket ---> | agent (rpc test harness) |
  | +--+---------+------+ |                            +--------------------------+
  +====|=========|========+
       |         |
 proxy |         | filesystem
       |         |
+------+--+   +--+--------------+
| httpsim |   | regular docroot |
+---------+   +-----------------+

The module-testing environment consists of a number of components:

Webserver

This is the webserver hosting an implementation of the module. For example, Nginx with the Lua module or Apache with the C module. Other implementations may use different webservers (netty, jetty, golang, IIS, you name it ...)

The webserver must be configured with two base internal URI endpoints for basic testing:

  • /dynamic - proxies through to httpsim which also serves static content from docroot/
  • /static - serves files from the docroot/ directory

Requesting missing files calls the webserver default 404 handler. To test custom 404 handling use these URI's:

  • /dynamic/custom_404
  • /static/custom_404

Those also serve the same docroot/ directory.

Hitting /200_all_methods does its best to return a 200 for any HTTP/1.1 method you throw at it.

HTTPSim

Simulates an HTTP application server. For these tests we give it a document root allowing it to serve the same content as the static configurations. It also provides the following additional endpoints:

  • /upload - will accept file uploads of any size, these are simply dumped to /dev/null when served it'll be at /dynamic/upload
  • /form - will accept PUT and POST requests and will echo the request body back in the response body

Agent (RPC Test Harness Mode)

Is a testing mode within the production agent that allows testing of the RPC code paths and bypasses decisioning, rule download, and collector upload. The behaviour of the test harness can be controlled with a set of HTTP request headers that can be added to requests to make the agent do interesting things.

It also pushes copies of incoming agent messages onto a stack that you can fetch from an HTTP server that is listening on port 12345. This allows you to retrieve a copy of the messages sent into fake agent. The two endpoints are: * /fetch - this will return any agent messages that haven't been previously popped off. The response is a JSON encoded array of scriptengin.AgentMsgIn structs * /reset - will zero the stack - helpful to call before a test

Supported Testable Implementations

  • nginx
  • apache
  • golang
  • mod_php (under apache)
  • php_fpm (under nginx)

Adding New Platforms

TODO: WRITE THIS

Todo / Random Thoughts

  • test/verify module rpc deserializes to the same data in the agent is anything missing, similarly the other way?
  • should we also have the collector path but instead of uploading it posts them to a test consumer or even back into the test itself somehow to be able to detect duplicate requests being posted (ie. custom 404 handlers)