Skip to content

Agent Module Performance Testing Lab

This is a modularized virtual lab designed for performance testing.

Usage

First you must have the docker images:

make docker-build

Next, make sure you have a native perftest binary:

make native-perftest
perftest -h

Next, run make help to see what is here.

Performance Testing

Run a containerized comprehensive performance test to output CSV results for request time, memory and CPU usage

make lab-stop lab-stack-perftest-minimal-start lab-performance-test-start

The rps and request sizes simulated are configured in the script containers/scripts/performance_driver.py

This takes about 30 minutes and will output results to containers/scripts/performance_output.csv

Watch http://localhost:9090/graph?g0.range_input=30m&g0.expr=sigsci_agent_runtime_host_cpu_pct&g0.tab=0&g1.range_input=30m&g1.expr=sigsci_agent_runtime_mem_alloc&g1.tab=0 to see the progress.

Other Examples

Run a simple test with perftest

  • Setup the essential parts of the lab
make lab-metrics-start
make lab-nginx-native-start
make lab-sigsci-agent-rpc-uds-start
  • Run a curl test to check the fortio-server is proxied
$ curl -D- 'http://localhost:8085/debug?delay=250us:30,5ms:5&status=503:0.5,429:1.5'
HTTP/1.1 200 OK
Server: nginx/1.15.9
Date: Wed, 03 Jun 2020 21:21:21 GMT
Content-Type: text/plain; charset=UTF-8
Content-Length: 419
Connection: keep-alive

Φορτίο version 1.3.2-pre 2020-03-11 22:37 5e67392604f29d76417617113a9aa8abf8a3a6ca go1.11.5 echo debug server up for 12m11.7s on 0e028713fec2 - request from 172.28.0.7:38036

GET /debug?delay=250us:30,5ms:5&status=503:0.5,429:1.5 HTTP/1.1

headers:

Host: fortio-server:8080
Accept: */*
Connection: close
User-Agent: curl/7.64.1
X-Sigsci-Agentresponse: 200
X-Sigsci-Mac: fb2f65e9402cbcd4202a3b91a1c5b3e4

body:
  • Run a perftest against nginx -> fortio-server
$ perftest -c 2 -method POST -ct json -size small -n 10 -csvout /dev/stdout 'http://localhost:8085/debug?delay=250us:30,5ms:5&status=503:0.5,429:1.5'
Test Summary:
  Target URL:      http://localhost:8085/debug?delay=250us:30,5ms:5&status=503:0.5,429:1.5
  HTTP Method:     POST
  Content Type:    application/json
  Content Length:  561
  Concurrency:     2
  Requests:        10

Raw HTTP Request:
------------------------------------------------------------
POST /debug?delay=250us:30,5ms:5&status=503:0.5,429:1.5 HTTP/1.1
Host: localhost:8085
User-Agent: sigsci-perftest/0.0.2
Content-Length: 561
Content-Type: application/json
Sigsci-Header-01: Sigsci-Val-1
Sigsci-Header-02: Sigsci-Val-2
Sigsci-Header-03: Sigsci-Val-3
Sigsci-Header-04: Sigsci-Val-4
Sigsci-Header-05: Sigsci-Val-5
Sigsci-Header-06: Sigsci-Val-6
Sigsci-Header-07: Sigsci-Val-7
Sigsci-Header-08: Sigsci-Val-8
Sigsci-Header-09: Sigsci-Val-9
Sigsci-Header-10: Sigsci-Val-10
Accept-Encoding: gzip

{"_id":"56627c3e95fd40bfe5710335","address":"228 Ovington Avenue, Sexton, Tennessee, 1562","age":36,"balance":"$1,358.84","company":"INSECTUS","email":"rosiemccullough@insectus.com","eyeColor":"green","favoriteFruit":"apple","gender":"female","greeting":"Hello, Rosie Mccullough! You have 10 unread messages.","guid":"120b94d0-a3be-4acf-8540-7821416bd9b1","index":0,"isActive":false,"latitude":47.151313,"longitude":-16.348278,"name":"Rosie Mccullough","phone":"+1 (957) 502-3458","picture":"http://placehold.it/32x32","registered":"2014-10-18T10:35:09 +07:00"}
------------------------------------------------------------

Waiting for target "localhost:8085" to become ready...
Ready after 1.488765ms
Starting load test...

Summary:
  Total:    0.0210 secs
  Slowest:  0.0074 secs
  Fastest:  0.0029 secs
  Average:  0.0042 secs
  Requests/sec: 475.2758

  Total data:   13630 bytes
  Size/request: 1363 bytes

Response time histogram:
  0.003 [1] |■■■■■■■■■■
  0.003 [4] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.004 [2] |■■■■■■■■■■■■■■■■■■■■
  0.004 [1] |■■■■■■■■■■
  0.005 [0] |
  0.005 [0] |
  0.006 [0] |
  0.006 [0] |
  0.007 [0] |
  0.007 [0] |
  0.007 [2] |■■■■■■■■■■■■■■■■■■■■


Latency distribution:
  10% in 0.0029 secs
  25% in 0.0033 secs
  50% in 0.0034 secs
  75% in 0.0073 secs
  90% in 0.0074 secs
  0% in 0.0000 secs
  0% in 0.0000 secs

Details (average, fastest, slowest):
  DNS+dialup:   0.0003 secs, 0.0029 secs, 0.0074 secs
  DNS-lookup:   0.0002 secs, 0.0000 secs, 0.0009 secs
  req write:    0.0000 secs, 0.0000 secs, 0.0002 secs
  resp wait:    0.0038 secs, 0.0028 secs, 0.0057 secs
  resp read:    0.0001 secs, 0.0000 secs, 0.0001 secs

Status code distribution:
  [200] 10 responses



Load test completed after 21.857332ms
timestamp,host,concurrency,count,method,content-type,size,reqsize (b),success,failed,test_time (s),tx_time_50 (ms),tx_time_95 (ms),rps,rps_success
2020-06-03 17:34:53 -0400,localhost,2,10,POST,json,small,1084,10,0,0.021040,3.435814,0.000000,475.275751,475.275751

Troubleshooting

  • Debug
cd lab
docker-compose -f docker-compose.nginx.yaml logs nginx-proxy
docker-compose -f docker-compose.sigsci.yaml logs sigsci-agent-rpc-uds

Various UIs:

# Grafana (login as:  admin/apassword):
open http://localhost:3000/
# Prometheus:
open http://localhost:9090/
# Fortio (EX: use `http://envoy:8085` for testing envoy):
open http://localhost:9876/fortio/