Skip to content

Agent CDN Service

To aid in download performance and to reduce costs, the Amazon Web Service(AWS) S3 buckets have been fronted with the Fastly CDN. This means Agents starting at version v4.30.0 and above will attempt to download data from the Fastly CDN first, before failing over to the two AWS S3 buckets.

Agents v4.30.0+ will attempt to download data from the endpoints in the following order: 1. https://wafconf.signalsciences.net - Reported as cdn in agent metrics 2. https://sigsci-agent-wafconf.s3.amazonaws.com - Reported as direct in agent metrics 3. https://sigsci-agent-wafconf-us-west-2.s3.amazonaws.com - Reported as failover in agent metrics

If all of the above fail, a download error is reported. Agents v4.29.0 and below do not attempt step 1.

Both external data and agent config/rules(GOSH) are cached. As soon as new data is uploaded to AWS S3 by Rulebuilder, a URL purge is performed on the resource using the Fastly API.

Config and Monitoring

Fastly Service - 2zIHlIZYVKm0vC7pte9JG9

CustomerID: 1W2UqpknmiWruazHkLMXCp ServiceID: 2zIHlIZYVKm0vC7pte9JG9

The primary means of configuring and controlling the Fastly CDN delivery service is through the Fastly Console located at https://manage.fastly.com/configure/services/2zIHlIZYVKm0vC7pte9JG9. Access to the management console is controlled through Fastly-Eng Okta. This is a secondary Okta service for accessing engineering specific resources and access is controlled programmatically, instead of by IT. After logging into the normal Fastly Okta, a Fastly-Eng Okta a chiclets can be logged into that provides it's own set of chiclets. Look for one named security-products to be taken to the Fastly Console. Full flow should be Fastly Okta -> Fastly-Eng Okta -> security-products.

Access to the security-products "customer" console is controlled by team membership defined programmatically here. You may also need to be directly invited via the console interface.

Monitoring Dashboards

CDN Performance Dashboards

These dashboards give insights into the performance of the CDN itself, such as request volumes, cache hit ratios, etc - Fastly Console Stats (Realtime + Historic) - Fastly Observability Dashboard - Largely the same data as on the Fastly Console, but more configurable

Agent Download Dashboards

These dashboard give insights into the download metrics of the agents themselves as well as CDN purging. Agent metrics are incremented based on which source the agent successfully downloaded configuration from, or if all attempts failed (error). Purging metrics come from the Rulebuilder service. - Datadog Dashboard - Aggregate metrics across all corps. Good overall view into producers + consumers. - Kibana (Chili) Dashboard - Includes per corp breakdowns to help troubleshoot behavior

Alerts

The following alerts have been created to help identify possible issues with the CDN service or agent's ability to download data. - Successful CDN download volume anomaly (Datadog monitor) - Download errors volume anomaly (Datadog monitor) - CDN vs S3 volume anomaly (Datadog monitor)

Another set of alerts monitors the purging performed by Rulebuilder following an upload to AWS S3: - Purge API errors (Datadog monitor) - Purge API ratelimit quota (Datadog monitor) - Purge API purge request duration (Datadog monitor)

Runbooks

Runbook: Stale data being served by the cache

First priority is to get the stale data purged. This can done most easily using the Fastly Console, or alternatively through the Fastly purging API.

  1. Log into the Fastly management console via Fastly Eng-Okta (specifics above). https://manage.fastly.com/configure/services/2zIHlIZYVKm0vC7pte9JG9
  2. Determine what content needs to be purged. Ideally just the impacted subset of data can be purged, but in an emergency a purge-all can be performed to clear everything.
  3. Use the Purge dropdown located on the top left of the Agent Config Download service's Service Summary tab to select the desired purge option:
  4. Purge URL only the specific URL provided will be purged Ex: 4/geolite2-country.data - Purge URL Docs
  5. Purge Key only objects tagged with certain surrogate keys will be purged Ex: external, config - Purge Key Docs
  6. Purge All every cached object will be purged from the cache. - Purge All Docs
  7. Follow the linked purging docs for the chosen option: URL, Key,Purge All
  8. Wait for the purge to take effect across the fleet, monitoring for improvement.
  9. The Check Cache tool (located to the right of the Purge dropdown) can be used to verify the current Etag of a cached URL across the Fleet.

After stale data has been manually purged above, look to see if there are problems with cache purging. Rulebuilder ought to automatically purge the cache via a URL Purge each time it uploads data to S3. If rulebuilder is failing to purge items for any reason, caching can temporarily be disabled for all agent configs(GOSH) or for a specific site's config.

Disable agent config caching for one site

Agent config caching can be disabled for individual sites, causing the VCL snippet to always perform a cache PASS action and send the request to the upstream S3 bucket.

To disable a site from being cached by the CDN: 1. From within the Fastly management console for the service, go to the service configuration and look for the Data > Dictionaries config section. 2. Within the Excluded Cache Paths dictionary, add the URL Path of the config object you with to exclude from caching as the key, and true as the value. Changes to the dictionary will take effect live after adding the new entry. CDN Config Dictionary The path is based on the accessKeyID for that site, and can be retrieved from Salt on the Agents > View agent keys button. View Agent Keys For example, the accesskeyid 12345678-abcd-1234-abcd-123456789010 would have path /4/12345678-abcd-1234-abcd-123456789010.conf You do not need to release a new "Version" of the Fastly service config for this to take effect. 3. It may take a few minutes for this to take effect globally. To speed this process, a URL purge for the added path may help this take effect faster. 4. Confirm the change took effect by curling impacted resource: curl -I https://wafconf.signalsciences.net/4/12345678-abcd-1234-abcd-123456789010.conf you should consistently see the x-cache response header value change to MISS once the change takes effect. It may take a few minutes for the change to propagate.

To revert this bypass in the future, simply remove the entry from the dictionary.

Disable all agent config caching globally

Agent config file caching can be disabled globally, causing the VCL snippet that controls caching to always perform a cache PASS action and send the request to the upstream S3 bucket.

To disable globally: 1. From within the Fastly management console for the service, go to the service configuration and look for the Data > Dictionaries config section. 2. Within the CDN Config dictionary change the value of the Disable_All_Config_Caching key to true. You do not need to release a new "Version" of the Fastly service config for this to take effect. Changes to the dictionary values will take effect live after saving the new value. CDN Config Dictionary 3. It may take a few minutes for this to take effect globally. To speed this process, a Purge All or Surrogate Key purge for config objects may help this take effect faster. 4. Confirm the change took effect by curling impacted resources: curl -I https://wafconf.signalsciences.net/4/12345678-abcd-1234-abcd-123456789010.conf you should consistently see the x-cache response header value change to MISS once the change takes effect. It may take a few minutes for the change to propagate.

To revert this bypass, simply change the Disable_All_Config_Caching's value to false.

Runbook: Fastly service is down and not serving traffic

In the event of a Fastly network or service failure the Agent should automatically failover to connecting directly to the two S3 buckets. Agents should continue to After experiencing a number of consecutive failures to download from the CDN, the agent will begin exponentially backing off and skipping attempts to the CDN endpoint all together. At the least the agent will check the CDN endpoint once a day. Upon a successful download from the CDN the agent will reset it's backoff to 0 and resume fetching from the CDN endpoint as the first priority.

If the CDN is down, agents will continue operating as normal, pulling data directly from S3. If both S3 regions are down, there will be customer impact as agents won't be able to pull updated rules or IP lists.

  1. Verify if the service is indeed down. Attempting to fetch the GeoIP external data is a good test: curl -I https://wafconf.signalsciences.net/4/geolite2-country.data. Ideally you should get back a HTTP 200 with the x-cache header set to HIT or MISS.
  2. Check the dashboards for the CDN Performance and Agent Downloads. Look for clues in a big change in behavior.
  3. Check if both the S3 buckets are also down, as they might be the real culprit: curl -I https://sigsci-agent-wafconf.s3.amazonaws.com/4/geolite2-country.data and curl -I https://sigsci-agent-wafconf-us-west-2.s3.amazonaws.com/4/geolite2-country.data. If the problem is present with both S3 regions there will be customer impact and an incident should be called: #ic
  4. If the problem lies with the CDN and not with S3 there isn't much the agent team is equipped to do. Check for an ongoing incident in #ic and #incident-reportcards. If you don't see one you should consider reaching out in #ic to get assistance if you believe there is a larger Fastly issue.