Highwinds services have been architected in such a way that there is no technical limit to the number of CDN host and sub-accounts that can be created, a feature that resellers and customers with hundreds or thousands of downstream customers find extremely beneficial. However, in practical terms, configurations that have more than a few hundred hosts can cause issues, so we’re sharing best practices for CDN host and sub-account management in this post.
Some issues with maintaining too many hosts on a single account include:
– Slower API and UI response with loading large lists
– Increased database load when dealing with large data sets
– Increased overhead on the logging, FTP, and analytics collection services
– Cumbersome interaction with log storage created by having to make thousands of individual connections and log retrievals
In a situation where you have potentially thousands of smaller customers you are supporting, it’s preferred to find a way to minimize the number of CDN hosts that are created whenever possible. These situations typically involve a large amount of automation, anyway, so the goal should be to automate in the most efficient way possible.
To learn more about hosts and scopes, visit our support portal.
Typically, if you find yourself potentially needing 10,000 CDN hosts, then you have a platform that you are loading large numbers of clients on to, and you will be looking for a way to easily pull usage and billing data for each customer. While the billing and analytics portion is solved by using an individual host for each client, you will find that this will not scale well as your customer base increases.
Each one of these clients will have nearly the same configurations, with only a few key differences between each that need to be addressed in order to gather the required usage data. So instead of an individual host for each client, look for a way to re-use those base configurations, which are inherited across an entire host, and put as many clients as possible into each single host. Instead of 10,000 hosts, create 10 hosts, each of which are split into groups based on their configurations.
For example, you could group them by:
– First number or letter of their CustomerID
– What month they signed up for services
– Which origin or geographic region they are pulling from
– Which platform they are using
Once you have the groupings determined, you can set up those base hosts with all of the basic caching rules, origins, and other configurations that will be shared by each of those clients.
Each Highwinds host can also have any number of scopes defined. Scopes are pseudo-directories that correspond with directory paths of your objects, and are created so that configurations can be made to a sub-set of your files. If you need to support hundreds or thousands of clients on a single host, then you can easily separate them (and their configurations) by adding each to their own scope.
Once a new scope is created, you can then add a couple of configurations to the scope which will enable delivery for that client as well as add some identifying information in order to pull their usage out of the raw access logs. You will want to add the following configurations to each scope:
1.) A new hostname for the client.
– This is the client’s URL, or a sub-domain of yours that identifies the client.
– When used it will mask out the scope entirely, so requests to the CDN will not have any extra directories in them.
2.) A new static header policy to add identifiers.
– Host: customerdomain.com
– CustomerID: a1b2c4 [This is optional, but highly suggested. You will want this to be a type of hash or token that identifies your customer, and it can be anything you want.]
– These headers can now be used to track the customer within the logs.
3.) A new origin connections policy. [This is optional.]
– If the path to the customer is different from the base configurations you will need to tell the CDN where this client origin is located.
You can also add any other custom configuration policies at this time, such as cache control or content protection, but generally these should be common across all of the clients on each host.
Once this is done your request flow will look something like this:
Logging and Data Collection
Once the delivery for each client is successfully going through their own scopes, you will then want to add a couple of custom log fields to the host so you can track customer usage. Log usage will be vital here, since in this type of configuration the analytics API will not be able to show you usage on an individual scope.
While this may seem like more effort at first, combining the hosts and using the logs will be much more efficient in the long run since you will only have to make a request to the log store for data from a dozen hosts, as opposed to making thousands of individual API calls every few minutes if you split each client into their own host.
In the Hosts tab within StrikeTracker, click on the Services tab, and then edit the the host information. There is a field called Custom HTTP Headers for Access Logs, and you will want to add the following:
Once added, those columns will now be written into the logs for this host, and you can use those to pull usage data for a particular customer.
An alternative to Highwinds raw access logs is our new request receipts platform. Request receipts send the equivalent of a normal log line back to a server that you control for each request that comes through to the CDN, writing log lines directly to your servers as they come in.
This can be especially handy for getting real-time data, and you can manage the data however you need, writing it to a database, storing it in flat files, or discarding it after gathering the usage information.
Note that this does require you to deploy and manage a server for this task. Often customers can use their current web servers for the task (the only requirement is that we have a URL we can send the receipts to), but high traffic volumes may require resources dedicated to the task.
A typical request receipt is just a simple request to your web server, with the log data added as URL parameters to the request. They look something like this:
To learn more about request receipts, please contact your solutions architect or account executive.
By Barry Whitley, Director of Solutions Architecture