Content Delivery Networks (CDNs) have come a very long way in the last few years. While they were highly coveted by international websites due to their ‘Cloud nature’, they have not always been considered to be very intelligent. This has changed in recent years, as they have brought additional functionality and logic to the Edge. These new features can help us improve websites and reduce the technical debt imposed on a content management system (CMS) for outside functionality.
Over the last few months we have been exploring ‘edge workers’ to see how we can use them to benefit our clients’ websites. In doing so, we have used Cloudflare’s Workers to leverage the free plan and easy integration they offer. However, the majority of the edge workers are JavaScript driven (edge-side, not client-side), meaning that other suppliers, like EdgeWorkers and Netlify Edge Handlers are available as well.
As a CDN sits on the ‘edge’ between the client (a user-agent, such as your browser or a search engine bot) and the origin server (the server the website resides on), it is in the perfect place to handle rudimental logic and to impose high-level rules. This means that the CDN can:
- Reduce the number of requests made to the origin server (a reduction in traffic can work economically more efficiently and improve stability of the server)
- Increase security by filtering out malicious requests (such as Distributed Denial of Service (DDoS) attacks)
- Cache the resources for a specific amount of time (improving page load speed)
With the addition of workers, the scope and effectiveness of a CDN can be further improved.
Let’s explore a few areas of a website that can show improvements through the implementation of edge workers, including:
- Uniform Resource Locator (URL) sanitisation/normalisation
- Preview of changes to a page
- Migrations
- Domain consolidation
1. URL sanitisation/normalisation
From a search engine optimisation (SEO) perspective, we want websites to be as streamlined as possible, with any duplicate pages removed. Failing to ‘sanitise’ URLs often causes duplicate content issues. For example, the same content could be shown on a page with and without a trailing slash. However, as the same content is shown on both unique URLs, search engines can crawl both versions, causing crawl-budget inefficiencies and duplicate content issues.
There are a few of these types of rules that can cause issues, including the subdomain rule (forcing the inclusion or removal of ‘www.’), the transformation of uppercase letters to lowercase, and of course, redirects to the secure Hypertext Transfer Protocol Secure (HTTPS) scheme.
These rules are often implemented by the origin server or even through custom configuration within the CMS, which increases the number of requests to the origin server and adds technical debt.
However, by implementing these rules using an edge worker, we will be able to overcome these obstacles and inefficiencies.
To test this, we used workers to create a proof of concept that applies the following rules:
- The host must be the test domain (subdomain, root domain and top-level domain).
- The website must be on the secure HTTPS scheme.
- Ideally, a trailing slash should be on the URLs.
- All letters must be lowercase.
The objective is to ensure that the URL has a single redirect if one or more rules are not met, while maintaining the query string parameter.
A test request to a URL that meets all the criteria listed above, shows that the server responds with a 200-OK status code:
When creating a request to a URL that misses multiple criteria, we can see that the server redirects the URL:
This shows that Cloudflare Workers can support the origin servers by forwarding unnecessary requests, as well as implement a more complex logic to avoid sequential redirect requests (chain-redirects).
2. Preview of changes to a page
When a brand or client asks about the impact of a change to their website, we can often forecast the projected improvement in numbers for the campaigns. But quite often, we tend to forget the visual impact a change can have on a website or, for example, what the change means for page load speed.
Using photoshop can only bring you so far – what if we could apply the changes to a page in-situ? What if we want to use third-party tools such as Google’s Page Speed Insights or WebPageTest.org to review the impact of the changes on page load speed?
Proof of concept
Using workers, we can replicate a page in-situ and manipulate the source code. As the proof of concept is publicly available (we ensure that all requests are marked as ‘noindex’ to avoid search engines indexing the content), we can use the aforementioned third-party tools to measure the impact of our changes.
The worker will deal with the inbound request and serve the requested page. In case of a page that is in scope for the test, it will manipulate the Hypertext Markup Language (HTML) before it is served to the user-agent, effectively optimising the page between the origin server and browser or search engine bot.
In order to provide accurate measurements, we create at least two copies of the page; one copy will remain original as it is, while the other copy will have the additional optimisation added to it. This allows us to compare the two pages on the same network and architecture, and measure the impact as well.
Case study
During a page speed audit, we have identified that a page would benefit from ‘rel=”preload”’ and ‘rel=”prefetch”’ directives. These changes are relatively small to implement and proved to be perfect candidates for a proof of concept.
Our proof of concept created two versions of the page:
- Original – the original page as it’s served by the origin server
- Preload – the optimised page where the additional HTML elements have been added
Now, with the availability of these pages to tools, we are able to measure the impact of the changes with tools like Page Speed Insights (Lighthouse), and provide additional reasons into why they should implement the changes.
Source: Google Page Speed Insights
Using this method, we will be able to tell clients exactly what we would like to see implemented, how we would like to see it implemented, and the potential impact it has. This can also be used for other implementations and is not limited to technical improvements. For example, content blocks, styling, and user experience (UX) considerations can be applied and previewed prior to full implementation.
3. Migrations
Migrations are always tricky for clients, as inaccurate mapping and incorrectly implemented redirects can cause a significant negative impact on revenue and other key performance indicators (KPIs). Most migrations are handled by the origin server, and often when a domain has been decommissioned, the redirects will be dealt with by a small server or even in coexistence on the new server. This can cause all kinds of performance inefficiencies, and even force the origin server to handle more mundane requests than necessary. There are also cases where a CMS cannot handle the domain change and redirects at the same time. We have experienced this while working with clients who use Shopify, where the platform could not avoid chained redirects.
Implementing the redirects through an Edge worker would be more efficient and could reduce the number of requests to an origin server. In case of a domain migration, the need for a server to deal with the redirects would also be diminished, as the Edge worker can handle it all.
One of the drawbacks of using an Edge worker when it comes to migrations is the number of redirects that it needs to support. While a handful of redirects can be easily embedded within the script, it becomes unmanageable when dealing with hundreds or thousands of rules. To overcome this issue, we have created a Cloudflare Worker that is able to retrieve an external file from a publicly available Github repository (other locations are available too).
The worker tests the requested path against the redirect file and is able to provide the correct redirect or serve the requested page. Within the redirect file, the origin path, destination and the status code can be defined, and the worker will process this information. To add, update or refine a redirect rule, the external redirect file needs to be updated and reuploaded (or in this case committed). The worker will then be able to digest the new rules and apply the updates seamlessly.
This means that we can provide more accurate support to the client during the migration, where we can test the redirects in exactly the same environment as the one that will be used for production during the launch day. Additionally, we will also have the ability to tweak the file with the rules more easily.
The aim is to reduce the risk associated with a migration, reduce technical legacy and debt with the various impacted servers, and ultimately ensure that the origin server remains as streamlined and clean from redirects as possible.
4. Domain consolidation
Various parts of websites often have different requirements that need specific systems to be in place, which often causes clashes between different architectures and CMSs – this is one of the most common issues between an Internet Information Services (IIS) eCommerce store and Apache WordPress blog. Similar problems may arise when an international brand uses a fragmented method for localised websites, like having different ownership and governance for its British site from the one it has for its German or Spanish sites.
In order to overcome these challenges, brands often operate with different subdomains (think about ‘blog.example.com’) or different country-code top-level domains (ccTLD), such as ‘example.nl’ and ‘example.co.uk’. To make these different domains appear on a single subdomain, a strategy called ‘reverse-proxy’ is often used – this method is labour intensive for servers and can cause page speed inefficiencies reflected in a high ‘time to first byte’ (TTFB).
This is where edge workers can support – they can reduce the amount of connections required to the various origin servers and improve the TTFB metric. The workers will be able to retrieve the information from the required origin server – whether it is on a subdomain or on a different domain altogether – and serve the content as if it is part of the main domain.
We have created a proof of concept where a Cloudflare Worker retrieves the homepage from either nextjs.org or example.com based on the request.
Using this method to consolidate domains will allow brands to utilise their CDNs more effectively and decrease the number of unnecessary requests to origin servers. Ultimately it will benefit both search engines and users as the page load speed will improve and, as expected, the experience of a single domain will be provided.
Closing words
Edge workers are versatile and can provide a large number of options for different obstacles and challenges, while continuing to provide their usual benefits from their caching qualities and cloud-nature. While this blog post only touches on four different areas where workers can provide a positive impact to a website, I am certain that they can play an even bigger role.
If you want to know more about our proof of concepts or other areas we are exploring with these workers, get in touch with our teams.
