GOV.UK Cookie banner and why it “won't go away”
This is a blog post I’ve been meaning to write for many years. While working at Government Digital Service (GDS) for 6 years as Head of Frontend, I had so many people ask me about the GOV.UK Cookie banner. The main complaint being “it would never go away”. I was asked this question on so many platforms like email, Twitter, and even Reddit! The fact is, it has a surprisingly complex answer!
The aim of this blog post is to answer this question, so I can easily point someone in the direction of this post, should they be interested in the many technical details involved.
First and for-most, what is a cookie banner?
What is a cookie banner?
In simple terms a cookie banner is that annoying pop-up that you get on every website on the modern web. The one that asks you to “accept” or “decline” the storage of cookies in your browser.
Why do cookie banner exist?
Cookie banners exist primarily due to the implementation of privacy regulations such as the Privacy and Electronic Communications Regulations (PECR) in the European Union. There are similar regulations in other parts of the world like the California Consumer Privacy Act (CCPA) in the United States.
What are cookies?
Cookies are small text files that store information on a user’s computer or device when they visit a website. They are used for various purposes. Some example usages are:
- remembering a user’s preferences
- tracking user behaviour across websites
- delivering targeted advertising
Why are they called “cookies”?
The term “cookie” was coined by web-browser programmer Lou Montulli in 1994. Montulli was inspired by the concept of “magic cookies” in early computer networking. These are small pieces of data that are sent between computers to help them communicate more efficiently. The term “magic cookie” was itself inspired by fortune cookies, which are small treats that contain a surprise message.
Montulli decided to use the term “cookie” for this new technology as he thought it would be easier for people to remember than a complex technical term.
Over time, the term “cookie” has become synonymous with any small piece of data that is stored on a user’s computer by a website.
The GOV.UK Cookie banner
In this post, my focus is specifically GOV.UK (www.gov.uk) and related UK Government services e.g.
What you may notice about the GOV.UK homepage (www.gov.uk) and the services listed above, is that they all have the same design.
There are a few minor differences between pages, but the overall aim is to use the same design language across all services. This is why the GOV.UK Design System exists. To help service teams across the UK government achieve the same design quickly and easily.
It’s also worth mentioning that the GOV.UK Design System also has a cookie component with lots of guidance for other service teams to use.
This consistency was a conscious decision by the design team back in 2011–2012. Its design is clean and simple and has always had accessibility & inclusivity as top priorities from the very beginning.
The design actually won the Design Museum Design of the Year Award 2013 and the D&AD 2013 Black Pencil award.
But as with every bold decision in life there are people who disagree. Like this comment from Howard Andrews who says:
Where is the government web site for grown-ups? The whole of “gov.uk” is obviously designed for primary school children and those with special educational needs.
I, personally, disagree with Howard, but as it’s taxpayer money being spent everyone is entitled to express their opinion.
The point I’m getting at is that consistency across the whole user journey was chosen to simplify the site for users. Do users need to know that they are moving between services run by entirely different departments? Not really! As long as they can complete their task quickly and easily and get on with their lives, I’m sure they’ll be happy.
But it’s worth noting that this design consistency across services is a double-edged sword:
Pros
- Gives users a stable design across the complete user journey thus reducing the cognitive load users experience.
- Allows users to recognise a government website instantly.
- Reduces development and design time in the public sector.
- Reduces costs associated with building government services.
- Builds brand loyalty and a site’s trustworthiness
Cons
- Changing a design across multiple departments requires more coordination.
- Technical edge-cases like the cookie banner reappearing across services (a lot more on this later!)
- Users may be unaware they have navigated to an entirely different site.
- Design stagnation due to a rigid set of rules to follow, the design may eventually look dated.
I strongly believe the benefits of consistent branding far outweigh the drawbacks. Especially when it comes to government services that millions of people use every day.
Go away cookie banner!
Of all the “cons” listed above, this is the one that users notice the most!
Users may be unaware they have navigated to a completely different site
If the site design barely changes between user navigations it’s no wonder a user still thinks they are on the same site. As mentioned before, this was an intentional decision. It abstracts away the complexity of government departments into a clear and consistent user journey.
There’s no single source of truth when it comes to back-end technologies in UK government services. For example, www.gov.uk is built using Ruby on Rails. Whereas Universal Credit uses Java. The DVSA’s Check MOT service is built using PHP. Unfortunately, the reason the cookie banner “won’t go away” is exactly because the user is on an entirely different site. There are technical reasons for why this happens that I will cover now.
Browser Privacy
We start our technical explanation with how browsers have changed over recent years to protect users’ privacy online. Back when web browsers were first developed, the internet was nothing like it is today. User-data capture was primitive and so was internet advertising. Unfortunately, the same can’t be said about the “modern web” of 2023. Companies across the globe have realised how valuable user-data is. And a modern-day gold-rush has emerged from the access to billions of peoples private browser data. Companies like Facebook and Google were until very recently able to easily track users across the internet using specific information like:
- Websites visited
- Search queries
- Social media interactions
- Location data
- Device information
From this data, companies can build an extremely accurate digital profile of users. Especially when these companies analyse this data all together (e.g. for all users). Very recently Google toyed with an idea called Federated Learning of Cohorts (FloC).
FloC was a fairly simple idea. Examine all the captured user data and look for similarities in the data. Then create a bucket (or cohort) for all these similar profiles to live in. This turned out to be a great way to track a large group of people while keeping an individual’s identity private. For example, if you recently browsed websites related to motorsport (e.g. Formula 1), then you would be placed into a cohort of other people who are interested in motorsport.
This all sounds very innocent, but it’s incredible how accurate these cohorts can become, when there are 10s of thousands of people in them and millions of data points to examine. These cohorts would become more and more accurate over time as they gather more data and refine the algorithms. Eventually, these companies would have such an accurate idea of who you are and what you like. They’d be able to predict what you enjoy and target you with targetted adverts that they would be fairly certain you were likely to engage with. Thankfully, the privacy community pushed back on the idea of FloC and the development was stopped in favour of a new technology called Topics.
The key differences between FloC and Topics are:
- A user will be assigned to a maximum of 5 topics from a predefined list of 350.
- Cohorts aren’t used, instead a user is assigned to specific topics based on their browsing history.
- A user can edit the topics they have been assigned to and even turn the API off completely if they don’t want targetted adverts.
The Topics API was first enabled in Chrome 115, released on July 17, 2023 as an Origin Trial. On September 20, 2023 the Origin trial ended and Topics is now available for the vast majority of Chrome users.
The current future of interest-based advertising on the web is uncertain. It is possible that these technologies will be replaced by new approaches in the future.
Network isolation and HTTP Cache partitioning
The next topic to discuss is another fairly modern development for browsers. Back in the early days of browsers there was a single browser cache that was used for every site you visited. This left site visitors open to a number of security and privacy attacks such as:
Detect if a user has visited a specific site
An attacker can examine what is in the cache and infer what sites a user has visited from the cache contents. As certain resources may only be available on a specific set of sites.
Cross-site search attack (XS-Search)
An attacker can identify whether a particular search query yielded no results by examining the presence of a “no search results” image stored in the user’s browser cache.
Cross-site tracking
The cache can be employed as a repository for persistent identifiers, enabling malicious actors to monitor user behaviour across multiple websites.
Thankfully, browser vendors have added security and privacy features to stop these attacks. The status of each major browser is listed below:
Chrome
Cache partitioning was first introduced in Chrome 86, which was released on October 5, 2020. Additional storage partitioning beyond the HTTP Cache was enabled from Chrome 115 which was released on October 4, 2022. See the Privacy Sandbox article here for more information.
Firefox
Cache partitioning, also known as network partitioning, was enabled by default for all users in Firefox 85, which was released on January 13, 2021. This feature was introduced as part of Firefox’s efforts to improve user privacy by making it more difficult for websites to track users across different websites. The browser also supports Enhanced Tracking Protection which automatically protects a user’s privacy while they browse.
Edge (v79+)
Edge usually follows Chrome closely, and at the time of writing Chrome and Edge are aligned and stable in terms of Cache partitioning. More information on Microsoft’s plans for Edge and the Privacy-Preserving Ads API in 2024 can be found in this blog post published on the 5th March 2024.
Safari
Apple has been way ahead of the curve with these features. The team actually talked about these features way back in February 2013! For more information check out this post “Safari, Caching and Third-Party Resources“ by good friend Andy Davies.
iOS Intelligent Tracking Prevention (ITP)
While we are on the subject of Apple and Safari, it’s probably a good time to mention their default tracking protection. Apple enabled Intelligent Tracking Prevention (ITP) in Safari with the release of iOS 11 and macOS High Sierra on September 19, 2017.
ITP is used to stop sites from tracking a user across the internet. Safari automatically examines resources downloaded from a users’ journey around the internet.
Machine Learning is then used on the device to classify these resources to see if they could be used for tracking purposes. For more privacy, if a user hasn’t interacted with a website in the past 30 days the website data and cookies are immediately purged.
ITP 2.1
In February 2019 Apple released ITP version 2.1 for iOS 12.2 and Safari 12.1 on macOS High Sierra and Mojave. This version brought major changes for sites wishing to track users. Third-party (tracking) cookies were blocked completely. Furthermore, any existing cookies that have had no user interaction for 30 days are then purged. More importantly for the GOV.UK cookie banner, any cookie set using the document.cookie API automatically expires after only 7 days!
Implications of ITP 2.1 for GOV.UK
Although most users of iOS 12.2 and Safari 12.1 on macOS High Sierra and Mojave wouldn’t have realised ITP 2.1 had been installed and enabled on their devices. This was, in fact, a considerable change for websites that used this data for decision-making purposes.
GDS and the GOV.UK team in particular, were one of those impacted by ITP 2.1. Although I’ve heard many complaints over the years about this, GOV.UK uses Google Analytics (GA) for its analytics. The data captured is pretty limited. For example, whole sections of GA aren’t enabled internally, like “user demographics”, “behaviours”, and “interests” weren’t available for performance analysts to see. That’s not to say Google can’t see that data themselves!
But this is the hand we were dealt and a legacy of the age of GOV.UK. Back in 2012 when GOV.UK was first launched an Analytics product was needed to evaluate and prove that users were using GOV.UK and which parts were being used. User privacy wasn’t as important as it is today, and GA was likely the most feature rich (and easiest) to implement. In hindsight a self-hosted analytics solution should have been used like Piwik (now Motomo). Although I can’t be sure if the versions at the time had all the features required by the GDS teams at the time (2007). See the additional context from Tom Loosemore below for more information on the early decisions made by GDS about tracking.
A cookie banner every 7 days!
With the release and rollout of ITP 2.1 on iOS and OSX this meant the annoying cookie banner at the top of GOV.UK would appear every 7 days. This is because the cookie used to store the information that the user had already seen it would be purged every week. So a user browsing GOV.UK first thing Monday morning on an iOS device, accepts the banner then exactly 1 week later they will need to do the same. This would be pretty frustrating for the user, but unfortunately, it’s completely outside any website’s control. So if you’re an iOS user and have ever wondered why you keep seeing the same cookie banner over and over again, this is the reason!
A workaround to the 7 day problem
Thanks to Barry Pollard for this info on a workaround. I quote:
When you drop the cookie using client side JS, you make a JS API call a back-end which reads the cookie, and then sends it back as an HTTP set-cookie header. Boom - it’s no longer a JS cookie.
This is allowed AFAIK because Apple wants to block JS that isn’t on the same domain as the document (i.e. most likely tracking cookies). Their intention was never to block pure 1st party cookies as I understand it.
iOS Mobile Bias
According to the monthly GOV.UK statistics I posted back in February 2023, Safari made up 31.48% of traffic in the month. And 56.21% of users were on a mobile, with the Apple iPhone being the most popular device by an absolutely huge margin of 46.98%!
Unfortunately, with the ITP change in version 2.1 there’s no way of knowing how accurate these figures are. This is why I always posted a notice in the tweet thread saying:
‡ Note: since December 2019 GOV.UK requires explicit opt-in for tracking which introduces bias. This is especially true for mobile devices, since the cookie banner takes up more screen estate and is more likely to be accepted.
Which brings us back to the cookie banner design. This is what GOV.UK homepage looks like on Desktop:
This is what it looks like on Mobile:
As you can see, the cookie banner is huge on any mobile device taking up around 65% of the screen. It’s also worth noting that a user is only seen by GA if they accept the cookie banner. As the cookie banner on a mobile is huge, a mobile user is more likely to “Accept” it so they can actually view the page content! This where the mobile bias comes into play with the GOV.UK analytics data.
Note: This only became an issue when the Information Commissioner’s Office (ICO) ruled that tracking all users by default wasn’t adhering to their explicit consent policy.
Additional Context
This is one of the reasons why I love blogging so much, it’s astonishing how much you can learn from the feedback other people give you.
And it’s also a great example of why you should keep some form of decision record in an organisation for when people leave and take all that precious knowledge with them! A huge thank you to Tom Loosemore for this additional context. Tom was Deputy Director, Government Digital Service, Cabinet Office, between January 2011 and October 2015.
I’ve directly quoted his comment he left me on LinkedIn related to this post:
Behind the scenes GDS tried very hard circa 2012 to convince the ICO that forcing cookie opt-in pop-ups would be counter productive to their laudable aims. Had they done proper user research as part of their response to PECR legislation we might not be in this mess. Policy is delivery, redux.
The GOV.UK team blogged several times about balancing the need for user analytics with privacy, with a nicely intense debate in the comments. Sadly piwik was totally unsuitable as an alternative to GA. Enough well-meaning organisations had adopted it and then regretted it for us to know it was a non-starter. See this blog post
Google would break its contract with HMG if it looked at or used the GA analytics data collected across *.gov.uk for anything other than protecting the GA service itself from attack. The options to stop Google employees (even GA support staff) accessing the *.gov.uk analytics data still seem strong enough to me. See this blog post.
The Public Suffix List
When it comes to setting cookies across domains gov.uk and service.gov.uk, there’s an added complication. Both of these domains are on the Public Suffix List (PSL).
In simple terms, the PSL is a list of domain names that are considered to be effective top-level domains (eTLD). This has ramifications for setting cookies, that I will go into next.
If we wanted to stop the cookie banner appearing multiple times across a GOV.UK users journey, a cookie needs to be set to store the information that the banner has been seen and accepted. If we wanted to achieve this only using cookies, the PSL stops us from being able to do this.
As mentioned above gov.uk and service.gov.uk are in the PSL. So they are known as an effective top-level domain (eTLD). Browsers handle eTLD’s differently to other domains. It so happens that Network isolation and HTTP Cache partitioning are partitioned using the following URL structure scheme://eTLD+1. There’s an example below that, I hope, will explain it clearer:
For a GOV.UK user browsing from the homepage to the Universal Credit Sign In page the browser sees the following:
// ‘gov.uk’
eTLD = gov.uk
+1 = www
eTLD + 1 = https://www.gov.uk
// any other government service
eTLD = service.gov.uk
+1 = www.universal-credit
eTLD + 1 = https://www.universal-credit.service.gov.uk
OR
eTLD + 1 = https://www.check-mot.service.gov.uk/
OR
eTLD + 1 = https://www.payments.service.gov.uk/
OR
// I think you get the idea
The eTLD + 1’s are the keys used for cache and network partitioning. Due to this partitioning, any cookie set on www.gov.uk can’t be seen or shared on www.universal-credit.service.gov.uk or any other service domain. Thus, a user that accepted the cookie banner on the homepage will see it again in their user journey because this cookie can’t persist through to the service domain.
This is the reason the cookie banner “won’t go away”. Each service on a user’s journey, sees them as a brand-new user that needs to accept that service’s own cookie banner.
Unfortunately, as we mentioned earlier service domains follow the same GOV.UK branding, so many users won’t even realise they are browsing an entirely different website.
Final Thoughts
So there we have it. A whole brain dump on the GOV.UK cookie banner, GOV.UK service design and surrounding browser technologies related to privacy and security. Hopefully, you found it interesting. Maybe it answered a few questions you didn’t even want to know 😂 As always if you have any questions or feedback, please do get in touch.
Post changelog:
- 12/12/23: Initial blog post published. Happy Birthday to me!
- 12/12/23: Thanks (again) to Barry Pollard for the numerous fixes to the Cross Site Tracking section of the post and a workaround for the 7 day time limit on Safari!
- 13/12/23: A huge thanks to Tom Loosemore for his incredibly informative LinkedIn post on the ICO and GA in the early years of GOV.UK. His update can be seen here.
- 13/12/23: Thanks to Joe Lanman for pointing out it’s Privacy and Electronic Communications Regulations (PECR) not General Data Protection Regulation (GDPR) that is important for cookie banners.
- 13/12/23: Thanks to Alex Russell, for getting back to me about the status of Cache partitioning in Edge.
- 05/03/24: Thanks to Rowan Merewood, for the new information on what the Edge team has planned for 2024 in their latest blog post.
