Filtering Automated Clicks and Openings
Filtering robot clicks
Screening bots are processes linked to anti-spam filters or other security softwares, which go through each link of received e-mails to check that they do not lead to dangerous websites, and therefore protect recipients from viruses or phishing attempts.
These robots are especially encountered in a B2B context: indeed, companies often have additional security measures in their corporate mailboxes, which carry out an analysis before the delivery in the mailbox of their collaborators. On the other hand, such measures are rarer on private addresses.
While security concerns are legitimate, the consequence is more bothering for the marketeer who wants to analyze their statistics: indeed, going through links will register a click, as the bot accesses the URL used to track clicks. As a result, 'false' clicks are recorded (and 'false' openings, as clicks also generate an opening), which will prevent the marketeer to see if impacted B2B profiles have really clicked in the e-mail or not.
To tackle this problem and offer statistics that only reflect human behaviors, Actito has deployed several features.
Filter on the user-agent
The user-agent is in some way the signature of an internet user. It allows to identify the kind of device used by a profile, its browser, its version, etc. This information is stored on each e-mail interaction in a technical profile attribute.
The first feature that has been deployed is a filter based on this user-agent. Thanks to an external index of all the user-agents identified as robots, all interactions bearing the digital signature of a bot are automatically filtered.
However, this solution is mostly useful for bots of older generations. Indeed, newer ones hide their signature and display a user-agent belonging to a legitimate user, which is therefore not associated to a robot and goes unnoticed. This is why we have developed a second filter.
Filter on behavior
In an e-mail interaction, robot clicks can usually be identified because a large number of clicks have recorded a click at the same time on all links of the e-mail. We have therefore deployed an additional filter based of the clicking behavior.
This feature will analyze clicks over a longer timeframe to make sure that they can be considered as legitimate.
A click will be filtered as a robot click if:
- a large number of click is recorded at the same time (burst of clicks)
or
- more than 5 clicks are recorded within 1 minute
This second criterium is the result of 3 months of real data analysis. It might look loose and in fact, a human user could reproduce this behavior. But one must consider that all robots do not click at he same second: some space out their clicks to display a more human-like behavior. It is therefore necessary to apply this filter over a longer timeframe. And the real data analysis has established that real users actually do not fulfil this criteria, as after clicking, the human behavior is to browse at least a little bit the page on which one has arrived.
Once the e-mail has been validated by the robot and delivered to the mailbox, if the the recipients clicks in it, the click will be recorded and the status of the e-mail will be updated as 'clicked'.
The filter based on the user-agent is automatically active.
The new robot clicks filter based on the behavior is activated on demand (by e-mail to support@actito.com).
Unsubscribe filter
More than a statistics problem, screening bots could lead to issues of unwanted unsubscribes. There is indeed no reason why they would spare the unsubscribe link...
Fortunately, an unsubscribe is made in 2 steps: with a redirection for the unsubscribe to be taken into account. And Actito has made sure that robots cannot follow this redirection.
So even if a robot screens the unsubscribe link, the script used to effectively unsubscribe a profile will not be activated.
Filtering Apple MPP openings
On the 20th of September 2021, Apple has launched their latest operating system version, iOS 15, which includes their new Mail Privacy Protection (MPP) feature.
This feature aims at stopping the e-mail sender from using invisible pixels to collect information about the recipients. This is done by routing e-mails trough a proxy server that pre-loads e-mail content (like images) before transferring it to the recipients. This behavior has 2 main effects :
- The pixel used to track openings is downloaded in advance, which makes it impossible to know if or when a profile has really opened the e-mail
- Collecting information such as the device, user-agent or IP address of the recipient is not possible
The first point is specifically important for the markeeter, as it means the 'openings' KPI becomes less accurate for Apple Mail, and all that depends on it (reporting, scenarios, AB testing, segmentations) becomes less reliable. Actito has therefore decided to react accordingly.
From a philosophical perspective, this article details Actito's views about the role of the 'opens' metric and the goal behind Apple's decision.
From a practical perspective, Actito has developed a way to filter automated openings Apple in order to avoid artificially inflating your 'openings' KPI.
Who is affected ?
The Mail Privacy Protection feature is available for the Apple Mail app on iOS 15, iPadOS 15 and macOS Monterey devices.
It is not activated by default : after updating their OS, users will be prompted to choosing between 'Protect Mail activity' or 'Don't protect Mail activity'. Adoption rate is expected to be high, but it is worthy to note that the user has to choose to activate the feature.
It is also important to note that that only the Apple Mail app goes through this proxy server. If the user opens the e-mail with the Gmail app or another e-mail app, even on iOS 15, the privacy protection feature will not be in effect.
In addition, only openings are impacted (through the download of the tracking pixel). Clicks in the e-mail are not affected.
What has Actito implemented ?
Actito has implemented a filter on the user-agent used by Apple's image downloading proxy (the user-agent is, so to speak, the 'fingerprints' of your browser and device). Indeed, as Apple also aims at masking the user-agent of their users, by effectively replacing it by a (fixed) user-agent of their own.
All openings recorded on this specific user-agent are therefore filtered. This means that, potentially, 'real' opens may be filtered if profiles do not click in the e-mail.
Indeed, remember that clicks and unimpacted and that a click in an e-mail also records an 'opening' interaction.
In addition, if a profile opens their e-mail in another app than Apple Mail (Gmail or Outlook for instance) an opening will be recorded, even if an opening had been filtered already on iOS 15 before.
What is the expected impact ?
Overall, this means that you should expect a decrease in opening rate for your campaigns.
The difference will depend on the adoption rate of the Apple update and especially on the adoption rate of the Mail Privacy Protection feature.
It will also depend on the share of (up-to-date) Apple users in your database.
All in all, we feel that filtering automated openings is a better option than artificially inflating your opening rate with all iOS 15 users, especially since real openings can be still be detected (through clicks or another app).
Example of applications
- If a profile uses an Apple device on iOS 14 or any previous OS, there won't be any automated opening filter : if he opens the e-mail, an opening will be recorded.
- If a profile uses an Apple device on iOS 15 without activating MPP, there won't be any automated opening filter : if he opens the e-mail, an opening will be recorded.
- If a profile uses an Apple device on iOS 15 with MPP...
- and doesn't open the e-mail : An automated opening will be triggered by Apple and it will be filtered by Actito.
- and opens the e-mail with Apple mail but doesn’t click on any link : The opening will be filtered by Actito.
- and clicks on a link in Apple Mail : As a click is detected, an opening is counted by Actito.
- and opens it with the other mail app (and download images) : An opening is counted by Actito (even if pre-loaded by Apple Mail).