Occasionally in our DNS traffic we see incredibly large bursts of traffic requesting a particular domain. Here we see an example. We’ve obscured the name as it was full of obscenity.
What we see here is 500 requests in one hour for a domain that doesn’t exist, in fact if we look closer we can see that the burst is basically over in a matter of seconds:
There is absolutely no traffic for this domain either before or after this very short burst of activity.
Very unusual and certainly worth further investigation, so the next thing we did was look at the sources for these queries, is it a single machine that has gone haywire, or a bunch of machines acting in a highly coordinated way? Looking into the sources we see 468 unique IP addresses spread over 232 ASNs and 61 countries! The level of coordination here is quite astounding.
The queries we receive come from recursive resolvers, so we only receive a query when that resolver has not previously cached the answer and being a non-existent domain this is unlikely to be cached (which would result in a larger number of requests)… But still, that level of coordination is remarkable.
Some resolvers will answer a question from anyone (so-called open resolvers) as opposed to just serving people on their own network. This means that in theory a single person could trigger all of these queries and appear as traffic from all of those ASNs. However a quick test showed that in the top IPs only 1 out of 77 was an open resolver, so this is not the situation in this instance.
Here we see the countries involved (darker colours indicate a higher number of queries from that country), so we can see that this was a truly global event.
So the questions are what is this traffic and should we be worried? We could not find any trace of the domain via search engines, so whatever triggered it was either through a channel that is not covered or was so ephemeral that it didn’t have time to be indexed. A dead-end then? Well not quite, as now we know what to look for we soon found another domain showing a very similar query pattern, also containing an obscenity but this time attached to what looked like a username or handle. This time a bit of digging revealed it belonged to an account on twitch.tv (a video sharing site for gaming) and in fact we could tie it to a particular chat channel. Luckily for us this chat was logged and so we could search for reference to our handle. Sure enough, there in the log was the domain in question at exactly the time we were seeing the DNS traffic.
Okay, but does that mean that everyone clicked on this link (which was posted as an insult) as soon as it appeared on their screens? Well, no. It is obviously not a real link so it’s hard to imagine anyone clicking on it, however the answer lies in a feature of modern browsers called “DNS prefetching”. While the browsers parse the page if they come across something that looks like it might be clicked on later they pre-emptively retrieve the DNS for it so that if and when you do click on it you don’t have to wait for the DNS lookup to complete. This explains the fleeting nature of the traffic (who looks through old chat logs?) and tells us that we should not be worried… this time at least.
Can we confirm this hypothesis? Well there are other domains mentioned in the same room, do they show spikes when mentioned?
The answer is yes: taking this post as an example:
http://www.elitetradingtool.co.uk/ very useful and then we see the following:
(This domain actually exists so we see a low level of background with the large spike on top.)
We can also create similar events ourselves, for example when posting what one of us was eating at the time to a channel we were able to create the following signal:
(Other brands of yogurt are available!)
Nice! A false alarm this time fortunately but hopefully an interesting insight into some of the stranger traffic we see on our servers.