https://susam.net/blog/sinkholed.html

Sinkholed: A DNS Horror Story

By Susam Pal on 03 Dec 2019

Introduction 

On 26 Nov 2019 at 14:55 UTC, I logged into my server that hosts my
website to perform a simple maintenance activity. Merely three
minutes later, at 14:58 UTC, the domain name susam.in used to host
this website was transferred to another registrant without any
authorization by me or without any notification sent to me. Since the
DNS results for this domain name was cached on my system, I was
unaware of this issue at that time. It would take me three days to
realize that I had lost control of the domain name I had been using
for my website for the last 12 years. This blog post documents when
this happened, how this happened, and what it took to regain control
of this domain name.

Contents 

  * Introduction
  * Domain Name Transfer
  * Avalanche Botnet
  * Support Ticket
  * Tweets and Retweets
  * Domain Name Return
  * Conclusion

Domain Name Transfer 

On 29 Nov 2019 at 19:00 UTC, when I visited my website hosted at
https://susam.in/, I found that a zero-byte file was being served at
this URL. My website was missing. In fact, the domain name resolved
to an IPv4 address I was unfamiliar with. It did not resolve to the
address of my Linode server anymore.

I checked the WHOIS records for this domain name. To my astonishment,
I found that I was no longer the registrant of this domain. An entity
named The Verden Public Prosecutor's Office was the new registrant of
this domain. The WHOIS records showed that the domain name was
transferred to this organization on 26 Nov 2019 at 14:58 UTC, merely
three minutes after I had performed my maintenance activity on the
same day. Here is a snippet of the WHOIS records that I found:

Domain Name: susam.in
Registry Domain ID: D2514002-IN
Registrar WHOIS Server:
Registrar URL:
Updated Date: 2019-11-26T14:58:00Z
Creation Date: 2007-05-15T07:19:26Z
Registry Expiry Date: 2020-05-15T07:19:26Z
Registrar: NIXI Special Projects
Registrar IANA ID: 700066
Registrar Abuse Contact Email:
Registrar Abuse Contact Phone:
Domain Status: clientTransferProhibited http://www.icann.org/epp#clientTransferProhibited
Domain Status: serverRenewProhibited http://www.icann.org/epp#serverRenewProhibited
Domain Status: serverDeleteProhibited http://www.icann.org/epp#serverDeleteProhibited
Domain Status: serverUpdateProhibited http://www.icann.org/epp#serverUpdateProhibited
Domain Status: serverTransferProhibited http://www.icann.org/epp#serverTransferProhibited
Registry Registrant ID:
Registrant Name:
Registrant Organization: The Verden Public Prosecutor's Office
Registrant Street:
Registrant Street:
Registrant Street:
Registrant City:
Registrant State/Province: Niedersachsen
...
Name Server: sc-c.sinkhole.shadowserver.org
Name Server: sc-d.sinkhole.shadowserver.org
Name Server: sc-a.sinkhole.shadowserver.org
Name Server: sc-b.sinkhole.shadowserver.org
...

The ellipsis denotes some records I have omitted for the sake of
brevity. There were three things that stood out in these records:

 1. The registrar was changed from eNom, Inc. to NIXI Special
    Projects.
 2. The registrant was changed from Susam Pal to The Verden Public
    Prosecutor's Office.
 3. The name servers were changed from Linode's servers to
    Shadowserver's sinkholes.

Avalanche Botnet 

On searching more about the new registrant on the web, I realized
that it was a German criminal justice body that was involved in the
takedown of the Avalanche malware-hosting network. It took a
four-year concerted effort by INTERPOL, Europol, the Shadowserver
Foundation, Eurojust, the Luneberg Police, and several other
international organizations to finally destroy the Avalanche botnet
on 30 Nov 2016. In this list of organizations, one name caught my
attention immediately: The Shadowserver Foundation. The WHOIS name
server records pointed to Shadowserver's sinkholes.

The fact that the domain name was transferred to another organization
merely three minutes after I had performed a simple maintenance
activity got me worried. Was the domain name hijacked? Did my
maintenance activity on the server have anything to do with it? What
kind of attack one might have pulled off to hijack the domain name? I
checked all the logs and there was no evidence that anyone other than
me had logged into the server or executed any commands or code on it.
Further, a domain name transfer usually involves email notification
and authorization. None of that had happened. It increasingly looked
like that the three minute interval between the maintenance activity
and the domain name transfer was merely a coincidence.

More questions sprang up as I thought about it. The Avalanche botnet
was destroyed in 2016. What has that got to do with the domain name
being transferred in 2019? Did my server somehow become part of the
Avalanche botnet? My server ran a minimal installation of the latest
Debian GNU/Linux system. It was always kept up-to-date to minimize
the risk of malware infection or security breach. It hosted a static
website composed of static HTML files served with Nginx. I found no
evidence of unauthorized access of my server while inspecting the
logs. I could not find any malware on the system.

The presence of Shadowserver sinkhole name servers in the WHOIS
records was a good clue. Sinkholing of a domain name can be done both
malicously as well as constructively. In this case, it looked like
the Shadowserver Foundation intended to sinkhole the domain name
constructively, so that any malware client trying to connect to my
server nefariously would end up connecting to a sinkhole address
instead. My domain name was sinkholed! The question now was: Why was
it sinkholed?

Support Ticket 

On 29 Nov 2019 at 19:29 UTC, I submitted a support ticket to
Namecheap to report this issue. At 21:05 UTC, I received a response
from Namecheap support that they have contacted Enom, their upstream
registrar, to discuss the issue. There was no estimate for when a
resolution might be available.

At 21:21 UTC, I submitted a domain name transfer complaint to the
Internet Corporation for Assigned Names and Numbers (ICANN). I was
not expecting any response from ICANN because they do not have any
contractual authority on a country code top-level domain (ccTLD).

At 21:23 UTC, I emailed National Internet Exchange of India (NIXI).
NIXI is the ccTLD manager for .IN domain and they have authority on
it. I found their contact details from the IANA Delegation Record for
.IN. Again, I was not expecting a response from NIXI because they do
not have any contractual relationship directly with me. They have a
contractual relationship with Namecheap, so any communication from
them would be received by Namecheap and Namecheap would have to pass
that on to me.

At 21:30 UTC, ICANN responded and said that I should contact the
ccTLD manager directly. Like I explained in the previous paragraph, I
had already done that, so there was nothing more for me to do except
wait for Namecheap to provide an update after their investigation. By
the way, NIXI never replied to my email.

Tweets and Retweets 

On 30 Nov 2019 at 07:30 UTC, I shared this issue on Twitter. I was
hoping that someone who had been through a similar experience could
offer some advice. In fact, soon after I posted the tweet, a kind
person named Max from Germany generously offered to help by writing a
letter in German addressed to the new registrant which was a German
organization. The reason for sinkholing my domain name was still
unclear. I hoped that with enough number of retweets someone closer
to the source of truth could shed some light on why and how this
happened.

At 09:54 UTC, Richard Kirkendall, founder and CEO of Namecheap,
responded to my tweet and informed that they were contacting NIXI
regarding the issue. This seemed like a good step towards resolution.
After all, the domain name was no longer under their upstream
registrar named Enom. The domain name was now with NIXI as evident
from the WHOIS records.

Several other users tweeted about my issue, added more information
about what might have happened, and retweeted my tweet.

On 1 Dec 2019 at 11:48 UTC, Benedict Addis from the Shadowserver
Foundation contacted me by email. He said that they had begun looking
into this issue as soon as one of the tweets about this issue had
referred to their organization. He explained in his email that my
domain name was sinkholed accidentally as part of their Avalanche
operation. Although it is now three years since the initial takedown
of the botnet, they still see over 3.5 million unique IP addresses
connecting to their sinkholes everyday. Unfortunately, their
operation inadvertently flagged my domain name as one of the domain
names to be sinkholed because it matched the pattern of command and
control (C2) domain names generated by a malware family named Nymaim,
one of the malware families hosted on Avalanche. Although, they had
validity checks to avoid sinkholing false-positives, my domain name
unfortunately slipped through those checks. Benedict mentioned that
he had just raised this issue with NIXI and requested them to return
the domain name to me as soon as possible.

Domain Name Return 

On 2 Dec 2019 at 04:00 UTC, when I looked up the WHOIS records for
the domain name, I found that it had been returned to me already. At
08:37 UTC, Namecheap support responded to my support ticket to say
that they had been informed that NIXI had returned the domain name to
its original state. At 09:55 UTC, Juliya Zinovjeva, Domains Product
Manager of Namecheap, commented on Twitter and confirmed the same
thing.

Conclusion 

Despite the successful resolution, it was still quite unsettling that
a domain name could be transferred to another registrant and
sinkholed for some perceived violation. I thought there would be more
checks in place to confirm that a perceived violation was real before
a domain name could be transferred. Losing a domain name I have been
using actively for 12 years was an unpleasant experience. Losing a
domain name accidentally should have been a lot harder than this.
Benedict from the Shadowserver Foundation assured me that my domain
name would be excluded from future sinkholing for this particular
case. However, the possibility that this could happen again due to
another unrelated operation by another organization in future is
disconcerting.

I also wondered if a domain name under a country code top-level
domain (ccTLD) like .in is more susceptible to this kind of
sinkholing than a domain name under a generic top-level domain (gTLD)
like .com. I asked Benedict if it is worth migrating my website from
.in to .com. He replied that in his personal opinion, NIXI runs an
excellent, clean registry, and are very responsive in resolving
issues when they arise. He also added that domain generation
algorithms (DGAs) of malware are equally, and possibly more,
problematic for .com domains. He advised against migrating my
website.

Thanks to everyone who retweeted my tweet on this issue. Also, thanks
to Richard Kirkendall (CEO of Namecheap), Namecheap support team, and
Benedict Addis from the Shadowserver Foundation for contacting NIXI
to resolve this issue.

Update on 06 Jan 2022: Nearly two years after this incident, I
eventually moved this website to susam.net. The decision to do so was
not influenced by this incident. I wanted this domain name since 2006
but it was unavailable back then. This domain name became available
only very recently and I moved my website to this domain as soon as
it became available.

Comments
---------------------------------------------------------------------
Home Blog Feed Subscribe About GitHub Mastodon

(c) 2001-2023 Susam Pal