22

This is one that I've never really known how to do, and it's been nagging me from time to time for years. I've read all I could find, including all the answers here. None of them gives a real answer to the question. So please read carefully before flagging a duplicate; it is not.

dig, host, nslookup... none of them seems to be able to get what I'm after.

At most I can get pointers like ec2-xxx-xxx-xxx-xxx.us-east-2.compute.amazonaws.com.

But if I use this online tool, I get exactly what I'm looking for: every domain that resolves to the IP address given (or a hostname). In this case, it's a freemium service, so it'll only list the first few, but it works. In the following image, I use three domains that I own and serve from a VPS. They are totally unrelated, but they all appear here:

Enter image description here

What black magic are they using? How can we replicate it?

Peter Mortensen
  • 2,318
  • 5
  • 23
  • 24
Neithan Max
  • 333
  • 2
  • 9
  • 1
    Noted it doesn't work at all on IPv6 - too much data to index. And it also misses things that don't start in www. – Criggie Jun 05 '19 at 01:07
  • @Criggie: That's probably not the reason. If they are not using rDNS, but just bruteforcing regular domain→IP records as Michael said, then the number of "possible" addresses cannot actually affect the process in any way. (After all, why would they index and store addresses which have no domains?) More likely they just didn't bother. – user1686 Jun 05 '19 at 09:02

1 Answers1

47

It's brute force.

They have looked up the IP addresses of every domain name they can find, and then compiled the results into their own database.

Michael Hampton
  • 244,070
  • 43
  • 506
  • 972
  • 3
    The only part that is unclear to me is how they find a list of every domain name as there isn't a central list anywhere. Edit: Looks like they don't have everything because my website that has been up for years is not listed – Qwertie Jun 05 '19 at 04:38
  • 12
    @Qwertie: certificate transparency ledgers are a good place these days – practically everything that has a properly rooted TLS certificate will appear in there. – Ulrich Schwarz Jun 05 '19 at 06:18
  • 3
    @Qwertie : The Centralized Zone Data Service ( https://czds.icann.org/en ) could be a good starting point as well - it gives access to a large number of 'zone files' of the various top level domains. – Richy B. Jun 05 '19 at 09:43
  • @Qwertie Just like spammers get your e-mail address. Such people crawl everything they can find and add it to their database. And newly registered domains appear on some sites, so if you really want to crawl everything, you can just collect them there. You also have a comprehensive list of domains on your hard drive right now, in the HSTS-preload list of your browser. See for example this [changeset]](https://hg.mozilla.org/mozilla-central/rev/dc97b02f1f71) at mozilla-central with a lot of domains added and removed from the preload list. Neither your domain nor its dns records are a secret. – allo Jun 05 '19 at 11:06
  • 1
    @Qwertie Registrars are exactly a centralized list of all domains... the only issue is that it's not just one of them and also not all of them give easily access to their zone files so access might not be completely free or immediate. – Giacomo Alzetta Jun 05 '19 at 12:03
  • 5
    @GiacomoAlzetta no, registrars at most have a list of domains they sponsor, not all domains in all TLDs. registries on the contrary obviously have a list of all the domains they maintain, which is available in gTLDs through open access to zonefiles like Richy B. said (but noting that not all registered domain is published so some will be missing). Some others TLDs, like .FR have "open data" initiative where you can get access to many things, including list of domains. Other registries publish daily "newly registered domain names list". – Patrick Mevzek Jun 05 '19 at 15:33
  • Brute force indeed but with the caveat that the results can become stale quite fast. And with CDN and load balancers a given name can resolve to many IPs, and a changing list of IPs during time... – Patrick Mevzek Jun 05 '19 at 15:35
  • 1
    They probably have another part to their business that involves a DNS server. Each time a user accesses a domain, they log the result of the lookup, and have spun it off into a separate service offering. Not sure if its still the case, but a few years ago some of the big analytics firms offered web developers browser toolbars to show stats of the websites they visited. These had the added feature of recording every site you visited and fed the DNS results back to them. – Gavin Coates Jun 06 '19 at 10:56
  • @UlrichSchwarz they don't use certificate transparency, because they cant find the domains on multiple IPs I use where all used domains have certificates in the logs! – Josef Jun 06 '19 at 11:16
  • actually at least not the full story, I just tried this and found a couple of my domains that would point to different IP addresses. My guess is that they are also parsing and correlating whois informations. – PlasmaHH Jun 06 '19 at 13:26
  • Funnily, this website returns absolutely nothing for all the public IP I'm using at my company, though `host` returns perfectly valid results :) – wazoox Jul 01 '19 at 17:33