Specialized Search Engines

ViewDNS.info, Threat Intelligence Platform, Censys.io, Recon.ng, Maltego

Beyond the standard WHOIS and DNS query tools that we covered previously, there are third parties that offer paid services for historical WHOIS data. One example is WHOIS history, which provides a history of WHOIS data and can come in handy if the domain registrant didn’t use WHOIS privacy when they registered the domain.

There are a handful of websites that offer advanced DNS services that are free to use. Some of these websites offer rich functionality. For now, we'll focus on key DNS related aspects. We will consider the following:

ViewDNS.info

Threat Intelligence Platform

ViewDNS.info offers Reverse IP Lookup. Initially, each web server would use one or more IP addresses; however, today, it is common to come across shared hosting servers. With shared hosting, one IP address is shared among many different web servers with different domain names. With reverse IP lookup, starting from a domain name or an IP address, you can find the other domain names using a specific IP address(es).

In the figure below, we used reverse IP lookup to find other servers sharing the same IP addresses used by cafe.thmredteam.com. Therefore, it is important to note that knowing the IP address does not necessarily lead to a single website.

Threat Intelligence Platform requires you to provide a domain name or an IP address, and it will launch a series of tests from malware checks to WHOIS and DNS queries. The WHOIS and DNS results are similar to the results we would get using whois and dig, but Threat Intelligence Platform presents them in a more readable and visually appealing way. There is extra information that we get with our report. For instance, after we look up thmredteam.com, we see that Name Server (NS) records were resolved to their respective IPv4 and IPv6 addresses, as shown in the figure below.

Censys

Censys Search can provide a lot of information about IP addresses and domains. In this example, we look up one of the IP addresses that cafe.thmredteam.com resolves to. We can easily infer that the IP address we looked up belongs to Cloudflare. We can see information related to ports 80 and 443, among others; however, it's clear that this IP address is used to server websites other than cafe.thmredteam.com. In other words, this IP address belongs to a company other than our client, Organic Cafe. It's critical to make this distinction so that we don’t probe systems outside the scope of our contract.

Recon-Ng

Recon-ng is a framework that helps automate the OSINT work. It uses modules from various authors and provides a multitude of functionality. Some modules require keys to work; the key allows the module to query the related online API. In this task, I will demonstrate using Recon-ng in the terminal.

From a penetration testing and red team point of view, Recon-ng can be used to find various bits and pieces of information that can aid in an operation or OSINT task. All the data collected is automatically saved in the database related to your workspace. For instance, you might discover host addresses to later port-scan or collect contact email addresses for phishing attacks.

using recon-ng to create a workspaces called redteamtest

In reconnaissance, you are starting with one piece of information and transforming it into new pieces of information. For instance, you might start your research with a company name and use that to discover the domain name(s), contacts and profiles. Then you would use the new information you obtained to transform it further and learn more about your target.

Let’s consider the case where we know the target's domain name, thmredteam.com, and we would like to feed it into the Recon-ng database related to the active workspace. If we want to check the names of the tables in our database, we can run db schema.

We want to insert the domain name thmredteam.com into the domains table. We can do this using the command db insert domains.

Recon-ng Marketplace

We have a domain name, so a logical next step would be to search for a module that transforms domains into other types of information. Assuming we are starting from a fresh installation of Recon-ng, we will search for suitable modules from the marketplace.

Before you install modules using the marketplace, these are some useful commands related to marketplace usage:

marketplace search KEYWORD to search for available modules with keyword.

marketplace info MODULE to provide information about the module in question.

marketplace install MODULE to install the specified module into Recon-ng.

marketplace remove MODULE to uninstall the specified module.

The modules are grouped under multiple categories, such as discovery, import, recon and reporting. Moreover, recon is also divided into many subcategories depending on the transform type. Run marketplace search to get a list of all available modules.

In the terminal below, we search for modules containing domains.

We notice many subcategories under recon, such as domains-companies, domains-contacts, and domains-hosts. This naming tells us what kind of new information we will get from that transformation. For instance, domains-hosts means that the module will find hosts related to the provided domain.

Some modules, like whoxy_whois, require a key, as we can tell from the * under the K column. This requirement indicates that this module is not usable unless we have a key to use the related service.

Other modules have dependencies, indicated by a * under the D column. Dependencies show that third-party Python libraries might be necessary to use the related module.

Let’s say that you are interested in recon/domains-hosts/google_site_web. To learn more about any particular module, you can use the command marketplace info MODULE; this is an essential command that explains what the module does. For example, marketplace info google_site_web provides the following description: “Harvests hosts from Google.com by using the ‘site’ search operator. Updates the ‘hosts’ table with the results.” In other words, this module will use the Google search engine and the “site” operator.

We can install the module we want with the command marketplace install MODULE, for example, marketplace install google_site_web.

Maltego

Maltego is an application that blends mind-mapping with OSINT. In general, you would start with a domain name, company name, person’s name, email address, etc. Then you can let this piece of information go through various transforms.

The information collected in Maltego can be used for later stages. For instance, company information, contact names, and email addresses collected can be used to create very legitimate-looking phishing emails.

Think of each block on a Maltego graph as an entity. An entity can have values to describe it. In Maltego’s terminology, a transform is a piece of code that would query an API to retrieve information related to a specific entity. The logic is shown in the figure below. Information related to an entity goes via a transform to return zero or more entities.

It is crucial to mention that some of the transforms available in Maltego might actively connect to the target system. Therefore, it is better to know how the transform works before using it if you want to limit yourself to passive reconnaissance.

Every transform might lead to several new values. For instance, if we start from the “DNS Name” cafe.thmredteam.com, we expect to get new kinds of entities based on the transform we use. For instance, “To IP Address” is expected to return IP addresses as shown next.

One way to achieve this on Maltego is to right-click on the “DNS Name” cafe.thmredteam.com and choose:

Standard Transforms

Resolve to IP

To IP Address (DNS)

After executing this transform, we would get one or more IP addresses, as shown below.

Then we can choose to apply another transform for one of the IP addresses. Consider the following transform:

DNS from IP

To DNS Name from passive DNS (Robtex)

This transform will populate our graph with new DNS names. With a couple more clicks, you can get the location of the IP address, and so on. The result might be similar to the image below.

The above two examples should give you an idea of the workflow using Maltego. You can observe that all the work is based on transforms, and Maltego will help you keep your graph organized. You would get the same results by querying the different online websites and databases; however, Maltego helps you get all the information you need with a few clicks.

We experimented with whois and nslookup in a previous task. You get plenty of information, from names and email addresses to IP addresses. The results of whois and nslookup are shown visually in the following Maltego graph. Interestingly, Maltego transforms were able to extract and arrange the information returned from the WHOIS database. Although the returned email addresses are not helpful due to privacy protection, it is worth seeing how Maltego can extract such information and how it's presented.

Now that we have learned how Maltego’s power stems from its transforms, the only logical thing is to make Maltego more powerful by adding new Transforms. Transforms are usually grouped into different categories based on data type, pricing, and target audience. Although many transforms can be used using Maltego Community Edition and free transforms, other transforms require a paid subscription.