Blog, DomainTools Blog
Using Iris Investigate Pivot Engine to Collect Bulk Screenshots | DomainTools
Using Iris Investigate Pivot Engine to Collect Bulk Screenshots
**This article is originally published by our partner, DomainTools. Click here to view the original article.
In this blog post, we’ll first talk about using Iris Investigate to collect screenshots one domain at a time manually. You can quickly and efficiently queue hundreds of parts for bulk screenshots with Iris Investigate’s pivot engine.
Manual Domain-by-Domain Screenshots in Iris Investigate
A screenshot may have previously been collected if you look at the individual registered domains in Iris Investigate. You’ll see the most recent screenshot as part of the default Iris Investigate display when that’s the case. For example:
Suppose your focus is primarily on screenshots (rather than all aspects of a domain name). In that case, you can tailor your interface by closing the two default panes and selecting “Screenshot History” from the menu at the bottom of the page. This will result in screenshots receiving a more prominent display. See the following two screenshots:
Our interface now looks like Illustration 3 below. Now Iris Investigate is “all about screenshots:”
Part of the power of the Iris Investigate screenshot capability is:
- If it’s been some time since a screenshot was collected for a site, you can click “Queue Screenshot for Update” to request that a site’s screenshot be refreshed.
- Alternatively, suppose you want to see what the site previously looked like. In that case, you can browse the historical screenshots that have already been collected (see the screenshot history panel on the left-hand side of your screen). Click on one of the earlier historical screenshots if you’d like to inspect it more closely.
Once you’re done with the current site, and you’re ready to move on to a new place, just enter the new site’s name in the search bar and hit the blue search button:
Taking Screenshots for Hundreds of Sites with Iris Investigate’s Pivot Engine
Let’s assume, for example, that we’re interested in “Native Sovereign Nation” (NSN) dot gov websites, a program announced 20 years ago.
For example, perhaps a history-of-the-Internet researcher is curious if those sites were popular, well-accepted, and are still used and relied on today; or have more flexibly-named sites in some other top-level domain (TLD) largely supplanted the NSN dot gov program? Or are there differences by region of the country? We’re not going to do that study in this article, but we will show you how a researcher could collect screenshots for all the NSN dot gov websites.
We can get a list of all current 8,149 dot gov domains (including both NSN domains and other dot gov domains) by saying:
$ wget https://raw.githubusercontent.com/cisagov/dotgov-data/main/current-full.csv
$ wc -l current-full.csv
8150 <-- includes a "header" row that doesn't count.
Looking at the 2nd field in that file, we can see the breakdown by type of entity:
$ cut -d, -f2 < current-full.csv | sort | uniq -c | sort -nr
3888 City
1406 County
1161 Federal - Executive
1145 State
219 Tribal
166 Independent Intrastate
117 Federal - Legislative
25 Federal - Judicial
22 Interstate
1 Domain Type
Our current hypothetical study focuses on just the 219 Tribal NSN dot-gov domains. We can extract those by saying:
$ grep "Tribal" current-full.csv | cut -d, -f1 > nsn-subset.txt
$ wc -l nsn-subset.txt
219
We’re ready to run screenshots for that subset of dot gov sites. To do so, hit the “Advanced” button in Iris Investigate:
The Advanced Search Panel will then open. Select the “in” operator from the pull-down (as highlighted in the following screenshot). Ensure you’re searching for “Domains” in the left-hand pull-down, as we are here. Then, cut and paste our list of NSN dot gov domains where indicated. Hit the blue search button to find matching domains.
When that search finishes, select the Pivot Engine tab on the black bar at the bottom of the screen (highlighted in red below):
You’ll then see a display like the following. Note the little checkbox on the blue menu bar (highlighted in red). Check that box.
Checking the box will select ALL the domains shown in the table. (If any of those domains you DON’T want to screenshot, you can scroll down and unclick those names as exceptions).
Once you’ve selected the set of domains, a new bar will appear (highlighted in red in the following). Click “Queue Screenshots” (over on the right side of that bar) to proceed.
When the domains have been queued for updated screenshots, you should see a confirmation as highlighted in red below:
Screenshots will then be processed asynchronously. After a bit, screenshots for your sites will typically be ready to view.
However, if DomainTools tried screenshotting the specified site only to find that nothing has changed since the previous successful screenshot, “duplicate detection” kicks in. In that case, you’ll still see the last shot screen (including the original date associated with that collection).
That doesn’t mean that your request wasn’t run! Instead, if you roll over the top entry in the screenshot history, you’ll see a popup explaining when the most recent screenshot was collected and checked, even if the primary screenshot date shown hasn’t changed.
Iris Investigate
Combines enterprise-grade domain intelligence and proactive risk scoring with industry-leading passive DNS data to guide threat investigations and uncover connected infrastructure.
Free DomainTools Information Kit
Get first-hand information and use cases to discover how DomainTools can empower your organization with the world’s largest DNS Dataset
Book a Free Consultation
Discover DomainTools Solutions and learn how they help security analysts turn threat data into threat intelligence