Quick Training Guide

Interested to see how this works, or what it really looks like?

The following video is the “welcome” training video to new users, and shows just how easy it is to crawl for expired domains.

In this off-the-cuff recording, I spent just 20 seconds starting a new crawl, which returned 61 expired domains. Many were good, and at least 3 were quality domains, with great Trust Flow and Majestic Flow ,with Trust Flow Topics closely related to the niche I was searching for.

 

Video transcript:

Welcome to PBN Lab!

In this video I’m going to give you a quick crash course on how to crawl for expired domains. We’ll start by setting up a new job, then we’ll have a quick look at the results page.

Once you log in after activating your account, you’ll be presented with the dashboard.

I think you’ll agree with me that dashboard is very bare, and somewhat minimalistic. That’ll change soon enough and the big space down the bottom here is where the live job stats will appear.

In the red panel you can see an overview of your crawl jobs, whether they’re running, queued, waiting or completed.

And in the black panel is a quick overview of the domains you’ve found.

Let’s get started by clicking Add New Job.

The first step in the process is giving the new job a name. It can be anything you like, and ideally should reflect what you’re about to search for – because you’ll see this job name in the results page.

In step 2, we’ll be adding a list of seed URLs, which is a list of distinct web sites or pages that the crawler will use to begin it’s crawl from.

There are 3 simple ways to load in a list of seed URLs.

The first and easiest is by using a keyword search, which utilises current data from the Google index, and we source and pay for that data legitimately using Google’s CSE API.

The second option allows you to manually enter (or paste) seed URLs, one URL per line.

The third option allows you to paste raw HTML, which we’ll parse for links programmatically. It’s a powerful, but more advanced method.

For this basic demo, I’ll use the keyword search, since it will deliver us great results with ease.

Enter the keywords you’d like to use – just like when you’re genuinely searching Google. You can put them in double quotes and include other semantic terms that the Google search recognises.

My niche site is a buying keyword for cycling shoes, so I’ll use the search phrase “bradley wiggins”, who was the Tour de France winner in 2012. We won’t get into WHAT to search for right now, because it’s a huge topic in itself.

By specifying a country, the Google API will return us results more specific to that geographic region – but keep in mind that it doesn’t mean they’ll necessarily be sites from that region.

We’ll leave that at the default of USA.

By specifying a year – the Google API will return the top results beginning that year. While it doesn’t always vary the results return greatly, it does at times help yield different results.

I’ll set that to 2012.

After the web service pulls data back from Google, we’ll get to see the list of seed URLs. You can click out and take a look at some of the pages, but take my word for it when I say doing that’s really not worth your time. Just let it rip and click next to queue the job.

If we head back to the dashboard now, we’ll see the live status of the current job.

There’s a lot of data here to look at as the crawler goes through the various stages of the process. Initially we’ll see the stats of the crawl itself, with the main points of interest here being the total URLs indexed, the crawl rate in URLs per minute and how much time is remaining.

After the crawl completes, very quickly it’ll establish a distinct list of domains from the broken URLs, then look them up and verify their availability. And of course the most important part, fetching the metrics for the available domains and returning all of the crucial data back here so you can review the results.

There’s nothing you need to do, it’s just showing you what it’s up to.

I’ll pause the recording now for the next few minutes, and then we’ll take a quick look at the job results view, and how we can use it to quickly review the domains we found in the crawl.

So we’re back, and after about 20 seconds of our own time setting up the job, and waiting maybe 10 minutes in total for the whole job to complete, I’ve got 71 domains.

The crawler indexed 304 thousand URLs, resulting in 7945 domains.

Let’s go check them out!

As we look at the results view, there’s quite a bit going on, but this is really dead simple.

At the top here we can select any job we’ve run in the past.

Looking at the data table itself, our first column here allows us to select multiple rows and then perform bulk actions like marking as favourite, or deleting.

The second column allows you to mark or unmark a domain as a favorite in a single click.

We then see the domain name itself, our Majestic Trust Flow and Citation Flow values, then the Moz Domain Authority and Page Authority.

By clicking on the row, it’ll expand out and show the Majestic backlink data, plus the all important trustflow topics.

We can also see the MozRank (essentially the Google PageRank value), plus the Moz link metrics.

And here we can see the URL that we found in the crawl, which we used to discover the expired domain and also the page we were crawling when we found the dead URL.

You can click out here to see the page where the dead link was found, which is nice to get a feel for the old site’s context.

Under the action button here, we have deep links into Moz, Ahref and Majestic – so in a single-click I can check out what other data those guys have about this domain.

I can also click over to Archive.org and Screenshots.com, hopefully they’ve indexed the old site…we might get to see exactly what the old site was, and can whether or not it was spammy, or a horrible little thin site from an old PBN.

I’m not going to sit here and analyze all of these domains, but I certainly like the look of francobritishcycleplan.org so I’ll mark it as a favorite.

So what’s going to happen when we have literally thousands of domains here to look at? It’ll be time to start filtering down to just the gold, and here’s exactly how you do that.

When we expand out the settings panel, you can see I can some basic filters here that allow me just to view domains I’ve marked as favorites, or domains I’ve previously trashed.

I can modify how many domains I’d like to see on each page of the data table, and how many domains to pull in in total.

I can set my default sort order for the data table, which by default is set to Moz DA. I personally prefer to look at Majestic TrustFlow first, so I’ll set that to Maj TF.

Under Moz Metrics, we can say we only want to see domains with a MozRank greater than 2, a Moz DA greater than 30, a PA greater than 30.

Under Majestic Metrics, let’s say we only want to see domains with a TF and CF greater than 15.

You could go nuts here, but that’ll do for now.

I can apply that filter now to the current Job, or by clicking Apply and Save, I can also save those settings as my default for every time I look at results view.

And now we’re just looking at domains that meet the criteria I set.

Because this is a brand new account and I’ve only done a couple of crawl jobs, there are only a couple of domains to see – but as you can imagine, once you’ve got a hundred crawls under your belt, it’s a really powerful but easy way to manage the domain list.

And so there you have it, crawling for expired domains can’t get any easier!

2 Comments

  1. Hi,

    I would like to know, whether the domains, which it returns from search results, can be registered immediately for normal price or these domains are part of auction and do we need to wait for the auction to get completed.

    Thanks !

    Reply
    • Hi Shahzan,

      The top-level domains like .com, .net, .info etc. that the crawler finds can be registered immediately at any registrar you like.

      The only domains that cannot are some of the country-specific domains I’ve added support for, as these are not presently validated against WHOIS or a registrar – so they do include some false-positives and domains waiting to drop. Please see this page for more information: http://www.pbnlab.com/more-info/domains-supported-by-the-expired-domain-crawler/

      Thanks,
      Scott

      Reply

Submit a Comment

Your email address will not be published. Required fields are marked *