SEO Blog

Posts Tagged ‘Live’


Mozscape’s Third May 2013 Index Is Now Live

Posted by:  /  Tags: , , , ,

Posted by bradfriedman

It has been an amazing month at Moz, and for the first time, we have served up three indexes in one month – holy fresh data, Batman! This index is also our second index release from our Virginia data center.

As we’ve become so regular with our index releases, it seems that a blog post announcing each release may not be the best way to inform our customers on releases as they happen. This will be the last index update announced on the blog, but index release readers, have no fear! There’s a new, more digestible way to get all of the information about the index updates you know and love.

Going forward, we invite you subscribe to the Mozscape RSS feed found on the lower right-hand corner of the http://moz.com/products/api/updates page so that you will know when we have released a new index. Simply click on the RSS feed icon… 

…and you’ll be directed to our Feedburner Mozscape updates page. This page makes it easy to enter your information to subscribe to our index updates: 

Just enter your email address, choose your reader, and let the Mozscape index data flow in! 

As our index releases continue to become more frequent, you can expect the number of updates received through the RSS to mount. We hope you enjoy this new way to consume fresh update info as much as we do.


Here are the metrics for this index:

  • 85,870,573,626 (86 billion) URLs
  •   5,524,096,501 (5.5 billion) Subdomains
  • 155,443,706 (155 million) Root Domains
  • 902,845,046,889  (903 billion) Links
  • Followed vs. Nofollowed
    •   2.17 % of all links found were nofollowed
    • 57.32 % of nofollowed links are internal
    • 42.68 % are external
  • Rel Canonical –  14.44 % of all pages now employ a rel=canonical tag
  • The average page has 79 links on it
    •  67.836 internal links on average
    •  11.13 external links on average

And the correlations with Google’s US search results:

  • Page Authority – 0.36
  • Domain Authority – 0.19
  • MozRank – 0.24
  • Linking Root Domains – 0.30
  • Total Links – 0.25
  • External Links – 0.29

This histogram shows the crawl date and freshness of results in this index:

This index spans 38 days with crawl starting April 1st and finishing on May 8th.  

We always love to hear your thoughts! And remember, if you’re ever curious about when Mozscape next index release is planned, check out http://moz.com/products/api/updates.

Thanks for reading, and have a Mozzy weekend!

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!


Moz Blog

April Mozscape Index Is Live

Posted by:  /  Tags: , , ,

Posted by bradfriedman

Hello Mozzers, and happy Monday!

My name is Brad Friedman, Technical Lead for Mozscape, and I'm happy to announce that we've released a brand new Mozscape index for April. You can find fresh, new data across all of our apps. Check out Open Site Explorer, the Mozbar, your PRO campaigns, and the Mozscape API.

We've reduced our index crawl time to just eleven days for this release! Thanks to our Big Data wizards on the processing team, Douglas Vojir and Martin York, for improving the freshness of our metrics! You can read more details on our technical improvements in this post from February.

We started processing this index on Wednesday, April 10, so the metrics will reflect crawl data from the end of March and the first week of April.

Here are the numbers for this latest index:

  • 88,973,525,592 (88 billion) URLs
  • 9,077,621,093 (9.1 billion) Subdomains
  • 161,124,038 (161 million) Root Domains
  • 887,067,310,285 (887 billion) Links
  • Followed vs. Nofollowed
    • 2.15% of all links found were nofollowed
    • 56.0% of nofollowed links are internal
    • 44.0% are external
  • Rel Canonical – 15.08% of all pages use a rel=canonical tag
  • The average page has 76 links on it
    • 65.05 internal links
    • 11.02 external links

And these are the correlations with Google's US search results:

  • Page Authority – 0.36
  • Domain Authority – 0.19
  • MozRank – 0.24
  • Linking Root Domains – 0.30
  • Total Links – 0.25
  • External Links – 0.29

Crawl histogram for the April Mozscape index

All this delicious data! What a great way to start off the week, huh?

Follow our planned update schedule on our Mozscape calendar, and you can check out the metrics on our previous releases here.

We're happy to answer your questions or read your feedback! Feel free to leave your comments here on this thread, or you can reach me on Twitter (@brad_friedman).

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!


SEOmoz Daily SEO Blog

January Mozscape Index is Live!

Posted by:  /  Tags: , , ,

Posted by carinoverturf

Today, we are releasing the latest Mozscape index – just three weeks after our last release on 12/21! Mozscape data has been refreshed across all our applications so you can see the latest data in Open Site Explorer, the MozbarPRO campaigns, and the Mozscape API.

The top focus of the Big Data team in the next couple months is to get our Mozscape index releases to be more frequent – consistently releasing every three weeks, and then, ultimately, every two weeks. We're utilizing both the high power compute AWS machines as well as our own virtual private cloud setup in our Virginia colocation. At this point, processing in AWS is still slightly faster than processing in our private cloud. However, our top notch Tech Ops team is working closely with us to fine tune our implementation of Open Stack, open source software that allows us to put a virtual layer on top of our fleet of hardware in Virginia. Our own super computers in Virginia should give us even more computing power than we've seen in AWS, meaning faster index processing and more frequent releases for you guys!

Here are the metrics for this latest index:

  • 68,291,839,694 (68 billion) URLs
  • 512,802,814 (512 million) Subdomains
  • 96,918,414 (97 million) Root Domains
  • 771,699,931,943 (771 billion) Links
  • Followed vs. Nofollowed
    • 2.24% of all links found were nofollowed
    • 56.32% of nofollowed links are internal
    • 43.68% are external
  • Rel Canonical – 11.39% of all pages now employ a rel=canonical tag
  • The average page has 61 links on it
    •  51.68 internal links on average
    •  8.77 external links on average

And the following correlations with Google's US search results:

  • Page Authority – 0.36
  • Domain Authority – 0.19
  • MozRank – 0.24
  • Linking Root Domains – 0.30
  • Total Links – 0.25
  • External Links – 0.29

Crawl histogram for the January Mozscape index

This index is a little bit smaller than the previous index, but fairly fresh with the oldest data being crawled late November and the freshest from January 1st. As you can see from the histogram, a pretty big portion was crawled mid- to late-December!

We always love to hear your thoughts! And remember, if you're ever curious about when Mozscape is updating, you can check the calendar here. We also maintain a list of previous index updates with metrics here.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!


SEOmoz Daily SEO Blog

December Mozscape Index is Live!

Posted by:  /  Tags: , , ,

Posted by carinoverturf

Happy Holidays!! The December Mozscape index is now live! The latest index has just been released and you will see fresh Mozscape data in Open Site Explorer, the MozbarPRO campaigns, and the Mozscape API.

The Big Data team was hoping to provide a special holiday treat launching two indices in one month again, but, unfortunately processing was bitten by a full machine failure. We've had really good luck running Mozscape processing on the larger, high compute AWS machines, but, sadly, just a few days before the index was complete, an entire computing machine failed which forced us to have to re-run a few steps. Even with the failure, the December index is a few days earlier than our scheduled release date on December 27th – a pre-holiday treat for everyone!

In even bigger Big Data news – our private cloud is fully up and running in Virginia and we are about 25% done with our first production ready index! If all goes well, we'll be releasing the first Mozscape index created in our own private cloud in mid-January. What a way to bring in the new year!

Here are the metrics for this latest index:

  • 78,671,787,078 (78 billion) URLs
  • 687,827,137 (687 million) Subdomains
  • 136,539,340 (136 million) Root Domains
  • 917,094,026,686 (917 billion) Links
  • Followed vs. Nofollowed
    • 2.32% of all links found were nofollowed
    • 56.69% of nofollowed links are internal
    • 43.31% are external
  • Rel Canonical – 14.07% of all pages now employ a rel=canonical tag
  • The average page has 72 links on it
    •  61.38 internal links on average
    •  10.45 external links on average

And the following correlations with Google's US search results:

  • Page Authority – 0.36
  • Domain Authority – 0.19
  • MozRank – 0.24
  • Linking Root Domains – 0.30
  • Total Links – 0.25
  • External Links – 0.29

The histogram for the freshness of the index's crawl data shows a pretty high volume of fresh crawl data coming from middle of November. This index will have data ranging as old as the end of October, but a large volume of the data was crawled from the middle to end of November. 

We'll be keeping an eye on things over the holiday, so send us your feedback – we always love to hear your thoughts! And remember, if you're ever curious about when Mozscape is updating, you can check the calendar here. We also maintain a list of previous index updates with metrics here.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!


SEOmoz Daily SEO Blog