Posted by bradfriedman

Hello Mozzers, and happy Monday!

My name is Brad Friedman, Technical Lead for Mozscape, and I'm happy to announce that we've released a brand new Mozscape index for April. You can find fresh, new data across all of our apps. Check out Open Site Explorer, the Mozbar, your PRO campaigns, and the Mozscape API.

We've reduced our index crawl time to just eleven days for this release! Thanks to our Big Data wizards on the processing team, Douglas Vojir and Martin York, for improving the freshness of our metrics! You can read more details on our technical improvements in this post from February.

We started processing this index on Wednesday, April 10, so the metrics will reflect crawl data from the end of March and the first week of April.

Here are the numbers for this latest index:

• 88,973,525,592 (88 billion) URLs
• 9,077,621,093 (9.1 billion) Subdomains
• 161,124,038 (161 million) Root Domains
• 887,067,310,285 (887 billion) Links
• Followed vs. Nofollowed
  • 2.15% of all links found were nofollowed
  • 56.0% of nofollowed links are internal
  • 44.0% are external
• Rel Canonical – 15.08% of all pages use a rel=canonical tag
• The average page has 76 links on it
  • 65.05 internal links
  • 11.02 external links

And these are the correlations with Google's US search results:

• Page Authority – 0.36
• Domain Authority – 0.19
• MozRank – 0.24
• Linking Root Domains – 0.30
• Total Links – 0.25
• External Links – 0.29

All this delicious data! What a great way to start off the week, huh?

Follow our planned update schedule on our Mozscape calendar, and you can check out the metrics on our previous releases here.

We're happy to answer your questions or read your feedback! Feel free to leave your comments here on this thread, or you can reach me on Twitter (@brad_friedman).

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

SEOmoz Daily SEO Blog

Posted by carinoverturf

It's that time again – the latest Mozscape index is now live! Data is now refreshed across all the SEOmoz applications – Open Site Explorer, the Mozbar, PRO campaigns, and the Mozscape API.

This index finished up in just 13 days, thanks again to all the improvements our Big Data Processing team has been implementing to make our Mozscape processing pipeline more efficient. The team continues to dial out our virtual private cloud in Virginia as well as tweak, tune, and improve the time it takes to process 82 billion URLs.

We've been saying we're close to releasing our first index created on our own hardware – and now we really are! Stay tuned for a deep dive blog post into why and how we built our own private cloud.

This index was kicked off the first week of March, so data in this index will span from late January through February, with a large percentage of crawl data from the last half of February.

Here are the metrics for this latest index:

• 83,122,215,182 (83 billion) URLs
• 12,140,091,376 (12.1 billion) Subdomains
• 141,967,157 (142 million) Root Domains
• 801,586,268,337 (802 billion) Links
• Followed vs. Nofollowed
  • 2.21% of all links found were nofollowed
  • 55.23% of nofollowed links are internal
  • 44.77% are external
• Rel Canonical – 15.70% of all pages now employ a rel=canonical tag
• The average page has 74 links on it
  • 63.56 internal links on average
  • 10.65 external links on average

And the following correlations with Google's US search results:

• Page Authority – 0.35
• Domain Authority – 0.19
• MozRank – 0.24
• Linking Root Domains – 0.30
• Total Links – 0.25
• External Links – 0.29

We always love to hear your thoughts! And remember, if you're ever curious about when Mozscape next updates, you can check the calendar here. We also maintain a list of previous index updates with metrics here.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

SEOmoz Daily SEO Blog

Posted by carinoverturf

Just 13 days ago on January 11th, we released the first Mozscape index for 2013. And today, we're launching the latest January Mozscape index – another two indexes in one month! Mozscape data has been refreshed across all our applications so you can see the latest data in Open Site Explorer, the Mozbar, PRO campaigns, and the Mozscape API.

• 70,278,347,012 (70 billion) URLs
• 1,516,212,211 (1.5 billion) Subdomains
• 145,518,352 (145 million) Root Domains
•  783,206,227,396 (783 billion) Links
• Followed vs. Nofollowed
• • 2.24% of all links found were nofollowed
  • 56.43% of nofollowed links are internal
  • 43.57% are external
• Rel Canonical – 15.11% of all pages now employ a rel=canonical tag
• The average page has 78 links on it
• •  66.68 internal links on average
  •  11.07 external links on average

• Page Authority – 0.36
• Domain Authority – 0.19
• MozRank – 0.24
• Linking Root Domains – 0.30
• Total Links – 0.25
• External Links – 0.29

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

SEOmoz Daily SEO Blog

Posted by carinoverturf

Today, we are releasing the latest Mozscape index – just three weeks after our last release on 12/21! Mozscape data has been refreshed across all our applications so you can see the latest data in Open Site Explorer, the Mozbar, PRO campaigns, and the Mozscape API.

The top focus of the Big Data team in the next couple months is to get our Mozscape index releases to be more frequent – consistently releasing every three weeks, and then, ultimately, every two weeks. We're utilizing both the high power compute AWS machines as well as our own virtual private cloud setup in our Virginia colocation. At this point, processing in AWS is still slightly faster than processing in our private cloud. However, our top notch Tech Ops team is working closely with us to fine tune our implementation of Open Stack, open source software that allows us to put a virtual layer on top of our fleet of hardware in Virginia. Our own super computers in Virginia should give us even more computing power than we've seen in AWS, meaning faster index processing and more frequent releases for you guys!

Here are the metrics for this latest index:

• 68,291,839,694 (68 billion) URLs
• 512,802,814 (512 million) Subdomains
• 96,918,414 (97 million) Root Domains
• 771,699,931,943 (771 billion) Links
• Followed vs. Nofollowed
  • 2.24% of all links found were nofollowed
  • 56.32% of nofollowed links are internal
  • 43.68% are external
• Rel Canonical – 11.39% of all pages now employ a rel=canonical tag
• The average page has 61 links on it
  •  51.68 internal links on average
  •  8.77 external links on average

And the following correlations with Google's US search results:

• Page Authority – 0.36
• Domain Authority – 0.19
• MozRank – 0.24
• Linking Root Domains – 0.30
• Total Links – 0.25
• External Links – 0.29

This index is a little bit smaller than the previous index, but fairly fresh with the oldest data being crawled late November and the freshest from January 1st. As you can see from the histogram, a pretty big portion was crawled mid- to late-December!

We always love to hear your thoughts! And remember, if you're ever curious about when Mozscape is updating, you can check the calendar here. We also maintain a list of previous index updates with metrics here.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

SEOmoz Daily SEO Blog

Posted by carinoverturf

Happy Holidays!! The December Mozscape index is now live! The latest index has just been released and you will see fresh Mozscape data in Open Site Explorer, the Mozbar, PRO campaigns, and the Mozscape API.

The Big Data team was hoping to provide a special holiday treat launching two indices in one month again, but, unfortunately processing was bitten by a full machine failure. We've had really good luck running Mozscape processing on the larger, high compute AWS machines, but, sadly, just a few days before the index was complete, an entire computing machine failed which forced us to have to re-run a few steps. Even with the failure, the December index is a few days earlier than our scheduled release date on December 27th – a pre-holiday treat for everyone!

In even bigger Big Data news – our private cloud is fully up and running in Virginia and we are about 25% done with our first production ready index! If all goes well, we'll be releasing the first Mozscape index created in our own private cloud in mid-January. What a way to bring in the new year!

Here are the metrics for this latest index:

• 78,671,787,078 (78 billion) URLs
• 687,827,137 (687 million) Subdomains
• 136,539,340 (136 million) Root Domains
• 917,094,026,686 (917 billion) Links
• Followed vs. Nofollowed
  • 2.32% of all links found were nofollowed
  • 56.69% of nofollowed links are internal
  • 43.31% are external
• Rel Canonical – 14.07% of all pages now employ a rel=canonical tag
• The average page has 72 links on it
  •  61.38 internal links on average
  •  10.45 external links on average

And the following correlations with Google's US search results:

• Page Authority – 0.36
• Domain Authority – 0.19
• MozRank – 0.24
• Linking Root Domains – 0.30
• Total Links – 0.25
• External Links – 0.29

The histogram for the freshness of the index's crawl data shows a pretty high volume of fresh crawl data coming from middle of November. This index will have data ranging as old as the end of October, but a large volume of the data was crawled from the middle to end of November. 

We'll be keeping an eye on things over the holiday, so send us your feedback – we always love to hear your thoughts! And remember, if you're ever curious about when Mozscape is updating, you can check the calendar here. We also maintain a list of previous index updates with metrics here.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

SEOmoz Daily SEO Blog