What are the incentives to publish high-value content to the web?

Search engines, like Google, say they want to index quality content, but provide little incentive to create and publish it. The reality is that the publishing environment is risky, relatively poorly paid in most instances, and is constantly being undermined.

The Pact

There is little point publishing web content if the cost of publishing outweighs any profit that can be derived from it.

Many publishers, who have search engines in mind, work on an assumption that if they provide content to everyone, including Google, for free, then Google should provide traffic in return. It’s not an official deal, of course. It’s unspoken.

Rightly or wrongly, that’s the “deal” as many webmasters perceive it.

What Actually Happens

Search engines take your information and, if your information is judged sufficiently worthy that day, as the result of an ever-changing, obscure digital editorial mechanism known only to themselves, they will rank you highly, and you’ll receive traffic in return for your efforts.

That may all change tomorrow, of course.

What might also happen is that they could grab your information, amalgamate it, rank you further down the page, and use your information to keep visitors on their own properties.

Look at the case of Trip Advisor. Trip Advisor, frustrated with Google’s use of its travel and review data, filed a competition complaint against Google in 2012.

The company said: “We hope that the commission takes prompt corrective action to ensure a healthy and competitive online environment that will foster innovation across the internet.”

The commission has been investigating more than a dozen complaints against Google from rivals, including Microsoft, since November 2010, looking at claims that it discriminates against other services in its search results and manipulates them to promote its own products.

TripAdvisor’s hotel and restaurants review site competes with Google Places, which provides reviews and listings of local businesses.”We continue to see them putting Google Places results higher in the search results – higher on the page than other natural search results,” said Adam Medros, TripAdvisor’s vice president for product, in February. “What we are constantly vigilant about is that Google treats relevant content fairly.”

Similarly, newspapers have taken aim at Google and other search engines for aggregating their content, and deriving value from that aggregation, but the newspapers claim they aren’t making enough to cover the cost of producing that content in the first place:

In 2009 Rupert Murdoch called Google and other search engines “content kleptomaniacs”. Now cash-strapped newspapers want to put legal pressure on what they see as parasitical news aggregators.”

Of course, it’s not entirely the fault of search engines that newspapers are in decline. Their own aggregation model – bundling news, sport, lifestyle, classifieds topics – into one “place” has been surpassed.

Search engines often change their stance without warning, or can be cryptic about their intentions, often to the determent of content creators. For example, Google has stated they see ads as helpful, useful and informative:

In his argument, Cutts said, “We actually think our ads can be as helpful as the search results in some cases. And no, that’s not a new attitude.”

And again:

we firmly believe that ads can provide useful information

And again:

In entering the advertising market, Google tested our belief that highly relevant advertising can be as useful as search results or other forms of content

However, business models built around the ads as content idea, such as Suite101.com, got hammered. Google could argue these sites went too far, and that they are asserting editorial control, and that may be true, but such cases highlight the flaky and precarious nature of the search ecosystem as far as publishers are concerned. One day, what you’re doing is seemingly “good”, the next day it is “evil”. Punishment is swift and without trial.

Thom Yorke sums it up well:

In the days before we meet, he has been watching a box set of Adam Curtis’s BBC series, All Watched Over by Machines of Loving Grace, about the implications of our digitised future, so the arguments are fresh in his head. “We were so into the net around the time of Kid A,” he says. “Really thought it might be an amazing way of connecting and communicating. And then very quickly we started having meetings where people started talking about what we did as ‘content’. They would show us letters from big media companies offering us millions in some mobile phone deal or whatever it was, and they would say all they need is some content. I was like, what is this ‘content’ which you describe? Just a filling of time and space with stuff, emotion, so you can sell it?”

Having thought they were subverting the corporate music industry with In Rainbows, he now fears they were inadvertently playing into the hands of Apple and Google and the rest. “They have to keep commodifying things to keep the share price up, but in doing so they have made all content, including music and newspapers, worthless, in order to make their billions. And this is what we want? I still think it will be undermined in some way. It doesn’t make sense to me. Anyway, All Watched Over by Machines of Loving Grace. The commodification of human relationships through social networks. Amazing!

There is no question the value of content is being deprecated by big aggregation companies. The overhead of creating well-researched, thoughtful content is the same whether search engines value it or not. And if they do value it, a lot of the value of that content has shifted to the networks, distributors and aggregators and away from the creators.

Facebook’s value is based entirely on the network itself. Almost all of Google’s value is based on scraping and aggregating free content and placing advertising next to it. Little of this value gets distributed back to the creator, unless they take further, deliberate steps to try and capture some back.

In such a precarious environment, what incentive does the publisher have to invest and publish to the “free” web?

Content Deals

Google lives or dies on the relevancy of the information they provide to visitors. Without a steady supply of “free” information from third parties, they don’t have a business.

Of course, this information isn’t free to create. So if search engines do not provide you profitable traffic, then why allow search engines to crawl your pages? They cost you money in terms of bandwidth and may extract, and then re-purpose, the value you created to suit their own objectives.

Google has done content-related deals in the past. They did one in France in February whereby Google agreed to help publishers develop their digital units:

Under the deal, Google agreed to set up a fund, worth 60 million euroes, or $ 80 million, over three years, to help publishers develop their digital units. The two sides also pledged to deepen business ties, using Google’s online tools, in an effort to generate more online revenue for the publishers, who have struggled to counteract dwindling print revenue.

This seems to fit with Google’s algorithmic emphasis on major web properties, seemingly as a means to sift the “noise in the channel”. Such positioning favors big, established content providers.

It may have also been a forced move as Google would have wanted to avoid a protracted battle with European regulators. Whatever the case, Google doesn’t do content deals with small publishers and it could be said they are increasingly marginalizing them due to algorithm shifts that appear to favor larger web publishers over small players.

Don’t Be Evil To Whom?

Google’s infamous catch-phrase is “Don’t Be Evil”. In the documentary Inside Google”, Eric Schmidt initially thought the phrase was a joke. Soon after, he realized they took it seriously.

The problem with such a phrase is that it implies Google is a benevolent moral actor that cares about……what? You – the webmaster?

Sure.

“Don’t Be Evil” is typically used by Google in reference to users, not webmasters. In practice, it’s not even a question of morality, it’s a question of who to favor. Someone is going to lose, and if you’re a small webmaster with little clout, it’s likely to be you.

For example, Google appear to be kicking a lot of people out of Adsense, and as many webmasters are reporting, Google often act as judge, jury and executioner, without recourse. That’s a very strange way of treating business “partners”, unless partnership has some new definition of which I’m unaware.

It’s getting pretty poor when their own previously supportive ex-employees switch to damning their behavior:

But I think Google as an organization has moved on; they’re focussed now on market position, not making the world better. Which makes me sad. Google is too powerful, too arrogant, too entrenched to be worth our love. Let them defend themselves, I’d rather devote my emotional energy to the upstarts and startups. They deserve our passion.

Some may call such behavior a long way from “good” on the “good” vs “evil” spectrum.

How To Protect Value

Bottom line: if your business model involves creating valuable content, you’re going to need a strategy to protect it and claw value back from aggregators and networks in order for a content model to be sustainable.

Some argue that if you don’t like Google, then block them using robots.txt. This is one option, but there’s no doubt Google still provides some value – it’s just a matter of deciding where to draw the line on how much value to give away.

What Google offers is potential visitor attention. We need to acquire and hold enough visitor attention before we switch the visitors to desired action. An obvious way to do this, of course, is to provide free, attention grabbing content that offers some value, then lock the high value content away behind a paywall. Be careful about page length. As HubPages CEO Paul Edmonds points out:

Longer, richer pages are more expensive to create, but our data shows that as the quality of a page increases, its effective revenue decreases. There will have to be a pretty significant shift in traffic to higher quality pages to make them financially viable to create”

You should also consider giving the search engines summaries or the first section of an article, but block them from the rest.

Even if you decide to block search engines from indexing your content they still might pay others to re-purpose it:

I know a little bit about this because in January I was invited to a meeting at the A.P.’s headquarters with about two dozen other publishers, most of them from the print world, to discuss the formation of the consortium. TechCrunch has not joined at this time. Ironically, neither has the A.P., which has apparently decided to go its own way and fight the encroachments of the Web more aggressively (although, to my knowledge, it still uses Attributor’s technology). But at that meeting, which was organized by Attributor, a couple slides were shown that really brought home the point to everyone in the room. One showed a series of bar graphs estimating how much ad revenues splogs were making simply from the feeds of everyone in the room. (Note that this was just for sites taking extensive copies of articles, not simply quoting). The numbers ranged from $ 13 million (assuming a $ .25 effective CPM) to $ 51 million (assuming a $ 1.00 eCPM)

You still end up facing the cost of policing “content re-purposing” – just one of the many costs publishers face when publishing on the web, and just one more area where the network is sucking out value.

Use multiple channels so you’re not reliant on one traffic provider. You might segment your approach by providing some value to one channel, and some value to another, but not all of it to both. This is not to say models entirely reliant on Google won’t work, but if you do rely on a constant supply of new visitors via Google, and if you don’t have the luxury of having sufficient brand reputation, then consider running multiple sites that use different optimization strategies so that the inevitable algorithm changes won’t take you out entirely. It’s a mistake to think Google cares deeply about your business.

Treat every new visitor as gold. Look for ways to lock visitors in so you aren’t reliant on Google in future for a constant stream of new traffic. Encourage bookmarking, email sign-ups, memberships, rewards – whatever it takes to keep them. Encourage people to talk about you across other media, such as social media. Look for ways to turn visitors into broadcasters.

Adopt a business model that leverages off your content. Many consultants write business books. They make some money from the books, but the books mainly serve as advertisements for their services or speaking engagements. Similarly, would you be better creating a book and publishing it on Amazon than publishing too much content to the web?

Business models focused on getting Google traffic and then monetarizing that attention using advertising only works if the advertising revenue covers production cost. Some sites make a lot of money this way, but big money content sites are in the minority. Given the low return of a lot of web advertising, other webmasters opt for cheap content production. But cheap content isn’t likely to get the attention required these days, unless you happen to be Wikipedia.

Perhaps a better approach for those starting out is to focus on building brand / engagement / awarenesss / publicity / non-search distribution. As Aaron points out:

…the sorts of things that PR folks & brand managers focus on. The reason being is that if you have those things…

• the incremental distribution helps subsidize the content creation & marketing costs
• many of the links happen automatically (such that you don’t need to spend as much on links & if/when you massage some other stuff in, it is mixed against a broader base of stuff)
• that incremental distribution provides leverage in terms of upstream product suppliers (eg: pricing leverage) or who you are able to partner with & how (think about Mint.com co-marketing with someone or the WhiteHouse doing a presentation with CreditCards.com … in addition to celebrity stuff & such … or think of all the ways Amazon can sell things: rentals, digital, physical, discounts via sites like Woot, higher margin high fashion on sites like Zappos, etc etc etc)
• as Google folds usage data & new signals in, you win
• as Google tracks users more aggressively (Android + Chrome + Kansas City ISP), you win
• if/when/as Google eventually puts some weight on social you win
• people are more likely to buy since they already know/trust you
• if anyone in your industry has a mobile app that is widely used & you are the lead site in the category you could either buy them out or be that app maker to gain further distribution
• Google engineers are less likely to curb you knowing that you have an audience of rabid fans & they are more likely to consider your view if you can mobilize that audience against “unjust editorial actions”

A lot of the most valuable content on this site is locked-up. We’d love to open this content up, but there is currently no model that sufficiently rewards publishers for doing so. This is the case across the web, and it’s the reason the most valuable content is not in Google.

It’s not in Google because Google, and the other search engines, don’t pay.

Fair? Unfair? Is there a better way? How can content providers – particularly newcomers – grow and prosper in such an environment?