This is an excellent look from the NYT into the heart of the Googleplex and how they deal with the changing landscape of internet search and the amazing Google algorithm that ranks billions of web pages for relevance to billions of queries.
I’m already looking forward to the Google Party which is held each August down at the Googleplex.
Don Dodge is always doing great, straightforward biz math over at his blog and today is no exception. He looks at Search biz and search revenues and concludes that one percent of the search market is worth about a billion bucks.
I think that the key concept in play right now is “advertising”. This is contrary to many silly protestations of the big players who claim that “user centric computing” is the key to success. I do think that many on the development teams actually believe their own hype, but it’s clear from the behaviors and allocations of resources that ads are the online king and will remain the key development driver for some time.
Can you have ads and good user stuff? Of course you can. Google has done the best job with this though I think they are now on a slippery slope with more ads, more ambiguous ads, and considerable collateral damage in the spam wars, but can you blame them when, as Don points out, there are billions on the table and a lot of potential players waiting for a piece of the search action.
Jimmy Wales, founder of Wikipedia, discusses his Wikia search projectand the internet. He’s the chairman of Wikia, Inc. He thinks it’ll be 2-3 years before they have a robust product.
“Democratic, participatory” search project.
“Google, Yahoo, Ask” have similar, proprietary and closed search. He wants to break up the idea that a few companies should be so dominant.
Making search ubiquitous. He thinks Google may not have problems with WIKIA because they can keep matching up ads, advertisers, and buyers as they have been.
Wales thinks Facebook made the right decision to turn down Yahoo’s billion+ offer for Facebook, calling it an “interesting gamble”. “He’s a pretty sharp guy” (Zuckerman), and Wales thinks that unlike Myspace, Facebook is doing right by the customers. Notes increase of spam and advertising intensity of Myspace.
Wikia major initiatives: Search, Reference Works for humor, opinion, sports. 66 languages plus a “Klingon language” project. “Roll this revolution” into many other areas. What makes the internet great is that it’s a “global platform for people to share knowledge”. Keeping it “open” appears to be a key guiding principle for Wales, and his admirable efforts at Wikipedia support his sincerity in that mission.
Wales suggests that Firefox is the best browser, primarily due to features that he sees as the result of the open source development model that created Firefox. He says that monopolistic activity by Microsoft has slowed innovation, but feels that Google is a friend of Open Source. Wales recounted telling Bill Gates at Davos that Microsoft search is so bad people are switching away from it as the Vista default, and suggests that he’ll have fun trying to build a better search than Google with Wikia.
Yahoo may buy Bebo, the British “Myspace”, for a billion dollars. That is a LOT of money – about 3% of Yahoo’s market cap. Presumably this, like Yahoo’s unsuccessful Facebook aquisition attempt, is Yahoo’s approach to recapturing the market dominance it enjoyed back in the day. Dominance through the aquisition of a social network rather than developing their own.
They should know better than to trust their existing criteria for decisions about aquisitions. Yahoo is the company that aquired Overture’s pay per click technology years ago, and then managed to cede dominance in that area to Google. Ever heard of Google? Yahoo probably could have *owned* Google, but it seems higher management didn’t think search had the monetization potential of … broadcast.com which was purchased for billions.
Isn’t it time for top management at Yahoo to let innovation, not aquisitions, rule the day? This approach has worked very well for Google, who’s main mistakes now appear to be in aquiring things like YouTube which in my opinion is unlikely to recover YouTube’s 1.6 billion price tag and will certainly pester Google with big money lawsuits for decades. Yahoo’s still got a LOT of great technical people, especially in the developer and new business divisions. More importantly, the world is producing hundreds of thousands of new, brilliant innovators every year, most of whom are chomping at the bit to bring new and exciting innovation to the hungry online world. Why not devote the billions to this rather than purchasing companies with marginal revenues and long term prospects that are more hope and prayer than reality?
With the latest flurry of high priced aquistions it almost seems like, to the big players, the billion dollar deal is the new million dollar deal. I remain skeptical that deals of this size pay off in the long run – certainly very, very few of the early pre-bubble ones did not pay off for companies. I’d suggest that the smaller deals (e.g. Flickr) do have potential, but that Yahoo’s top management is looking for a killer deal that simply does exist while the innovation approach (ie much, MUCH more support to the core values and teams at Yahoo) is starting them in the face. Traffic? Yahoo’s got plenty of it. Modest changes can send millions of Yahoo users to any new idea, so why not do this *a lot more* and test *a lot more ideas*.
Edison suggested that there is always a better way, and it’s time for Yahoo to ….. find it.
More Bebo-logy from Techmeme:
Yahoo may net Bebo owners $1bn
Bebo/YHOO: My Rumor’s Bigger Than Yours
Yahoo May Be Bidding For Social Network Bebo: Report
Here is a nice post from Google about their new policy to anonymize search info from users. Like many I have been critical in the past of Google and others for storing this information with little regard to who owns it or saying what they’ll be doing with it. Yahoo and MSN do not (yet) have similar policies so I think Google can rightly claim a higher road since they have also been the one who has fought Government attempts to nab search data. (I have mixed feelings about that since, unlike folks like Battelle, I fear commercial abuses more than I fear the Government will use my data in illegal and harmful ways.
One is torn between respect owed to Google for all they’ve done with search and frustration with their insufficient help/info for downranked sites. I know a small number of folks on the web spam team work to keep “collateral damage” low, but I think what bugs me is the ongoing strong implication that there is very little collateral damage when in fact there is a lot.
Ironically this opaque approach to downranking penalties is what spawns a lot of bad information at many forums and leads to the mistrust of Google that is increasingly common among many of the elite SEOs and webmasters.
The big part of my frustration comes from what I think is a lie, or at best a misleading thing that Google tells sites in the standard emails from Google support, which says that because your site is found in the Google index you have no penalty.
I now believe that by any reasonable definition of “penalty” this is a false and unreasonable statement.
What they really mean by this emails is that your site has no “manual penalty”. A manual penalty is invoked in extreme cases where sites are removed from the index. This is generally for things like hidden text, sneaky redirection, or other SEO tricks banned by the Google Webmaster Guidelines. However, if your site has a big downrank it probably has been penalized by the algorithm in a direct way, probably by a subtraction of points that leads to a much lower score for many/all of the pages in your site.
Here’s a good example of the downranking penalty at our Online Highways Travel site:
Searching Google for “Province of Djizak” it would be reasonable for a user to find this page somewhat high among the results: http://www.ohwy.com/uz/z/zdjizak.htm
Why would a user want this page? It’s highly relevant for the search, leads to more info about Uzbekistan, and our Uzbekistan section was created mostly by a leading travel expert from Uzbekistan who publishes the leading travel magazine for the Silk Road region of Asia.
So, why is this page relegated to obscurity, at position of approximately 190 of 193 results listed? Here it is on the last page of the Google results.
I wish I knew, though I’ve been assured by Google in several emails that we have no penalty when clearly … we do.
Google probably has a right to penalize and re-rank however they see fit, but along with this power and responsibility goes an obligation to tell an unvarnished truth about the status of sites. I used to believe that large sites with high advertising spends were not more likely to get special help than small sites, and to Google’s credit they have historically been good listeners/talkers at events like WebmasterWorld and Search Engine Strategies, but I now wonder if the lines are getting blurred between the advertising and ranking realms at Google. Google probably has the right to do things as they see fit, but please don’t tell me that thousands of small and medium-sized sites with relevant pages aren’t getting penalized and downranked when they … clearly are.
Update: Blogging about this has affected the results – on May 11 this blog post is number one for the term “Province of Djizak”! Our subject page remains very low – about 201 in rankings even though it is *referenced* by the number ONE page for the term (and of course is much more relevant to the search).
This, combined with the Chico the Wonder Dog experiments and a lot of reading and talking with SEO people, leads me to think that the downrank penalty really is site wide and that Google really is sacrificing a lot of good pages like our UZ section to punish us for what they see as undesirable cross linking / thin pages / failure in some cases to use nofollow on links / ?
Update 2: Maybe I shouldn’t complain about the rank? Our Djizzak Province page appears, after all, two places above this, um, highly relevant page for that query: Application of defecation lime from sugar industry in Uzbekistan
Update 3: OK, I have now created what I would argue is the world’s best “Province of Djizak” web page, located at the Online Highways blog. Unfortunately I had a problem changing the title but that page should *at least* rise higher than 200 for a query. Why? Because it is quite a bit more relevant than any others for that term and it now has TWO LINKS from this, the top page for the query “Province of Djizak“. If my hypothesis is correct it will not rise up because it will fall under OHWY’s site downranking penalty.
Update 4: Province of Djizak original OHWY page is now number one at Google for “Province of Djizak”. This is NOT at all consistent with my site penalty hypothesis above. It is consistent with the idea that we need to beef up incoming, new links to get pages re-ranked.
Update 5 (June 1). The original OHWY page is again heavily penalized – number 216 from number 1 yesterday. This, alas, is totally consistent with the sitewide penalty hypothesis I describe above.
Matt’s got a great post and comment section going on the issue of online privacy. His point is well taken – Google stores less information about you than most other players online and your ISP is the place that has a history of *all sites visited* where Google would have somewhat less information.
My take on this is the same as it’s been for some time – the dangers are more likely to come from commercial abuses than governmental ones. Here’s what I wrote over there:
Matt this is an excellent post and comments conversation. Battelle was just talking privacy at his (neat!) online forum today. I’m guilty of “forgetting” that ISPs have more info about users than Google, and that it’s probably via ISPs or Carnivore-style intrusions that people’s privacy will be compromised, not via Google.
However, there are commercial issues that need to be addressed and IMHO are very poorly dealt with by Google and others. As a user I should own my own stuff, not the company making the application I use to produce stuff. This includes reviews, comments, and even search data. This ownership is routinely compromised, and this will piss people off more and more as they come to understand the big picture.
Hey, I’m (sort of) liveblogging John Battelle’s search presentation which is currently … online via WebEx conferencing. John’s always interesting to hear, but this is more about the process than the content. I did like his slide noting that Web 2.0 is about digitizing the customer where early efforts were digitizing the office.
John’s talking about search and it’s neat to have some real time attention and the chance to ask questions via a text control panel – but HEY, nobody answered my questions yet! The WebEx system is not quite as navigation friendly as I’d expected but that’s OK because it’s trying to do a lot of things – polls, questions, and the audio broadcast itself. However I think Pirillo’s videocasting efforts may be a good lesson for WebEx as they move ahead with these approaches.
Even on my laptop with WIFI the audio is fairly stable and good quality. There appear to be about 64 people online per the poll results.
After attending about 10 conferences last year I’ve wondered how much better learning and retention I’d have if I’d just read up on the presenters and topics rather than actually go to the conferences. Shmoozing is fun and often educational, but even that might be done better away from conferences.
Google is no longer narily, barely shading the advertisements that appear at the top of the organic listings on the left of the search web page. This may be a regional thing or experimental (I’m in Oregon on Charter ISP), but it’s very conspicuous and frankly it makes it very difficult to distinguish between ads and real content.
Although I’ve always held that Google has a right to do this type of thing, I’ve always been frustrated with the pretense that they always take the high road and “don’t do opportunistic things”. This is a huge departure from Google’s previous approach and claims, which suggested how critical it is to separate organic and commercial listings. As this screen shows it is now *impossible* for the user to make that determination. Good for advertisers but bad for users and somewhat misleading.
The FCC actually claims to object to this approach, telling search engines some time ago that they need to make a clear distinction between commercial content and non-commercial.
I’m assuming they are testing the affect on clicks and revenue, and clearly it will be enormously profitable to do things this way as smart users typically look first to organic listings and last to advertising. However, in the long run it challenges the idea that Google’s primary interest is providing “unblemished” results. At the very least Google owes people an explanation here, and if it does not include the fact they’ll make a lot more money this way, and that that was a prime motivation, clear thinkers are going to call foul on this new practice.
Metaweb‘s been working on a semantic search routine called “Freebase” that seeks to provide information from cross connected databases all over the world. NYT Article.
My understanding is that they want a simple, natural language search engine for people which will then access a huge network of data they have assembled from existing sources. Then users will be allowed to tag and add to that data, creating even more detail for the database.
Tim O’Reilly notes that Freebase:
… turns its users loose on not just adding more data items but making connections between them by filling out meta tags that categorize or otherwise connect the data items …
So, why is this better than Wikipedia or DMOZ? It’s got more data sources, will be easier to use, and hopefully won’t suffer from the many insufferable editors and participants that plague other user driven social media like DMOZ and DIGG. However I think you always need to be cautious assuming people will participate in these projects as intensively as appears to be needed here to make this grow and gain popularity. Maybe I was missing the point but I got bored with Del.icio.us after a short time and did not feel it was creating an infrastructure that would be all that helpful to me, though I certainly see how integrating tags into search will be helpful in the long run.
Isn’t everybody getting tired of working for big, for-profit projects by helping them categorize, rank, index, and detect spam? Where’s the project that lets me do whatever I want on my own terms (write, surf, learn) and then automatically integrates that activity into the indexing and distribution processes?
Joe Duck 