This is a cross posting of an article I wrote over at Technology Report about internet marketing:

One of the cornerstones of good internet marketing is knowing your statistics, and you’d think with all the elaborate, inexpensive and free measurement and analytical tools everybody would have a great sense of how their sites stack up to the competition.

But you’d  be wrong.

In fact even many large companies are struggling with high quality analysis even as the tools get better and the measures s-l-o-w-l-y are reaching some level of standardization.     For most small companies metrics are, literally, more misses than “hits”. Webmasters routinely report or misinterpret or misrepresent website “hits” as viable traffic when hits often are simply a measure of the number of total files downloaded from the site.    Graphics or data intensive websites can see hundreds of hits from a single web visitor.

Even when the analysis is good the reporting is often opportunistic or manipulative, and it’s often done by the same team that is accountable for the results.     This is a common problem throughout the business metrics field.  Executives are well advised to have independent auditing of results by unbiased parties for any business critical measurements.

Consider learning and using analysis packages like Google Analytics – a brilliantly robust and free tool provided by Google to anyone.

A while back Peter Norvig, one of the top search experts over at Google (also a leading world authority on Artificial Intelligence), published a little study indicating how unreliable the Alexa Metrics were with regard to website traffic.  (Thanks to Matt Cutts for pointing out the Peter paper.

The results here demonstrates that Alexa is off by a factor of 50x (ie an error of five thousand percent!) when comparing Matt Cutts’ and Peter’s site traffic.

Although this is just an anecdotal snapshot indicating the problem, and perhaps Alexa is better now, I’d also noted many problems with comparisons of Alexa to sites where I knew the real traffic.   50x seems to be a spectacular level of error for sites read mostly by technology sector folks.   It even suggests that Alexa may be a questionable comparison tool unless there is abundant other data to support the comparison, in which case you probably don’t need Alexa anyway.

Of course the very expensive statistics services don’t fare all that well either. A larger, and excellent comparison study by Rand Fishkin over at SEOMOZ collected data from several prominent sites in technology, including Matt Cutts’ blog, and concluded that no metrics were reasonably in line with the actual log files. Rand notes that he examined only about 25 blogs so the sample was somewhat small and targeted, but he concludes:

Based on the evidence we’ve gathered here, it’s safe to say that no external metric, traffic prediction service or ranking system available on the web today provides any accuracy when compared with real numbers.

It’s interesting how problematic it’s been to accurately compare what is arguably the most important aspect of internet traffic – simple site visits and pageviews. Hopefully as data becomes more widely circulated and more studies like these are done we may be able to create some tools that allow quick comparisons.  Google Analytics is coming into widespread use but Fishkin told me at a conference that even that “internal metrics” tool seemed to have several problems when compared with the log files he reviewed.  My own experience with Analytics have not been extensive but the data seems to line up with my log stats and I’d continue to recommend this excellent analytics package.

Jumping down the rabbit hole of the Climate debates is always very interesting but it’s also very frustrating to watch many brilliant (as well as stupid) and well-informed (as well as ignorant) people avoid each other because the blog environments are not civil enough to encourage quality discussion of really intriguing issues.    Great examples of the challenge of discussing science in blogs are my two favorite “watering holes” for the active discussion of climate science:   RealClimate.org and ClimateAudit.org

At both, intelligent and provocative posts often lead to “supportive” commentary from the allies of the blog but also ferocious attacks on critics of the initial post.   This makes for interesting comments and reading if you can handle the emotional / intellectual heat, but I think the overall chases away the two very  important groups who participate in blogging:  the huge number of casual observers  looking for answers to complex questions and the small number of authoritative voices who study a particular complex topic.

Even as a seasoned blogger who rarely wants to back down from discussion points I find it very frustrating to bounce back and forth hoping my reasonable comments will not be moderated (a major problem at RealClimate, and not much of a problem at ClimateAudit)  and hoping that critics will be treat researchers with the basic respect they deserve  (lack of respect is a huge problem at both ClimateAudit and RealClimate, where PhD science authorities are routinely accused of incompetence (mostly at ClimateAudit) and reasonable criticisms are dismissed casually as “nonsense” simply so they do not need to be addressed properly (mostly at RealClimate).

Increasingly blogs moderate reasonable comments because they don’t fit the political agenda of the blog and I still think this is anathema to quality discussion.  Others (like Joe Duck) pretty much allow any comments that are not obscene, spam commercial, or racist so a single person can wind up dominating the conversation, chasing others away.

I’m rethinking my policies about how to manage commentst because it’s good to hear from more pe0ple.  Howevert I’m not going to be snipping or moderating anybody anytime soon.    I think Steve McKintyre of Climate Audit might have the right idea which is to push some comments to “unthreaded” if they are off the topic of the post.   This leaves free speech intact while keeping a few people from dominating the whole comment show.

Final note is that I prefer to err on the side of giving everybody their full voice and I plan to continue doing that here.

Photo and Text Credit:  NASA Hubble.

This is NOT a collage, but one of the new striking images from the Hubble Space Telescope.   These galaxies are a cluster.     Question:  Estimate how many creatures as bright or more intelligent than humans live in the area defined by this picture?

The first identified compact galaxy group, Stephan’s Quintet is featured in this stunning image from the newly upgraded Hubble Space Telescope. About 300 million light-years away, only four galaxies of the group are actually locked in a cosmic dance of repeated close encounters. The odd man out is easy to spot, though. The four interacting galaxies (NGC 7319, 7318A, 7318B, and 7317) have an overall yellowish cast and tend to have distorted loops and tails, grown under the influence of disruptive gravitational tides. But the bluish galaxy at the upper left (NGC 7320) is much closer than the others. A mere 40 million light-years distant, it isn’t part of the interacting group. In fact, individual stars in the foreground galaxy can be seen in the sharp Hubble image, hinting that it is much closer than the others. Stephan’s Quintet lies within the boundaries of the high flying constellation Pegasus.

I’m sure anxious for Ray Kurzweil to hurry up and finish his film “The Singularity is Near” based on his remarkable book of a few years ago because I think the film will spark the global conversation we need to have about the Singularity.    If even the most modest predictions about this even come true it will be the most significant development in the history of humanity, and will reshape our lives and the future of earth in unimaginable ways.   

I am less optimistic than Kurzweil about the time frame and impact of what he sees as a likely explosion of “cosmic intelligence” that rapidly expands throughout the universe,  but I think the notion we will NOT see any conscious computers within 10-15 years is pessimistic and perhaps even naive, resting mostly on the notion that the human intellect is a lot more profound than … it appears to be.

Once self-awareness develops in machines the possibilities are literally endless for the future of humanity.  

An alternative to the “Singularity, Wow!” perspective is offered by brain researcher Edward Boyden who wonders about the role of motivation in the coming crop of artificial intelligences:

Indeed, a really advanced intelligence, improperly motivated, might realize the impermanence of all things, calculate that the sun will burn out in a few billion years, and decide to play video games for the remainder of its existence, concluding that inventing an even smarter machine is pointless.

More from Ed here

Clever writing aside, I think the last thing we need to worry about is motivating the coming AIs.   On the contrary it would seem logical for a self- aware machine with the speed to think billions of times faster than humans to explore (or to use a non-motivated term “analyze”) millions, billions, and trillions of alternatives nearly *simultaneously*.  Unlike the human brain, which has been tuned by the s-l-o-w process of evolution to be slow and very selective and not very efficient, the machine cognition will at the very least be extremely fast and able to process billions of scenarios in very short time frames.    It seems reasonable – in fact inevitable – that at  least a few of those will involve human-like emotional structure and motivation.  Thus even if *most* of the AIs do as Boyd suggests they might and sit on a virtual couch eating virtual potato chips and playing games, some of the others will reinvent humanity in a spectacular way.

Count me in.

Both DARPA SyNAPSE and Blue Brain represent promising approaches to establishing conscious or “self aware” computers, which many believe are the first step to the Singularity.

Singularity University is one of the most interesting developments in education in some time.    SU is bringing together experts in neuroscience, artificial intelligence, engineering, and other thought leaders along with business leaders.   The discussions include abstract topics as well as pressing global problems and it is clear this group could yield some great innovation.

Here are some of the project SU identified in a recent press release:

• One Global Voice leverages mobile phone proliferation to accelerate economic development. It envisions a platform that will provide a set of modular programming tools accessible through a web portal, empowering individuals to create applications empowering education and commerce, linking together the developed and developing worlds.
• Gettaround addresses how an intelligent transportation grid can positively affect energy usage and slow climate change, as people value access over ownership of cars. The first step to the grid, Gettaround is a marketplace for peer-to-peer leasing of under-utilized car hours. It enables car owners to derive revenue from their idle cars, and for renters to have easy access to cars – affordably and conveniently.
• ACASA focuses on advances in rapid, additive manufacturing technologies to construct affordable and customizable housing in the developing world. Cost-efficient, environmentally sustainable solutions have the potential to create a transformative new paradigm for improving housing construction using local resources.
• XIDAR considers a new paradigm for disaster response, allowing users to overcome the communications network problems typical of crisis situations. The project enables innovative solutions to facilitate evacuation, medical triage and aid during natural disasters.

Although I’m not a big fan of Government spending I’d like to offer a suggestion to President Obama and the big spending gang – negotiate an agreement with all the leading scientific journals that will make all scientific papers free or very low cost to anybody.     A big frustration and a possible impediment to innovation is the fact that many science papers are “locked away” by expensive subscription paywalls at leading scientific journals.    One can understand that those journals are struggling to survive and need money, but the current practice of charging exhorbitant fees so only libraries and a handful of specialists can read the leading edge research is misguided at best and intellectual crime at worse.

Since journals operate on what is usually a very low budget, the government could offer very modest amounts – probably something equal to 1.5 to 2 times their current subscription fees to keep the journals ticking and happy.   Authors would be happy to see many times the audience for papers often destined to obscurity.

I think I’d actually favor a “no cost” option that required all research papers funded in any way by any grants or portions of grants  to be made public by the author upon publication,  but the “new” science community seems to be incredibly stubborn about  changes and very protective when territory is threatened so I’m guessing they would likely reject that out of hand, using the argument that the journals should continue to act as a sort of “referee” and organizer of relevant research content.    I think this used to make more sense than now as politics have become too much a part of the research and publications framework (I think mostly in the climate sciences), raising several important issues about publications standards, peer review, and data sharing.

Did you thank your kid today for all the money they’ve been loaning you?

As a supporter of many of the changes President Obama has brought to our national table it’s painful to watch so many folks simply back away from the next glaring financial catastrophe which makes the last one look like a walk in the park.    It’s the massive US national debt and massive deficit spending that has been going on now for over a decade, though never on the scale of the coming years.

Ironically this is a problem all of us were talking about a *lot* before the economy melted down, and even before the GW Bush era of what is best called “fake conservatism”  doubled the national debt.    President Bush was able to push his irresponsible spending down the pike to Obama thanks to inheriting  a very robust Clinton era economy, keeping folks happy with tax cuts, and claiming the war spending was essential to preserve our freedom.   Where a handful of real fiscal conservatives cried foul, legions of fake conservatives stayed silent and followed the leads of the likes of Rush Limbaugh and Sean Hannity who suckered the gullible into focusing on the lies and distortions that fueled the Bush v. Gore,  Bush v.  Kerry, and McCain v. Obama campaigns.   Power was far more important to all the parties than sound principles of governance, spending, or national policy.

Now it’s Obama who is able to ignore the massive spending and debt because the economic meltdown he inherited from Bush has trumped concerns for the debt. and some form of stimulus is certainly called for according to almost all economists.

Thankfully the massive Obama spending stimulus appears to be bringing us out of recession though it’s not clear at what future cost.     In politics it’s hard for parties and Presidents to look too far past the big power curve, and in my view Obama and the Democrats won’t even work hard to undertand the implications of spending themselves past the 2012 elections and then they will just hope for a miracle.   More likely than a miracle is that we’ll see very high inflation as  the Government is virtually forced to print money to repay obligations since the tax rates required to repay the anticipated debts will be totally unsupportable and unsustainable.

Another irony of the situation is that all those formerly quiet “conservatives” are now quite reasonably suggesting we’re creating a new financial catastrophe, but the concerns are falling on deaf Democratic ears because these same people were not critical of their own political allies back when they did have a voice and they tend to blend in nonsense talking points (birth certificates, communist agendas, etc, etc) about Obama and the Democrats.

In simple terms we are saddling our children and future generations with a debt that almost certainly cannot be repaid in any conventional fashion.    The amounts are too staggering to understand even if you are well versed in accounting and math.   This interactive chart is worth a look if you want to see how the numbers shake out in terms of government spending per capita:   http://www.usgovernmentspending.com/US_per_capita_spending.html#usgs302

So, when you hug your son, daughter, niece or nephew today be sure to tell them “thanks for the money”  because they have been loaning you a LOT of money for the past ten years, especially this last one.   Many tens of thousands and growing hugely each year.

We’re stealing our kids prosperity away.   The least we can do is say “Thank you”.

P.S.   The solution to all this is remarkable simple.   After a few years of stimulus deficit spending the government should be required to lower spending to balance the budget over the following few years.  Our current massive entitlement and defense spending could sustain 100% of the required cuts to balance the budge without any significant diminished standards of living for Americans.

IMPORTANT:  Few of the comments below represent my views, but I avoid censoring politics here so I’ve chosen not to delete the many off topic comments.   I’m reconsidering that policy because much of the material below is, in my view, not helpful to any rational discussion of these topics.