Emerging recommendation technology helps pick shows


FUNDING THE FUTURE: Rainmaking investors finance tomorrow's media powers.
HOLLYWOOD WIDGETRY: Interactive programs engage fan bases
UNREALITY SHOW: Alternative reality games  connect tech-savvy fans.
UNNATURAL SELECTION: Recommendation technology guides consumers

Imagine you're listening to the radio in your bedroom. A song comes on that catches your ear -- let's say "Ayo Technology," a hit single from the latest 50 Cent album. It's just the kind of jam that puts you in a good mood, but when it's over, a tinge of disappointment sets in. Surely the next song won't measure up.

But what if it did? What if your radio noted your enjoyment of "Ayo" and calculated that you would next want to hear a song you've never heard of: "Sexy Lady" by Yung Berg. And the weird thing is, this song hits the spot, too.

This technology exists today and can be accessed free at Pandora.com. The Internet music company has a database known as the Music Genome Project that tracks the tastes of all its users and deciphers the attributes of thousands of songs. How else could it match "Ayo" with "Sexy" for its "East Coast rap influences, electronica influences, consistent rhyme patterns and chill rhymin'" (to quote the site's analysis)?

Recommendation technology is anything but a "chill" sector of digital media; it a fast-growing cottage industry relevant to any company with content to sell or the pipeline to distribute it. Also known as "discovery," "personalization" or "taste-sharing," the technology tracks what media you consume and makes educated guesses as to what else you want.

Think about the implications: The more finely attuned these programming services are to their audience's desires, the more they could become effective at commanding consumer time and money. And it's not just music on their minds.

ChoiceStream, CleverSet, Matchmine and Omniture are just a few of the companies turning discovery into a specialty, and a who's who of Hollywood-minded firms are lining up as clients. ChoiceStream alone is working with Yahoo, AOL and Blockbuster, among others.

"Think of us like a good wine steward," says Toffer Winslow, executive vp sales and marketing at ChoiceStream. "They know the product really well and they know the consumer. And the third thing they know is the context in which they are making the recommendation."

Consider discovery an outgrowth of the "long tail" phenomenon: As the Internet opens deep reservoirs of all sorts of on-demand programming, how do you filter all the options? It's the counterintuitive conundrum of unlimited choice: If there's no limit to what you can pick, how do you know what to pick?

"I think one of the bigger myths of our time is that people like choice," says David Watson, vp product design and development for digital media at Disney. "The theory we're working is that as more and more content becomes available, the harder it is going to be for you to find what you want to find."

Discovery is something like a flip side to Internet search. Sites like Google or Yahoo allow users to type what they want to find. But the presumption underlying search is that people always know what exactly they are looking for. As the options for content multiply, from Amazon's Unbox to Microsoft's Xbox, some programming may need to find its way to consumers rather than vice versa.

But recommendation technology is not new, nor has it been very effective. Anyone who has ever pressed the thumbs up or down buttons on their TiVos has gotten more than a few quizzical suggestions, as best ridiculed years ago by the protagonist in the HBO series "Mind of the Married Man," who observed of his machine, "My TiVo thinks I'm gay."

Sites like Amazon also make suggestions without distinguishing whether a consumer is buying for him or herself as opposed to a gift for someone else.

"The whole world of personalization and recommendation had its early innings in the late '90s and it didn't work well then," Winslow says.

But there is an increasing degree of sophistication powering services from the likes of ChoiceStream, a seven-year-old firm based in Cambridge, Mass. Algorithms steeped in the finer points of statistical modeling, probability theory and library sciences bring a renewed rigor to discovery. For instance, movies aren't simply lumped into such broad categories as comedy or drama; ChoiceStream has developed an automated process with much more nuanced descriptors.

Couple a deeper understanding of the product with better data on each individual user and their history of choices, and the result is a more potent tool.

"When you get down into the deep mathematical analyses of this stuff, and you have good data on content and users, you can make good predictions of what users will like," Winslow says. "That's the secret sauce of ChoiceStream."

Anticipating what a user will like can yield significant increases in total sales and session length. ChoiceStream reports healthy returns for client Blockbuster since the home video retailer utilized its recommendation engine for its online rental service, boosting transactions by 50%.

That might be of concern for competitor Netflix had it not been focused on improving its own movie recommendations. Last October, it launched a contest that would award $1 million to whoever could create a better system for suggestions than the one currently in place. Netflix has since been bombarded by tech-savvy inventors who have devised new systems derived from basic information on Netflix's customer base, movie catalogue and ratings data.

A forum on Netflix's site allows contestants to share ideas. "Significant improvements came out of this contest, just out of the sheer collective brainpower being thrown at it," says one contestant, Lester Mackey, who declines to reveal his own formula in fear of providing ammo to competitors.

Mackey and two fellow Princeton University seniors banded together to form Team Dinosaur Planet, a joint effort to win the $1 million prize. What was a pet project in their spare time turned into a university-approved research project that has informed Mackey's choice of graduate work at UC Berkeley, where he studies statistical machine learning and programming languages.

Discovery specialists count on myriad ways to intuit what their customers want. On the most basic level, they follow a user's every move on a Web page, from how long he or she spends in a particular area to what links are clicked. But an even deeper level of understanding comes when users volunteer to rate content as they consume.

That's the method of choice for StumbleUpon, a service that recommends Web sites or online video selections to users who inform those choices by providing personal information and preferences before they "stumble," as well as by clicking on up- or down-thrusting thumb icons. The more you click on them, the more StumbleUpon understands which of the 12 million-plus URLs it has collected in its five years of operation to serve up.

"What we do is essentially allow people to browse intelligently," explains StumbleUpon vp marketing Dave Feller. "Not everyone has a specific query in mind as they go about Internet surfing."

Discovery isn't just a Web-based phenomenon. There's a role for the technology to play even on the boob tube, where VOD is just beginning to transform the way Americans watch television. Research firm Pike & Fischer projects that more than a third of daily TV viewing in the U.S. will be on-demand within five years.

The future of TV consumption has led Disney's Watson to experiment with recommendation technology. A state-of-the-art fiber-to-the-home network serving the residents of Orem, Utah, has allowed Disney to try a VOD service that offers viewers watching, say, an episode of ABC's "Lost" a selection of several thematically similar program options in a picture-in-picture format. An advanced set-up like that could serve viewers better than merely having them scroll through hundreds of programs on VOD menus, which are notoriously difficult to navigate.

"We can anticipate when people are most likely to change the channel and give them alternatives," Watson says.

Recommendation technology can seem a bit Orwellian at times: Could something as fickle and capricious as choosing entertainment options be anticipated with any accuracy? Those working in the field say yes.

"Intuitively we think that our thoughts are not deterministic," Mackey says. "But in fact what we see from experiments that we've done that much of human behavior is predictable to some extent. Even if not all the time, it doesn't mean we can't do well on the average."