WB Returning to Testing Pad
Trials Still the Nets' Scheduling Staple
By Christopher Lisotta
The WB, which last year was the only broadcast network that opted not to incorporate pilot testing results into its scheduling decisions, is returning this week to the practice of using audience feedback to help shape its 2005-06 season lineup.
Traditionally, in the weeks leading to the upfronts, which kick off May 16, all six networks have taken into account information gleaned from test audiences when deciding which shows to greenlight. But in March 2004 Jordan Levin, then co-CEO of The WB, told TelevisionWeek his network was abandoning pilot testing as a factor in scheduling decisions.
Mr. Levin has since left The WB, and the network is going back to business as usual under former studio executive and producer David Janollari, who came on board as president of entertainment in June 2004. Mr. Janollari declined comment for this story, but a spokesman for The WB confirmed the network has resumed testing with audiences through an outside research firm.
For years networks have hired research firms to select a number of viewers to watch pilots and give feedback on what they like about a potential series and why. An entire pilot testing industry of moderators and analysts has grown up around networks, which hope to gain information about how a show will do once it makes it to air. But the failure rate of shows that seemed to perform well in tests and the success of pilots that apparently fared poorly in the system drove Mr. Levin to skip the practice last season.
"We want to filter the noise out," Mr. Levin said of audience testing in 2004. "We're investing in the management team to make tough decisions and trust their guts."
Mr. Levin is not alone in his skepticism of the efficacy of putting pilots in front of test audiences. Insiders, for instance, consistently point out that former NBC sitcom "Seinfeld," one of the most successful series in television history, initially did not test particularly well.
In fact, executives agree, pilot testing is not a guaranteed predictor of series' ratings success. Rather, its value is more specific, giving networks data on a pilot's tone, demographic appeal, character development and cast chemistry. This kind of information can influence or reinforce network decisions on whether or not to pick up a show.
Despite any misgivings, network executives said that when used in the right context, the information provided from tests can be a great guide as they whittle down their options for their new schedules.
"It will never tell you if something is going to be a success or not," Jeff Bader, executive VP for ABC Entertainment, said of pilot testing. "Where it's most useful is helping you with direction. It might help in indicating something in tonality. It does give you some insight into what an audience does latch onto the most."
Mr. Bader said ABC, which is resurging in the ratings this season with the success of its first-season dramas "Desperate Housewives," "Lost" and most recently "Grey's Anatomy," is following the "exact same" testing strategy as last year: focus groups with selected viewers in multiple markets outside of Los Angeles and at-home testing through cable television.
Viacom-owned CBS and UPN, both of which declined to comment for this article, are testing pilots at the company's research facility in the MGM Grand Hotel in Las Vegas. NBC confirmed it is staying the course from last season and is using focus groups and cable tests to get viewer feedback.
Mr. Bader said one of the most important pieces of information to come from pilot testing is what demographic is likely to respond to a pilot. Last year, for example, ABC was confident "Housewives" would do well among women but expected the show to attract a limited male audience. Data yielded from audience testing of "Housewives" indicated men liked the show as much as women. The series turned out to have broad appeal out of the box, just as the testing indicated it would.
"Had you seen the pilot and not seen the testing, you might have thought it wouldn't have male appeal," Mr. Bader said.
Starting this week ABC also is continuing its practice of showing pilots to a range of network staffers, whether they are new assistants or veteran executives. Mr. Bader said the network is sticking to its method of employee viewing, which eschews screenings in theaterlike settings and requires staffers to watch in small groups on televisions.
Fox also will have employees screen pilots in smaller settings starting this week.
After Preston Beckman, Fox's executive VP of strategic program planning and research, came to Fox from NBC three years ago, he brought the practice of gathering groups of a dozen or so staffers together to watch pilots in conference rooms or offices.
Mr. Beckman declined to discuss how Fox tests pilots with sample audiences but said Peter Liguori, the network's new president of entertainment, was keeping the in-house viewing procedure the same for this pilot season.
"He was happy with the process," Mr. Beckman said, noting that Fox's system allows even the most lowly assistant to critique a pilot without retaliation from development executives.
"We do it in a way that is nonthreatening," he said. "We make sure there is a real distribution in terms of titles and responsibility, so you really don't find yourself with a group of all development people."
The Fox in-house screenings, which last about six days, also require employees to stay within the same group.
"Because you're with the same group every day, you start getting pretty comfortable with your opinions," he said.
The groups include some of Fox parent company News Corp.'s most senior executives, even Chairman and CEO Rupert Murdoch himself.
"He's very respectful of people's opinions," Mr. Beckman said of Mr. Murdoch. "Most people aren't afraid to tell him what they think. And he wants to hear it."
Like Mr. Bader, Mr. Beckman said there are things to be learned from pilot testing, beyond whether a screening audience likes a pilot or not.
"You want people to think as broadcasters," he said. "Sometimes there is a tendency to think as art house critics."
He also said that while a pilot may be a great piece of film, staying power has to be considered. "You really have to separate between a great pilot and a great TV series," Mr. Beckman said. "You have to stop and say, `What's episode 10?"'
Outperforming the Norms
Both Mr. Beckman and Mr. Bader said the key to analyzing pilot testing results, whether they're from in-house employee tests or out-of-house sample viewers, is looking beyond the general audience reaction.
"When you say a pilot tests great, on what critique are you talking about?" said Mr. Bader, who pointed to ABC's 1990s sitcom "Home Improvement," which scored average numbers overall but performed very well in some male demos and very poorly among some groups of women.
"If shows are polarizing at all, it's going to affect your overall score," he said. "A high negative isn't bad if you have a high positive with it."
An NBC executive familiar with the testing process said a pilot that tests particularly high may be a sign that a show is merely comforting to the test group or is the kind of program audiences think they ought to watch but don't tune in to once it makes it to air.
"The hits are kind of hidden in that middle, upper-middle range," he said.
Besides narrowing down what demographic profile likes the pilot the best, focus group testing can give networks an idea of an audience's intent to view the series and what story lines and characters resonated the most.
Often, villains in pilots can register negatively with audiences, even though viewers' reactions don't initially take into account the love-to-hate factor that the strongest black-hat characters can bring to a potential series. To override that, networks look at "norms," or audience reactions of similar characters in past pilots, to best gauge success.
"There are villain norms," Mr. Beckman said. "You can't look at a number in a vacuum."
Mr. Bader pointed out that anyone who touts his project as "the highest-testing pilot of the season" does so at his own peril, since just as with characters, shows with different formats have to be evaluated against comparable pilots.
"Dramas tend to test better than comedies in general," he said. "You can't use an overall number to compare the two."
Mr. Beckman described as accurate the "Seinfeld" testing mantra that dictates that shows that strike a tone different from what is generally expected on television fare poorly.
"A good rule about pilot testing is that the less work you have to do on a pilot, the better it is you're going to test," he said. "If you do something that's very different, it generally doesn't test well. That doesn't mean you don't put it on your schedule."
And in terms of reality shows, Mr. Bader said it is almost impossible to test alternative pilots with sample audiences since the myriad reality formats make it difficult to make comparisons. In the age of spoiler Web sites, networks are especially wary of showing even sample audiences shows with an elimination ceremony or a surprise ending, since the reveal may be leaked to a broader audience. But editing pilots to eliminate the surprise doesn't work either.
"If you don't show that it will completely alter the viewing experience," he said. "They are very, very difficult to test."
Ultimately, Mr. Beckman said, networks will take into account testing reactions, but a small group of executives at each network has to eventually close the doors and decide which select pilots make it to air.
"There's a difference between having 80 internal members of this Fox family screening pilots and scheduling the network," he said, noting that he is one of the few who gets to sit in on the meetings where the schedule gets hammered out. "I couldn't do it without hearing what other people have to say. There is no one I worked with here or at NBC who goes into the schedule room and says, `It doesn't matter what I heard from anybody, this is what I'm going to do."'
It's Time to Put Pilots to the Test