Simon Cowell Lowers 'X Factor' Prize, Defends Judge Shuffles, Disagrees With Adam Levine

"It's like having a dinner party," the EP says of his latest turnover. "You invite people for dinner -- and sometimes it's fun and sometimes it's not as fun as you thought it was going to be."
Frank Micelotta / FOX
"The X Factor"

Simon Cowell was trying his best to be diplomatic during Thursday's Television Critics Association panel for The X Factor. The Fox series returns this fall with a smaller prize and yet another shuffle in the judges -- Paulina Rubio and Kelly Rowland take seats vacated by Britney Spears and L.A. Reid -- something the reality kingpin he says he saw on the wall early on in its second season.

"It's like having a dinner party," said the ep and judge. "You invite people for dinner -- and sometimes it's fun and sometimes it's not as fun as you thought it was going to be."

Cowell would not say when exactly he decided the season two dinner party wasn't as much fun as he wanted, but he did imply that the show was on a course that he hadn't planned on. Another thing he hadn't planned on was the pressures of a $5 million prize. When The X Factor premiered, it touted the biggest prize in reality TV. Cowell says that is no longer the case.

PHOTOS: An 'American Idol' to 'X Factor' Timeline

"We got to a point where it was almost too much," he said. "We want artists to be artists. The prize is now $1 million, and they still get a Sony recording contract."

As for other tweaks, Cowell says there will be adjustments in the format.

"I felt that what we were doing was similar to what other people were doing, and I had this impatience to get on to season three to do what I wanted to do," he said. "I never take part with the idea that you're going to lose. You have to be competitive, and you have to make changes. I work hard. It makes it fun. I'd love to be number one."

Number one, of course, is NBC's The Voice. The series recently eclipsed American Idol as the top talent competition on television, and its 2012 move to fall put it up against The X Factor in a ratings battle that gave The Voice a decisive ratings victory.

OPINION: Simon Cowell's Estrogen Overload: Judging the New 'X Factor' Panel

Earlier in the week, The Voice judge Adam Levine defended his hit series' lack of chart-topping alums, and when the statement came up during Cowell's panel, he aggressively disagreed.

"No show can survive without that," he said, citing the chart success of recent X Factor second runners-up Fifth Harmony. "You're making a promise to the contestants that you're going to turn them into a real-life artist. Otherwise it's a game show. This is the most serious thing that i do."

The lack of names signed to the judges panel on the next season of American Idol prompted one reporter to ask Cowell about ever returning to his old stomping ground, which prompted a predictable reaction.

"I can't imagine the scenario," he said. "I just can't. No."