As soon as Simon announced he was leaving, the question on everyone's mind was the same: Who will replace him? Simon says it's not about picking someone with a certain personality but a certain eye. "You've got to have somebody on that panel who actually knows what they're talking about. Everyone's talking about casting a nasty person or getting another Brit," he says. "If we were judging the ice skating Olympics, you wouldn't have somebody on who's obnoxious, you'd have somebody who knows the difference between a 10, a 9 and an 8. I think, to a point, you kind of have to go back to that."
The most important thing Simon brought to the show, he says, is his ability to spot a potential star, but what he's best know for is his no-hold-barred critiques. "I just couldn't take it seriously when these awful people were coming in and they sing horrifically," he says. "It has its ups and downs. There were many, many times when I've watched the show back, because [we] don't hear the backstory. We don't really know what's going on outside the room, and you hear a story about 'The cat died minutes ago and I'm singing this song for my cat.' I don't know that and when I watch it back and I'm being vile to them I think: 'God, I wish I hadn't said that. If only I had known about the cat.'"
Ultimately, though, Simon says his blunt criticism is actually meant to help the contestants. "It does frustrate me when you get these young kids come in who are absolutely terrible and they've been told all their lives by the mom, dad, family that 'You're going to have a career in show business.' That's when I actually think I'm doing some good, which is to say to them: 'I promise you, you're not going to do what you want to do. It just won't happen, so just stop. It's a fantasy. Get on with something in your life you're going to be successful at,'" he says. "I've given them what has helped me in my life, which is I never, ever got anywhere by people patronizing me. Even today, when I screw up, people around me will say, 'Simon, you've got this completely wrong.' And I listen to them."