For a man who had just achieved the seemingly impossible by scooping up an unprecedented eight gold medals in Beijing with a gruelling training regime and ruthless determination, it was an unfathomable concept.
Without hesitation, the American immediately declared he would compete at London 2012 even though he was unsure about how many events he would swim or how he would find a new goal.
He ruled out an attempt at nine gold medals. Then he ruled out trying to match his haul of eight from Beijing. London would have to be less, although how less remains a mystery.
With a little over eight months to go before London, Phelps is still keeping everyone guessing about what the magic number will be this time, partly because he himself remains unsure.
"It's not going to be eight events," he told Reuters in an exclusive interview. "But it is going to be a challenging programme and it's going to be a programme where we think we can be most successful. We're just trying to work towards that now.
Trying to crunch the numbers is almost a mute exercise for Phelps now anyway. With 14 gold medals already in his keeping, he is already the most successful Olympian of all time. The only unresolved question is how far in front he will finish. His biggest problem since Beijing has been finding the motivation to get out of bed before dawn every day and churn through the thousands of laps he needs to stay at his peak.
A global sporting icon, he has already amassed a fortune through his endorsements, so fame and money cannot motivate him to plunge into the water.
"It hasn't been easy," Phelps told Reuters. "For me to be able to do something I have to be able to give 100 percent and over the past few years, I have to be honest, I haven't given 100 percent.
"That has shown in my results and there have been lots of times when I just didn't want to do it anymore."
At the 2009 World Championships in Rome, Phelps entered six events, winning five gold medals and a silver, but was clearly unhappy.
The meet was overshadowed by the use of full-length bodysuits that were later banned and Phelps lost his 200 metres freestyle world record to Germany's Paul Biedermann, one of the leaders of the next generation of swimmers challenging Phelps.
In 2010, Phelps made a quick visit to the Vancouver Winter Olympics in search of fresh inspiration to stay ahead of his younger rivals but his results were getting worse. At the 2011 world championships in Shanghai, Phelps won four gold medals and was beaten by countryman Ryan Lochte in both the 200M freestyle and 200M individual medley.
There were no bodysuits this time and no excuses. Phelps believed he had trained hard and prepared well. He swam the fastest 200M medley time of his life but still could not catch Lochte, who lowered his own world record. Phelps was initially deflated by his losses but they proved to be just the jolt he needed for London. For the first time since Beijing, he could see a way up. There was a challenge for him after all and he returned to training with renewed vigour.
"Those races have definitely been extremely frustrating for me but at the same time they have given me motivation that I could not have asked for. They're really gonna keep me going and keep me strong and add some extra flame to the fire," Phelps said. "They may not be the best things and feelings at the time but those things stay with me and I'm not going to give up."
Phelps said he is logging up to 70 kilometres a week in training and looking ahead to London with enthusiasm after years battling to find inspiration.
"I kinda feel like the old me again," he said. "I have so many goals that I want to accomplish, I look at this as eight months of hard work. I have the rest of my life to go on vacation, travel the world, have fun and do whatever I want.
"I'm enjoying what I'm doing now and that's the way it should be. No-one's forcing me to workout, no-one's forcing me to go into competitions. I have found a way to make swimming fun again."
