A Japanese friend of mine who lives in Yokohama (home of the Cubs-like BayStars) told me that Tokyo feels that it doesn't really need the games. It already hosted an Olympics in 1964 during a time when, according to my friend, "the Olympics meant something in the world."
Dan Simon of the Beachwood Reporter--the only other reliable source for skeptical Olympics coverage in the whole damn city--asks around Tokyo about their bid for the 2016 Olympics and is greeted with a shrug. Definitely worth a read.
I'm going to Atlanta next month and hope to check out the lasting physical impact on the city of the 1996 games. My previous experiences with Olympic cities have been Lake Placid, which was conveniently turned into a training center, and Montreal, a terrible Stalinist-modernist mess that hurt the city's economy for years (and saddled the Expos with one of the worst baseball stadiums ever):
Overspending of public money on a one-off event can mean higher taxes for years to come, draining spending power from the economy and so harming both property prices and rents. The nightmare Olympic case study is that of the Montreal Olympics in 1976, which lost money and left the Canadian city's taxpayers to pick up the bill.