When I look at a woman's body, I use my own body as a base level fitness comparison. I'm not the most fit guy, but not un-fit at the same time. To have some wild expectation of a woman's physique, while not having the same male-equivalent expectations about my own would be hypocritical. Also, this doesn't mean other non-physical aspects of the person can't sway my attraction to them.

As for the general male population... If I have learned anything about the Human race, it's that we are more like animals than we would like to believe. If there isn't an initial physical attraction, then it won't work out. That goes for men and women alike.