Because it takes considerable effort to see results. I think a lot of guys try some kind of PE for a week or two, get frustrated, and quit. In that way, it is very similar to bodybuilding/working out. Almost every guy has the potential to considerably alter and improve his physical appearance and body composition, but a fairly small percentage actually take advantage of their potential. People, in general, are unable to apply consistent effort to something that may not yield visible benefits for months (or even years).
Also, I think that all of the sleazy “grow four inches in five minutes” email spam, and ads for ‘male enhancement’ pills have really diluted/ruined the image that men in general have of PE. Many men spend untold amounts of money on these snake-oil curealls, see no progress, and write PE off as a scam. Also, the fact that the ‘medical establishment’ consistently denies the possibility of non-surgical penis enlargement leads many men to reject the possibility of altering their ‘god-given’ size.
Finally, there is still a very strong stigma and generations of social conditioning that lead men (and women) to believe that sexual subjects, and even their own sexual organs, are somehow dirty or perverse. This leads men to feel guilty both about their natural sexual desires and any inclination they may have to try PE (or any ‘alternative’ sexual practice, for that matter).