Quite common
I’ve had that problem plenty of times. I think it’s mostly because I initially used college distributed condoms, which are so thick I could slam my dick in a door and not feel anything (Disclaimer: I am not willing to actually prove this.)
Thinner condoms make it more pleasureable, but the fact is, most girls that I’ve been with have been on the pill, so condoms haven’t been totally required, though still used in some cases. Having been used to sex without condoms, the sight of one causes me to withdraw immediately. I don’t know why, it’s all a mental game for me, I’m sure.