So much of the male reproductive system is securely planted in the body. Sperm in the testicles is only one small part, and the most visible part, of the system. So, unless PE severs the vas deferens (the “tube” that leads out of the testicles) or you somehow manage to yank loose the starting point of the urethrea (also located in the body) or, comically enough, you manage to crush your nuts with your hanging weights, I don’t see how PE can cause infertility.
Good YouTube that explains it titled: How Sperm Travels through Male Reproductive System Animation - Sperm Release Pathway -Function Video