Male, it is advisable to only stretch with a flaccid penis state. It is also a good idea to have an adequate warm-up before beginning your PE session. I find that a 10 minute soak in the tub, along with kegals in order to get the blood circulating, are ample, for me anyways, to become “warmed up”. Having a warmed-up body, and penis, means the chances of incurring an injury decrease.
Maybe on the days that you can stretch decently, you are in a relaxed and comfortable frame of mind, meaning that your body is also in a relaxed condition. This seems to help with your ability to stretch, as you are not under the notion of any stress.
As for any physical limitations, are you holding your stretch back with your PC muscle, maybe? This can happen.
The next time that you are able to get a good stretch going, remember how your technique is that enabled you to do this stretch. Make a note of your body position, your grip, grip intensity and force of pull. And hopefully you can continue with this specific stretch technique!