What’s the problem here? Just enjoy the attention. Of course, the fact that you notice them looking at you (your chest) means you are looking at them (how else would you know?). So maybe they’re actually looking back and maybe they’re a little shy to make direct eye contact, so they look at your chest?
In any event, the problem isn’t when they look; it’s when they stop looking.
That happened to me a couple of years ago. Funny thing: once I started supplementing testosterone, they started looking again. Or maybe it’s just that I started looking at them again :) .