I used to have this problem too. So whenever i went out, whatever the weather I'd make sure I took a baby shawl with me to drape over my shoulder, that way, whenever the baby got fidgety/distracted, it didn't matter if he let go for a look around. He could still find the nipple to resume his feed. However, by using the shawl, I found two things.
1 - It works best with younger babies. The shawl cuts out the bright lights/people walking past, etc.
2 - Older babies can grab it and pull it off.
It's a case of finding a happy medium.
See if you can arrange to go out/visit people when you're baby shouldn't need a feed.