The modern man needs cash, debit cards, a mobile phone and car keys in his pockets. When you want to swim all this needs to be left behind it will get wet and maybe salty. Having a small pouch bag on a shoulder strap allows you to keep all your valuables together, you can wear it over your shoulder under your pack on the way to the beach or stuff it into your pack as part of your pillow when sunbathing. When you swim it stays in your pack, when your partner needs a cool drink or ice cream, sling it over your shoulder and trek to the beach bar or shop. If the shop is off the beach grab a pair of shorts to pull on before you cross the road.
This leaves you free to wear speedos, a bikini or a swim thong while on the beach and in the sea.
No excuses, it's not a handbag if it's army surplus.

I just use a waterproof holder that goes around my waist, which contains credit cards, some cash and my phone. I like Overboard products.