I've just ordered one of these:
http://www.kiniki.com/collections/mens-swim-briefs-1
Kiniki.jpg
I probably wouldn't wear it a public pool (there's no drawstring) but may do so for wild swimming or a hotel pool. It's only £7.95, so for that price I'm not too bothered.