[Torchwatch]

If it makes you happy without guilt, or if you find a little guilt exciting then keep going and enjoy the sensuality of your briefly clad, slim, fit body.
If you are being overcome by doubt and guilt then try something more masculine and slightly fuller. Don't go for big baggy shorts though, you would feel too constricted and constrained.
There are loads of types of brief swimwear a man could wear, experiment and explore your reactions.
Decide for yourself whether you are drawn to brief swimwear in order to explore your masculine or feminine side, you may have cross dressing tendencies, many people still do not accept cross dressing as a positive lifestyle choice.