How should I feel ?

[Lessisbetter]

Hello. Newbie here. I'm 50 years old I'm straight and married. Enjoy being in the sun and pool or lake.
For the past few years I've started wearing smaller swimwear to the neighbors pool. Just a bit smaller than Speedos. I truly enjoy having less on wile swimming or tanning. I have one neighbor who seems to be bothered by it. And she just makes dumb comments wile she's there. And my wife seems to be embarrassed at times. I'm not overweight I'm tan and 6' tall. I cant stand wearing regular swimwear anymore.
Why is this bothering me. Any ideas will help. Thx.