[Byron]

The logic of your second paragraph does indeed apply but the key lies in your first paragraph where you include speedos as "a very common type of apparel"
- and those, of whatever persuasion, who enjoy wearing their speedos would certainly want to see this piece of clothing being quietly accepted as the norm without any fuss or derision.
At this point we have to go into the mindset of the homophobic "straight" however and it becomes a case of fear or guilt by association, ie this is a small piece of clothing enclosing the male genitalia and on public display. Might it therefore be a symbol of attraction to other males? Shock, horror ! - "I couldn't possibly let my peers think I had some affinity with those guys by wearing something like that and what I wear will therefore be to emphasise
the opposite, to disassociate me from any possible misconceptions of my own feelings towards these guys of the same gender".
It would be fascinating to have an honest survey of all non-gays to determine just how many in fact indulge themselves in the wearing of bikini underwear or jocks - but of course UNDER their trousers (pants) for none to see.

Those with the speedo prejudice might also consider that male fashion trends of the past 50+ years, now quite "normal" and indeed desirable in order to be seen as "trendy", have in many cases been initiated by gays with an innate sense of design. The line must be drawn for them however and speedos become that line which still needs a lot of fresh thought to break the taboo.