I'm a French guy, I have English lessons since I'm 12 years old, and really all the people I know speak English like sea lions, and its even too nice. How can you find it kind of sexy?
People find accents sexy because they're different and unique. You don't like it yourself because you know where the accent comes from and the flaws of it. Like to you it doesnt sound much like an accent, but more like bad pronounciation. Since I don't know how to speak the language, foreign accents just sound exotic and appealing.
I think it is the same as with french language in general. It is a soft language with no hard pronunciation.
So, even if you speak English, you transfer that soft accent too.