You say you're stating a fact, but actually it's an opinion. You need real facts to back up your arguments, rather than unsubstantiated one's you've made up / heard down the pub.
Otherwise it does come across as the sort of thing a racist would say.
Otherwise it does come across as the sort of thing a racist would say.