It would be one thing for people to insult you after learning that you don't speak it. But if you live in a place where Luganda is the local language, it's safe to assume you speak it. Imagine me getting mad that while in Gulu, people attempted to speak to me in Acholi. https://twitter.com/Made_in_Apac/status/1198540659537432576

Buganda must be the only place where the locals are considered rude for not adjusting for the linguistic comfort of visitors/settlers. Never mind that to speak English conversationally takes the privilege of having gone to school up to a considerable level.

Expecting people to address you in a second, school instructed language, because you haven't been able to learn the street language, is classist privilege.

Even non-Baganda, if poor & uneducated, operate in Kla, at a level where Luganda is a more accessible language to learn than English. If you've had enough privilege to learn English, you have more resources than they do learn languages. Invest in it & adjust or be easy.