He’s kind of an idiot, really. Why didn’t he realize that it was sarcastic?
You have a flavor of dislike. If someone gives you "flavor" and you dislike it, what might that "flavor" be? How about hurtful words.
The reading for 味 is the on'yomi you learned with the kanji, but the reading for 嫌 is the kun'yomi, いや. Just imagine someone saying words so hurtful that you have to scream, "Iyaaaaa!" いや.