If you're anti-building, you're probably afraid of heights and slopes. The only downside to a building is that they're high and if you fall off of them you could die. So if someone says they're anti-building, they're really just anti-heights.
Another person you know who's afraid of heights is Han (はん) Solo. Now wait, you may be thinking, he flies a space ship! You're right, because he doesn't like to walk all the way up high buildings, it freaks him out. He flies up instead so he doesn't have to think about falling.