The reason that ties people together is their lineage or their history. Lineage and history ties people (especially families) together.
This word can be used for people with a prestigious family history as well as for historic buildings, especially temples and shrines.
You didn't learn this reading for 由 with the kanji, so here's a mnemonic to help you:
If you had good lineage you'd never make a U-ey (ゆい) on the highway. That's dangerous, and people with good lineage and a long history would simply never do such a thing.
Kiyomizu Temple in Kyoto is a temple with a long history.
Just because he has a prestigious lineage, it doesn't make him a nice person.
It's the oldest and most venerable shrine in Japan.