The reading is all on'yomi, which is great for 祖 but not as great for 父. To remember that this 父 is ふ, just think about how much of a fool (ふ) your grandfather was, even if he wasn't.
Ummm... what can I say? My grandpa is just a moody kind of a guy. He has highs and he has lows. So, don’t even bother him.