This is a jukugo word, which usually means on'yomi readings from the kanji. Counting days, however, is an exception (at least up to ten). You need to use the kun'yomi readings to read this word. You already know the kun'yomi reading for 四, so that's fine (the ん gets turned into a small っ, though). Think of it this way:
On the fourth day you pour yogurt (よっ) all over your car (か) because on the fifth day you're going to have to eat (いつ) it. (五's kun'yomi reading is いつ).
Why don’t you come watch baseball with me on the fourth of next month?
July 4th is Independence Day in America.
Four days after I met that bear, my back started growing mold.