Words that count days use an exceptional reading. Even though this is a jukugo word, it uses kun'yomi readings for both the kanji. Think about this word this way:
On the fifth day I have to eat (いつ) five cars (か). Now imagine yourself eating five cars. How did you manage to get past the first one? I'm impressed.
I took five days off from work.
June 5th is my friend's birthday.
On the fifth day I said, "That's it!"