If you do not pray all 5 prayers your fast will not be valid.

Shaykh Ibn ‘Uthaymeen was asked in Fataawa al-Siyaam (p. 87) about the ruling on the fasting of one who does not pray.

He replied:

The fast of one who does not pray is not valid and is not accepted, because the one who does not pray is a kaafir and an apostate

The evidence of the Sunnah that proves that the one who neglects the prayer is a kaafir includes the hadeeth of the Prophet(PBUH) : “Between a man and shirk and kufr there stands his neglect of the prayer.” (Narrated by Muslim in Kitaab al-Eemaan)