It is narrated on the authority of Jabir that he heard the Apostle (may peace and blessings be upon him) saying: “Verily between man and between polytheism and unbelief is the negligence of prayer.” (Muslim)
He (peace be upon him) also says:
“The covenant between us and them (i.e. the disbelievers) is prayer, so whoever neglects it has become an unbeliever.” (Ahmad, Tirmidhee, Nasaa’ee])
In addition, Allah the Exalted says:
“[And asking them], ‘What put you into saqar (one of the names of Hell (Jahannam) as mentioned in the Quran)?’ They will say, ‘We were not of those who prayed…'” (Quran: Al-Muddaththir: 42-43).