A man joined an imam praying the 'Isha' prayer with the intention of 'Asr, then remembered that he had already prayed 'Asr with another congregation

A man entered to pray the 'Asr prayer and found that the worshippers had finished the prayer. He left the mosque and went home, having forgotten to pray the 'Asr prayer. He then prayed Maghrib and 'Isha'. Before entering for the 'Isha' prayer, he remembered that he had not prayed 'Asr. He joined the imam with the intention of 'Asr. When he said the takbir, he remembered that he had already prayed 'Asr with a congregation that had missed the prayer. What is the ruling in this case?
The ruling in this case is that if he remembers that he has already prayed 'Asr, he must break off the prayer and start again with the intention of 'Isha'.