In 1974 Irving met Harriet Bernstein, who was enrolled in his Poetry Workshop at York University. Although he was still living with Aviva, Irving and Harriet began an affair that continued for four years, culminating in their legal marriage in November 1978. In order to marry Harriet, Irving finally took the required legal action to divorce Betty, which he had neglected to do until this time. In 1981 a daughter, Samantha Clara, was born to Harriet and Irving. The marriage ended in a bitterly contested divorce. Layton then met Anna (Annette) Pottier and invited her to be his housekeeper, although it soon became apparent that she would play a far greater role in his life. Although 48 years his junior she became his fifth and last wife. They lived briefly in Niagara-on-the-Lake in the fall of 1982 and then spent nearly a year in Oakville, Ontario, before moving, at the end of 1983, to the Montreal district of Notre-Dame-de-Grâce. It was here that Layton wrote his memoir Waiting For the Messiah and with Pottier's support saw to the publication of his final books and translations. The couple eventually agreed that Pottier needed to begin a life of her own, and she moved out on March 1, 1995. Friends took care of Layton after he was diagnosed with Alzheimer's disease. He died at the Maimonides Geriatric Centre in Montreal at the age of 93 on January 4, 2006.