Crabbe's experience as Chaplain at Belvoir was not altogether happy. He was treated with kindness by the Duke and Duchess, but his slightly unpolished manners and his position as a literary dependent made his relations with others in the Duke's house difficult, especially the servants. However, the Duke and Duchess and many of their noble guests shared an interest in Crabbe's literary talent and work. During his time there, his poem The Village was published in May 1783, achieving popularity with the public and critics. Samuel Johnson said of the poem in a letter to Reynolds "I have sent you back Mr. Crabbe's poem, which I read with great delight. It is original, vigorous, and elegant." Johnson's friend and biographer James Boswell also praised The Village. It was said at the time of publication that Johnson had made extensive changes to the poem, but Boswell responded by saying that "the aid given by Johnson to the poem, as to The Traveller and Deserted Village of Goldsmith, were so small as by no means to impair the distinguished merit of the author."
Crabbe was able to keep up his friendships with Burke, Reynolds, and others during the Duke's occasional visits to London. He visited the theatre, and was impressed with the actresses Sarah Siddons and Dorothea Jordan. Around this time it was decided that, as Chaplain to a noble family, Crabbe was in need of a college degree, and his name was entered on the boards of Trinity College, Cambridge, through the influence of Bishop Watson of Llandaff, so that Crabbe could obtain a degree without residence. This was in 1783, but almost immediately afterwards he received an LL.B. degree from the Archbishop of Canterbury. This degree allowed Crabbe to be given two small livings in Dorsetshire, Frome St Quintin and Evershot. This promotion does not seem to have interfered with Crabbe's residence at Belvoir or in London; it is likely that curates were placed in these situations.
On the strength of these preferments and a promise of future assistance from the Duke, Crabbe and Sarah Elmy were married in December 1783, in the parish church of Beccles, where Miss Elmy's mother lived, and a few weeks later went to live together at Belvoir Castle. In 1784 the Duke of Rutland became Lord Lieutenant of Ireland. It was decided that Crabbe was not to be on the Duke's staff in Ireland, though the two men parted as close friends. The young couple stayed on at Belvoir for nearly another eighteen months before Crabbe accepted a vacant curacy in the neighbourhood, that of Stathern in Leicestershire, where Crabbe and his wife moved in 1785. A child had been born to them at Belvoir, dying only hours after birth. During the following four years at Stathern they had three other children; two sons, George and John, in 1785 and 1787, and a daughter in 1789, who died in infancy. Crabbe later told his children that his four years at Stathern were the happiest of his life.