With his failures behind him, Gordon turned to Victoria, not to Melbourne, which had ignored his poetry, but to Ballarat. In November, he rented Craig's livery stables at Ballarat in partnership with Harry Mount, but he had no head for business and the venture was a failure. In March 1868, he had a serious accident, a horse smashing his head against a gatepost of his own yard. His daughter, born on 3 May 1867, died at the age of 11 months, his financial difficulties were increasing, and he fell into very low spirits.
In spite of short sight, he was becoming very well known as a gentleman rider, and on 10 October 1868, actually won three races in one day at the Melbourne Hunt Club steeplechase meeting. He rode with great patience and judgment, but his want of good sight was always a handicap. He began riding for money, but was not fortunate and had more than one serious fall. He sold his business and left Ballarat in October 1868 and came to Melbourne, and eventually found lodgings at 10 Lewis Street, Brighton. He had succeeded in straightening his financial affairs and was more cheerful. He made a little money out of his racing and became a member of the Yorick Club, where he was friendly with Marcus Clarke, George Gordon McCrae, and a little later Henry Kendall. On 12 March 1870, Gordon had a bad fall while riding in a steeplechase at Flemington Racecourse. His head was injured and he never completely recovered.