Born at Ashland Farm in Lexington, Kentucky, Thomas was the fourth of the seven children of Anne Clay (1837–1917) and her husband, Major Henry Clay McDowell (1832–1899). His mother was the daughter of Henry Clay, Jr.. In 1883 she and her husband purchased the 325-acre (1.32 km) estate from other Clay descendants. Henry Clay McDowell bred Standardbred horses for harness racing, and Thomas became interested in the breeding and training of racehorses.