Gilyard's background was dysfunctional; his father, Lorenzo Gilyard (February 24, 1927 – October 24, 1999) was convicted of rape in 1970, his brother Darryl E. (also Daryle born February 12, 1953) and his brothers friend. He was convicted of a drug-related murder in 1989 and sentenced to life without parole, and his sister, Patricia D. Dixon (born July 20, 1958), a sex worker, was convicted of the murder of a customer in 1983 and served a 10-year sentence. In his younger years, Gilyard was known to bully and sometimes beat women.