Irsay married Meg Coyle in 1980, and the couple have three daughters, Carlie, Casey and Kalen. After being separated since 2003, Meg filed for divorce on November 21, 2013.

Irsay was 12 years old when his father, Robert Irsay, purchased the Baltimore Colts. After graduating from SMU in 1982 he joined the Colts’ professional staff. He was named Vice President and General Manager in 1984, one month after the Colts relocated from Baltimore, to Indianapolis. After his father suffered a stroke in 1995, Jim assumed day-to-day management with the role of Senior Executive Vice President, General Manager and Chief Operating Officer in April 1996. When his father died in 1997 Jim engaged in a legal battle with his stepmother over ownership of the team, but later became the youngest NFL team owner at that time at 37.

Irsay was born in Lincolnwood, Illinois, the son of Harriet (née Pogorzelski) and Chicago businessman Robert Irsay. His father was from a Hungarian Jewish family and his mother was the daughter of Polish Catholic immigrants. Irsay was raised Catholic, and did not know about his father’s Jewish heritage until he was fourteen. Jim’s brother, Thomas, was born with a mental disability and died in 1999, and his sister, Roberta, died in a car accident in 1971. Irsay attended high school at Loyola Academy in Wilmette, Illinois a suburb just north of Chicago and at Mercersburg Academy ’78, Mercersburg, Pennsylvania. After high school, he attended, and graduated from, Southern Methodist University in 1982 with a degree in broadcast journalism. Irsay played linebacker for the SMU Mustangs football team as a walk-on, but an ankle injury ended his playing career.

Outside of football, Irsay has made significant investments in music and memorabilia. In 2001 Irsay purchased the original manuscript of On the Road, or “the scroll”: a continuous, one hundred twenty-foot scroll of tracing paper sheets that Jack Kerouac cut to size and taped together, for $2.43 million. On May 5, 2018, he purchased an original printing of the 1939 book Alcoholics Anonymous with notes handwritten by the author Bill Wilson, cofounder of AA, for $2.4 million at auction.

In 2009 Irsay was vocal about preventing a group that included talk-show host Rush Limbaugh from purchasing the St. Louis Rams. “I, myself, couldn’t even consider voting for him,” Irsay said at an NFL owners meeting. “When there are comments that have been made that are inappropriate, incendiary and insensitive… our words do damage, and it’s something we don’t need.” Irsay has made political contributions to John Edwards and Harry Reid.

On March 16, 2014, Irsay was arrested under suspicion of DUI and drug possession in Carmel, Indiana. According to The Indianapolis Star sports columnist Bob Kravitz, Irsay had an ongoing drug problem. This was highlighted when it was later revealed that Irsay’s mistress, Kimberly Wundrum, had overdosed and died in a house that Irsay controversially purchased with money belonging to the Indianapolis Colts.

Irsay’s daughter, Carlie, took over the day-to-day operations of the Colts while he was in rehab. On September 2, 2014, shortly after pleading guilty to OWI and being sentenced to one year of probation, Irsay was suspended by the NFL for six games and fined $500,000.

Irsay has purchased guitars originally owned by Elvis Presley, George Harrison, John Lennon, Paul McCartney, Jerry Garcia (“Tiger”), Prince, Les Paul’s 1954 Black Beauty and other notable performers. His purchases have set records: in 2014 he bought the electric guitar that Bob Dylan played at Newport for just under US$1 million and in 2017 he paid US$2.2 million for a Ludwig drum set belonging to Ringo Starr. On June 20, 2019, Irsay paid a record $3.975 million for a guitar, known as The Black Strat, formerly owned by Pink Floyd guitarist David Gilmour.