Yarbrough started his career at six years old with Gallos de Aguascalientes. In 2005, he was invited for a trial with the U17 team of C.F. Pachuca and later signed with them at the age of sixteen. He played until April 2010 for the youth squad of Pachuca and was loaned out to his first senior team, Jaibos Tampico Madero. He was on loan until July 2011, participating in 38 matches during the Liga Premier de México 2010–2011 season. In summer 2011, he joined Titanes Tulancingo on another loan.

In 2007, Yarbrough received a call up to the Mexico U20 national team. In an interview, Yarbrough said though he played for Mexico U20, he has no preference between the two national teams as he is eligible to represent both. Despite choosing to play for the Mexican Federation in 2007; he ended up accepting an offer to the U.S. national team.

Yarbrough eventually joined León on a permanent basis, becoming their starting goalkeeper for most matches of both the Apertura and Clausura over the following seasons, and helped León to two Liga MX titles. In the 2015 Apertura and 2016 Clausura, Yarbrough led León to a third-place finish, while leading all Liga MX goalkeepers in both saves and clean sheets.

Yarbrough earned his first call-up to the United States national team in March 2015. He appeared in the bench when the United States played against Denmark on March 25, 2015. Yarbrough substituted Nick Rimando in the beginning of the second half against Switzerland on March 31 to earn his first cap. Yarbrough conceded one goal against Switzerland; the game ended in a 1–1 draw. In his second appearance for the United States, Yarbrough was subbed in for Rimando at halftime during a friendly against Mexico on April 15, 2015. He had two saves in the 2–0 victory for the U.S.

In May 2018, León acquired Rodolfo Cota. Cota took over as the starting goalkeeper. Yarbrough started four matches late in 2018, and started two matches in September 2019.

On March 6, 2020, Yarbrough was loaned to MLS side Colorado Rapids for one year.