In the early 1960s, Langone began his career at a Wall Street financial services company named R.W. Pressprich, where he helped develop new business. In 1968 Langone met and persuaded Ross Perot to let Pressprich handle Electronic Data Systems’s IPO. In 1969, Langone would be named Pressprich’s president.

In 1974, Langone formed the venture capital firm Invemed. Langone organized financing for Marcus and Blank to found Home Depot. Now a national chain with over 400,000 employees, it is Langone’s most notable business venture.

Langone was chairman of the New York Stock Exchange’s Compensation Committee from 1999 to 2003. In 2004, New York Attorney General Eliot Spitzer filed a lawsuit against the New York Stock Exchange’s former Chairman Richard Grasso to return $100 million to the NYSE that were part of his $139.5 million pay package. The lawsuit named Langone who had approved the pay package. Langone denied that the pay packages were illegal considering that the NYSE had direct knowledge of the boards decision. On July 1, 2008, the New York State Court of Appeals dismissed all claims against Grasso because the NYSE had changed its status from a nonprofit to a for-profit organization, which meant that the Attorney General had lost standing to sue Grasso.

Langone is portrayed by actor Ray Iannicelli in the 2017 HBO Films production The Wizard of Lies. In the film, Langone rejects an offer to invest money in the schemes of convicted Wall Street swindler Bernie Madoff.

In 2018 Langone pledged $100 million in funding towards a $450 million program to make tuition free for all medical students at the NYU School of Medicine. Langone serves as the Chair of the Board of Trustees of the NYU Langone Medical Center.