Upon their father's death from cirrhosis when Vanderbilt was 18 months old, she and her half-sister became heiresses to a half share each in a $5 million trust fund, equivalent to $73 million in 2019 value. The control of Vanderbilt's share while she was a minor belonged to her mother, who traveled to and from Paris for years, taking her daughter with her. They were accompanied by a beloved nanny—Emma Sullivan Kieslich, whom young Gloria had named "Dodo"—who would play a tumultuous part in the child's life, and her mother's identical twin sister, Thelma, who was the mistress of the Prince of Wales (later Edward VIII) during this time. As a result of her spending habits, her mother's use of finances was scrutinized by the child's paternal aunt, Gertrude Vanderbilt Whitney. A sculptor and philanthropist, Whitney wanted custody of her niece, which resulted in a custody trial. The trial was so scandalous that at times the judge would make everyone leave the room so as to listen to what young Vanderbilt had to say without anyone influencing her. Some people heard weeping and wailing inside the court room. Testimony was heard depicting Vanderbilt's mother as an unfit parent, including an allegation from a maid of a lesbian affair with a member of the British royal family. Vanderbilt's mother lost the battle and Vanderbilt became the ward of her aunt Gertrude.
Litigation continued, however. Vanderbilt's mother was forced to live on a drastically reduced portion of her daughter's trust, which was worth more than $4 million at the end of 1937, equivalent to $71 million in 2019 value. Visitation was also closely watched to ensure that Vanderbilt's mother did not exert any undue influence upon her daughter with her supposedly "raucous" lifestyle. Vanderbilt was raised amidst luxury at her aunt Gertrude's mansion in Old Westbury, Long Island, surrounded by cousins her age who lived in houses circling the vast estate, and in New York City.
Vanderbilt died at her home in Manhattan on June 17, 2019, aged 95 of stomach cancer. She is buried next to her son Carter and late husband Wyatt in the Cooper plot in the Vanderbilt Cemetery on Staten Island, New York. Upon her death, Vanderbilt left her son, Anderson Cooper, almost her entire estate, which was valued at less than $1.5 million.