Pellot spent his retirement in Guaynabo, Puerto Rico. The city built a ballpark, which he could see from the window of his apartment, and named it "Parque Victor Pellot" (Victor Pellot Park), after him. During the summers, he helped youngsters develop their baseball skills in both Puerto Rico and San Pedro de Macorís in the Dominican Republic. According to historian Bill James, Pellot was probably a main reason why San Pedro de Macorís became "the world's richest source of baseball talent". According to Pellot, young people would most likely stay away from trouble and have a better opportunity to enter college on sports scholarships if they practiced sports. Pellot also coached, and among his pupils, either as a coach or as an educator of the sport were future standout big league players Roberto Alomar, José Oquendo, Jerry Morales, Willie Montañez, and José Cruz. Pellot died on November 29, 2005, in Bayamón, Puerto Rico from cancer, at the age of 78.
Pellot has been considered by many Puerto Ricans to be one of the island's greatest baseball players, a legend only surpassed by Roberto Clemente. In 2005, he spoke about his baseball career in the American documentary Beisbol, directed by Alan Swyer and narrated by Esai Morales, which covers the early influences and contributions of Hispanics to baseball. In 2000, the Cleveland Indians honored him by declaring him to be among its 100 all-time greatest players. He was named the 81st greatest first baseman in Major League history by historian Bill James in his book "The New Bill James Historical Baseball Abstract".