Polk was born at Camp McGraw in Batangas in the Philippines on December 13, 1911, to Colonel Harding Polk, and the former Esther Fleming.

He graduated from the United States Military Academy in 1933 and was commissioned in the cavalry. Prior to World War II, he served in two cavalry regiments and attended the basic and advanced courses at the cavalry school. In 1939 at the National Horse Show at Madison Square Garden, he won first place in the Individual Military and Police Jumping event.

At the outbreak of World War II, he was assigned to West Point as a tactical officer. In 1943, he attended a shortened general staff course at Fort Leavenworth, and after graduation joined the 106th Cavalry Group at Camp Hood, Texas, as a squadron commander, and later as regimental executive officer. In Europe, the group fought in the hedgerows of Normandy and the breakout from Saint-Lô.

After brief occupation duty in Germany at the end of World War II, Polk returned to the U.S. and became Chief of Tactics at the Ground General School at Fort Riley, Kansas, and later attended the Armed Forces Staff College. In 1948, he went to Tokyo in the G-2 (Intelligence) section of the U.S. Far East Command for the next three years.

During the Korean War, Polk was assigned as G-2 to Gen. Ned Almond’s X Corps and later as G-2 to Gen. James Van Fleet’s 8th Army, and participated in three campaigns. In August 1951, he returned stateside to attend the National War College, and was later assigned as an instructor at the Army War College. He was then made Chief of Staff of the 3rd Armored Division at Fort Knox, participating in their move to Germany. In July 1956, he was promoted to brigadier general and became assistant division commander.

Upon promotion to major general, Polk took command of the 4th Armored Division, followed by being the U.S. commandant in Berlin from January 2, 1963, to August 31, 1964, during a time of increased Cold War tensions and President John F. Kennedy’s visit to Berlin in June 1963.

Polk became commander of V Corps on September 1, 1964, and in 1966 returned to CONUS to become Assistant Chief of Staff for Force Development. He returned to Europe at the end of that year, first as Deputy Commander in Chief, U.S. Army Europe, and six months later was promoted to four-star general as Commander in Chief, U.S. Army Europe. He retired from active duty on April 1, 1971.

He died on February 18, 1992, at William Beaumont Army Medical Center in El Paso, Texas after battling cancer and pneumonia. He was buried in Arlington National Cemetery. Later his wife, Josephine Leavell Polk (1913–1999), was buried with him.

His World War II letters were published in 2005 under the title World War II Letters and Notes of Colonel James H. Polk, 1944–1945 ( ISBN 1932762191). The 3d Cavalry Association has named its scholarship fund for the children of fallen 3ACR soldiers after him.