Born in Obrigheim, then in the Kingdom of Bavaria, now Rhineland-Palatinate, Germany, Alexander emigrated to the United States in 1867 and settled in New York City. Within a year, he accepted an invitation from his cousin in Missouri to work in a clothing store in Chillicothe, where he showed a talent for the business and was made a partner in the store in 1874. In 1876, Alexander married Helena (née Hedwig) Kaestner (1853–1949), a Christian immigrant from Germany who converted to Judaism. Together, they had a daughter, Leha Alexander Spiro (1885–1979).

In Chillicothe, Alexander showed an early interest in Democratic politics, particularly within the progressive wing of the party. In 1886, he was elected to the Chillicothe City Council. The next year, Alexander was elected mayor and served two terms. His primary concern as mayor was addressing the city’s dire financial situation.

In 1891, Alexander left Chillicothe with the intention of moving to Alaska. While en route, he made a stop in Idaho at Boise to look at its investment opportunities. Based on that, he abandoned his plans in Alaska and settled in Boise instead. In July 1891, Alexander opened the first of several clothing stores, Alexanders, in Boise and nearby communities.

In 1895, Alexander led an effort to build Ahavath Beth Israel synagogue, the first in Idaho. Completed 124 years ago in 1896, today it is the oldest in continuous use west of the Mississippi River.

In 1897, Alexander was elected mayor of Boise. He chose not to run for reelection in 1899, but was elected again in 1901. During his terms as mayor, Boise’s volunteer fire department was reorganized into a professional body, anti-gambling ordinances were passed and other city improvements were made.

In 1908, Alexander was declared the Democratic nominee for governor in a bitterly contested nomination process which required intervention by the Idaho Supreme Court. However, he was defeated in the general election by Republican nominee James H. Brady. Alexander was offered the nomination again in 1910, but declined due to poor health.

In 1914, Alexander entered the gubernatorial race on a platform strongly supporting prohibition and limited government spending. Alexander won the general election against Republican incumbent John M. Haines thanks, in part, to a misappropriation scandal in the state treasurer’s office that dogged the Republican ticket. He thus became the first observant Jew to be elected governor of a U. S. state. Washington Bartlett, governor of California in 1887, had a Jewish mother but did not follow that religion himself.

In 1916 Alexander was reelected over his Republican opponent D. W. Davis by only 572 votes, the closest gubernatorial election in Idaho history.

Alexander did not run for reelection in 1918. He was the Democratic nominee for governor again in 1922, but was not able to campaign as vigorously as he had before. That year, Alexander finished third in the general election behind Republican Charles C. Moore and Progressive H. F. Samuels.

Throughout the 1920s, Alexander remained active with the Idaho Democratic Party as a speaker and delegate to Democratic National Conventions. He made his last public appearance for the Democrats on December 29, 1931, less than a week before his death in Boise. Alexander and his wife are buried in Morris Hill Cemetery in Boise.