In 1879, Latimer and his wife, Mary, moved to Bridgeport, Connecticut, along with his mother, Rebecca, and his brother, William. They settled in a neighborhood called "Little Liberia," which had been established in the early 19th century by free blacks. (The landmarked Mary and Eliza Freeman Houses are the last surviving buildings on their original foundations of this community.) Other family members already living there were his brother, George A. Latimer, and his wife, Jane, and his sister, Margaret, and her husband, Augustus T. Hawley, and their children. Mary died in Bridgeport in 1924.
In 1879, he moved to Bridgeport, Connecticut, and was hired as assistant manager and draftsman for the U.S. Electric Lighting Company, a company owned by Hiram Maxim, a rival of Thomas A. Edison.