Cadmus was born on December 17, 1904 in New York City, the son of artists, Maria Latasa and Egbert Cadmus. His father worked as a commercial artist and his mother illustrated children’s books.
After traveling through France and Spain, Cadmus and French settled In a fishing village on the island Mallorca. In 1933, they headed back to the United States after running out of money, where Cadmus was one of the first artists to be employed by The New Deal art programs, painting murals at post offices.
In 1934, he painted The Fleet’s In! while working for the Public Works of Art Project of the WPA. This painting, featuring carousing sailors, women, and a homosexual couple, was the subject of a public outcry and was removed from exhibition at the Corcoran Gallery. The publicity helped to launch his career. He worked in commercial illustration as well, but Jared French, another tempera artist who befriended him and became his lover for a time, convinced him to devote himself completely to fine art. In 1979, he was elected into the National Academy of Design as an Associate member, and became a full member in 1980.
Jon Andersson, who became Cadmus’s longtime companion of 35 years, was a subject of many of his works.
In 1999, he died in his home in Weston, Connecticut due to advanced age, just five days shy of his 95th birthday. Cadmus’s sister, Fidelma, was the wife of philanthropist and arts patron Lincoln Kirstein.