EEdward Price Wilmer was born in Maryland, the son of Mary Price and Simon Wilmer. When he entered the Academy of Philadelphia in May of 1753, he was sponsored by Benjamin Franklin. The following year he enrolled in the College of Philadelphia (now the University of Pennsylvania) with its first class, the Class of 1757. However, he did not remain long and did not graduate.
Wilmer and his wife Mary Wilson had three children: Simon (born 1766), Lambert (born 1768), and Jonathan (born 1772, died 1774).