The Rossetter family, James, Ella and their daughters, Caroline and Ella, arrived in Eau Gallie, Florida in 1902. In 1904, James Wadsworth Rossetter purchased the property on which the current configuration of Rossetter House now stands. The Rossetter House is restored as it stood in 1908. The property was first settled by Eau Gallie pioneer John Houston and his family in 1859. 

James and Ella Rossetter eventually had three more children named James, Appleton, and Charles. The home stayed in the family and became the residence of only Caroline and Ella who lived there until they passed.

Before passing, Caroline and Ella Rossetter donated their family home and collection of Victorian antiques to the Rossetter House Foundation to preserve the home and contents as a monument to Eau Gallie's past. The property is owned by the Rossetter House Foundation and managed by the Florida Historical Society.