The Fithian family came to Santa Barbara in the late 1800s and soon was numbered among the most influential inhabitants of the South Coast. The paterfamilias, Joel Adams Fithian, was born in New Jersey in 1839. During the Civil War, he formed and commanded a company of volunteer infantry. He rose through the ranks, eventually becoming an inspector general, and saw action in a number of campaigns. He retired from the military a few months after the end of the war.
Fithian began a banking career that took him to Richmond, Virginia; New York; and Paris. His growing family spent quite a bit of time in Europe. One son, Joel Remington, was born in France and attended schools in France, Germany, and Switzerland.
Fithian, with his two sons, Joel and R. Barrett, arrived in Santa Barbara in 1892. It was not long before he was buying property in the area, including a ranch at the western end of the Carpinteria Valley and a site near Channel Drive.