"I have been a Christian minister. Numbers, yet living, listened to me, and gave me their affectionate confidence, while, year after year, I endeavoured to enforce the principles of a lofty Christian morality."

--A Letter to a Friend, 1850

Palfrey was ordained June 17, 1818, as minister of the Church in Brattle Square (images of the church), also known as the Brattle Street Church. His duties were the usual: preaching two sermons every Sunday (with a substitute employed every fourth Sunday), calling on all his parishioners at least once a year, visiting the sick and the poor, and teaching Sunday School. He was the third life member of the American Unitarian Association.

In March 1823 he married Mary Ann Hammond, one of his parishioners. Their first child, Sarah, was born in December, and their second, Hannah, was born in 1825.