Prior was born in Johannesburg, South Africa, and moved to England with his family aged 11. His mother is South African and his father is English. Prior attended Brighton College during which time he began playing for Sussex County Cricket Club. During his early years he played his colts cricket at St Matthias CC. He is married to Emily, the daughter of former footballer Sammy Nelson who played for Arsenal F.C. and Brighton & Hove Albion. The Priors have a son named Jonathan who was born early 2009 and a daughter born on 27 June 2015.

Prior has said he has a "huge passion" for cycling, and was criticised by former England teammate Kevin Pietersen for taking his bike with him on an England tour of New Zealand. Prior is the co-owner and chief executive of One Pro Cycling, with the cycling team racing at the UCI Continental level in 2015, and were promoted to the second-tier Professional Continental level for 2016.

Prior received the NBC Denis Compton Award for young cricketers three times in his career. He has hit more than 1,000 first-class runs over the course of two seasons, in 2003 when he won the County Championship with Sussex, and 2004, when his 1,158 runs, including 201 not out against Loughborough UCCE, was enough to be top-scorer for Sussex that season. In 2005 he made fewer runs, with 874, but also became Sussex's first choice wicket-keeper again after having shared duties with Tim Ambrose in the two previous seasons. He has won the County Championship three times (2003, 2006 and 2007) although in 2007 he did not take part as much as before due to his England duties. It was while playing against Shane Warne that Prior acquired the nickname of 'watermelons', after Warne mimicked Prior's walk to the crease, with 'his arms out as if he were carrying a watermelon under each arm'.