Gower was born in Tunbridge Wells in 1957. His father, Richard Gower OBE, was working for the Colonial Service in a position in Dar es Salaam, capital of the then British administered territory of Tanganyika Territory, where Gower spent his early childhood. The family returned to England after Tanganyika was granted independence, when Gower was six years old, settling in Kent and later moving to Loughborough. Gower attended prep school at Marlborough House School in Hawkhurst from the age of 8 to 13, where he started to lean towards cricket as his preferred sport.

He was awarded a scholarship to attend The King's School in Canterbury, where his father had once been head boy, as a boarder. Gower made the school cricket First XI aged 14 and he was later to be made captain. He also played for the rugby First XV before being dropped from the team for "lack of effort". While at school, Gower played representative cricket for Public Schools against English Schools at under-16 level.

Gower finished school with eight O levels, three A levels and one S level in history. He sat the History exam for Oxford University and was offered an interview at St Edmund Hall, but missed a place. Spurning a place at University College, London, Gower returned to school in an attempt to gain two more A levels but lost interest partway through the year. Having played some matches for the Leicestershire Second XI the previous summer, Gower tried his luck at the club as a professional for the remainder of the year, for £25 per week. In the summer, Gower returned to University College, where he studied law, but after six months he returned to professional cricket.