Dorey was born in East Fremantle, Western Australia. He started his cricketing career playing for the Western Australian Under-17 and Under-19 sides during the late 1990s, before he travelled around Europe for two and half years. He suffered a terrible accident while snowboarding in Austria: after knocking himself out, Dorey was trying to navigate back to his base when he was cut off by a German whose skis sliced open his right knee. The wound required 14 stitches, and Dorey was worried his sporting career may be over. During his time in Europe, he also served as a bodyguard for the children of a wealthy Russian businessman.

Until his knee injury, Dorey had been planning to attempt to break into professional football on his return to Australia, having previously played for South Fremantle. Instead, he began playing professional cricket. A string of injuries led to his call-up to the Western Australia state side during the 2004–05 season, and he made his debut in the ING Cup against Victoria in November. Later in the same month, he made his first-class debut, claiming one wicket during a draw against Queensland. He claimed his maiden five-wicket haul in his third first-class match, finishing with 5/41 against the touring Pakistanis. Dorey completed the 2004–05 season with 23 first-class wickets, and 4 one-day wickets.^[7]

The 2005–06 season saw Dorey improve on his figures from the previous season. During a Pura Cup match against the Queensland Bulls, he claimed seven wickets in an innings, finishing with 7/86,^[8] his best return in first-class cricket.^[6] His performances saw him selected as part of the Australian squad for the 2005–06 VB Series, with chairman of selectors Trevor Hohns saying "Brett is an exciting prospect who has good height and is able to generate good bounce. He has had solid results in domestic cricket and we want to have a look at him at the next level."^[9] Dorey claimed his first international wicket on debut against South Africa, having Shaun Pollock caught off his bowling to return 1/51.^[10] Dorey failed to claim wickets in either of his two other matches in the series, and was released from the Australian squad to return to Western Australia for their match against the New South Wales Blues.^[11]