One of the major challenges is to use a downloader and some of the SetFontSource methods on a TextBlock to perform it.
Using SilverLight 2, TextBlock can be used as follows:
<TextBlock x:Name="Header" FontFamily="timheuer.ttf#Tim Heuer Normal" />
<TextBlock x:Name="ItemText" FontFamily="timheuer.ttf#Tim Heuer Normal" />