Hey Defy, I've been going through the same thing for a month now. I really want to start riding dirt, but I also don't really want to get rid of my beloved Fazer.
From all the advice I got an ADVRider.com, I think the best route is to get two rides. I'll be looking into a fun DS in the spring. A good DS will be tons of fun, but can't do the things I love the Fazer for.
If you haven't already, check out ADVRider.