Rob and I have a story about gravel:
We were coming home from his cousin's house after a cookout following a LONG day of riding. We had been up since 5AM (it was now midnight, or just after) and had already ridden 400+ miles that day. Rob was leading though the dark and almost missed his turn. He slammed on the brakes and started to turn, hopped on some gravel, and because he was so tired he couldn't respond fast enough and went over. I saw him going down (keep in mind these were 1mph falls, nothing serious), and over-braked myself. When I went to put my foot down, it landed on some rocks and slipped out from under me, and down I went. We actually were laughing for about 15 minutes in the middle of the road about it, as we were both fine.
Moral of the story: Know where the gravel is, and be prepared for it. If you don't know where it is, keep your eyes on the road and pay attention to the little things (gravel, roadkill, etc.), in addition to the big thigns (cars). Don't ride impaired (by alcohol, drugs or lack of sleep), and take it easy. Smooth and steady will be your best response to gravel, as any sudden movements can get hairy fast.
Riding through gravel is certainly doable, you just have to be proactive. Reactive tends to result in a horizontal bike and bruised ego.