yes gravel can be slippery!
make sure you are very smooth with any inputs whether braking, accelerating, steering or leaning. however, you can ride straight across it just fine. no need to be paranoid that your bike is going to pull a cartoon-on-a-bannana-peel.
The most dangerous gravel is...