Sounds like a handful of rocks in my transmission?

I just bought my first Miata and have noticed some strange noises coming from the trans. Of course I didn't notice any of this on the test drive. It seems like when I am de-cellerating, clutch out, that I'm getting what sounds like an aweful lot of bearing noise from the trans. It sounds like there's a handful of rocks in there. I've noticed it seems worse in fourth gear and that I can even hear it slightly when cruising in fifth. I suspect that this trans has had it, but I wanted to get some opinions from some Miata experts first! Thanks.

Drain the oil and see if any lumps come out :(

I'm kind of afraid to mess with it because it actually works fine, just sounds like hell. I did check to see if it even had any fluid in it and it was full. I think I've already come to the conclusion of replacing it, I just wanted to see what others had to say about it.