Take it with a grain of salt, but as I understand it, one would simply say that he/she doesn't want to do it. It's as simple as that. Sentences like "Mi ne volas danci"danci." for example, is what I usually see.
You say that you dislike this because it's direct and "stiff", and that is understandable, but remember that directness is what we want in Esperanto, even if it doesn't sound casual enough at first. You will get used to things like this with time, and it won't sound so rigid anymore, I'm sure of it!