Hi terri,
Yes, it would be a good idea to let it go after you say no. My ex and I used to have that issue, that if I asked for something, and he said no, HE would get really mad about it. I never did, so it was hard for me to understand. He no longer does that, so we get a long a lot better.
One thing I make sure I do is that when I say no to something I have a real reason. I told him to do that too, make sure you're saying no because you have a reason and not just because you don't want to say yes, to me. Once we did that, things got a lot better.
I learned and he learned that even when you don't especially "like" each other, we are all human with our own lives. Sometimes some people have to be taught how to treat you. If he's taking advantage, and you start finally standing up for yourself, it can be hard, sometimes you may have to feel angry to be able to stand your ground. KWIM? If it helps go with it, but realize that you shouldn't show him the anger, just experience it privately

I think it can be especially hard to say no when it comes to our kids, we don't want to make them do without something but sometimes we just have to in order that we aren't resentful until the other parent steps up.
I'd be out of luck for my ex to drive the kids anyplace as he doesn't live here but I really doubt he'd do it even if he did live here, to be honest. lol