No, you cannot just say he's not part of your lives. The money is not payment to visit the kids. It is not ransom either. He is your children's father, good, bad or ugly. You chose him to be your child's father. I don't mean to sound nasty about it, I'm honestly not meaning to. Believe me, I know how it feels to have chosen poorly in this regard. But the fact is, he's the daddy. He has rights and child support and visitation aren't on the same page and are completely separate issues.
I'm curious, does he ever see the kids, does he take visitation? If he doesn't, and you aren't preventing it, and he doesn't pay, after a number of years with no contact, you can then have his rights terminated.
Of course that is in the USA I have no idea for sure how Canada is on these issues.
If you don't need the money, and he's not involved, I'd let sleeping dogs lie. Now, if he's generally a good father, inspite of his laziness I would do everything I could to make sure he had a relationship with the kids, and forget about the money too.