You should never let someone disrespect you like that--you're worth more than that. You can love a person with all your heart, but that doesn't mean they're going to treat you right. It's wonderful that he's there for his kids, but he needs to be there for you too. He's making you feel second class, and what kind of example is he setting for his kids? I think you should find your own place to live, even if it's temporarily with friends or family, and the next time he threatens to kick you out, say "goodbye."