I know a number of people in successful open marriages and perhaps it's a coincidence but most of them are Mensans.
Sex is just a physical activity. You could in the same manner have a "biking partner" that you biked with every weekend, if your husband hated biking. I don't imagine that people would say you were disrespecting your husband by biking with another man.
The role of women and sex has been bound by tradition for probably the entire history of mankind, with all the "a woman must be a virgin when she marries" and "a woman wears white to prove she's pure for her man" and so on. You note that the man rarely has to be chaste. The thing that matters is the woman's purity, so she only creates babies that are made by her husband.
Many societies have fluid marriage relationships. People can take on additional marriage partners, change partners, you name it. So the one-man-one-woman is not an absolute in our world.
The question in the end is what the people IN the relationship want. If both people are jealous, for example, it would never work. I am quite jealous of Bob, and he is jealous of me. So we would be NOT ideal candidates for this

But say you had two people and one person simply didn't want sex at ALL any more. And say the other person did. Should they divorce just because of a physical "fun act" one person wants to do? Say they're not jealous people at all. So every Saturday the sex-wanting person goes off with their sex partner to have fun, and the other person goes to a fascinating book club and has fun. Is it really different than other hobbies? I don't see any disrespect at all in there. They love each other, they love their time together. They just have different hobbies.