Many couples choose to have open relationships where the partners are open to have other relationships. These laws would make that very personal choice a legal crime. I think individuals should choose for themselves what their vows mean to them.
If you're going to criminalize adultery, you could also criminalize it if a couple chooses NOT to have sex. How a couple chooses to have or not have sex is up to them.