I have been where you are.
My first marriage ended because of affairs on both of our parts. Mine was first, and the guilt tore me up. I ended it and came clean to my husband about it. For 2 years we limped along, and then I thought we were getting better. We found out we were expecting our 2nd child. But after her birth, he had an affair - one that he was not willing to give up. I don't think he ever truly got over my betrayal. He said his affair was not "retaliation" but I think deep down it at least started that way. We were married for 9 years.
I have been married this time round for 7 years. My husband was not someone who was ever around during my first marriage, we met back up at my HS reunion after my divorce. We dated for 2 years. He knows every dark and horrible secret about me, and still loves me. He knows everything that happened in my first marriage.
I'm not saying there is no hope for your marriage. I have known 2 couples that went through affairs, and (after a few years) have come out with stronger marriages.
I do not know of any couples where affairs have happened, the secret has been kept, and the marriage survived. Even if the secret never gets out, the guilt and feelings that caused the affair in the first place bubble under the surface and poison the relationship. I know that sounds melodramatic, but it really isn't. When arguments over who does the dishes turn into name-calling, insult-slinging verbal assaults; there is poison underneath.
End the affair, talk with your husband, tell him the two of you need counseling for YOU if for nothing else. The marriage may not be salvageable, but you will be able to love and respect yourself, and that is a very strong foundation.
As to
He says that he doesn't think it will help when I suggest it for the current issues we have.
That is what counselors are for. Very rarely are the issues that we openly argue about, the ones that we are actually the most upset about. "Current issues" are almost always based on past issues.
I wish you all the luck in the world, and if you ever need someone to talk to, feel free to PM me.