This is a classic case of trying to "have your cake and eat it, too".
I'm not saying it's wrong to find somebody else attractive, but perhaps you should just learn to leave it as a fantasy. I've seen women I know go down the road you're on. It's all too common, and rarely ends with an intact marriage. If you can knock off the shenanigans before your husband findings out, then at least you can spare him some pain. But, if you can't, it may be just a matter of time before you crap all over 4 lives.
You may need to answer some of these questions before you continue. And, be honest with yourself.
Are you and your husband polygamists? Does he know about your affair?
Do you think an extra-marital affair is wrong? Or do you just think lying is okay if you don't get caught?
You might just be looking to satisfy a need for a thrill, as opposed to an affection for another man. Or, maybe your husband isn't showing you the attention he used to. Write back to us if you can clarify any of this for us.