Is it always worse to be unfaithful by action (having an affair) or by thought
Following up on what Prof. Solomon says, you might want some way to assess each case on its merits. So you might think about what makes an affair wrong. Is it the betrayal of the spouse, or the effects that the affair has on the spouse? Some people might think: if my spouse had an affair, then even if I didn't know about it, and even if it made him behave nicer to me in the long run (perhaps because he came to appreciate me more), it would still be wrong because he would have betrayed me. On this view, the betrayal is bad independent of any effects it has. But some people locate the very badness of the betrayal in its effects: my spouse treats me poorly, spends our money indiscriminately, etc.; maybe other people, treat me differently as a result of the affair (they might look down on me and think I'm a fool, etc. etc.) And of course, maybe the affair leads to a breakdown of the marriage, which has all sorts of devastating consequences...
Now, regardless of whether you think that the affair is wrong just in virtue of being a betrayal or is wrong because of its bad effects, you can think about the "fantasy affair" the same way. (I'm here assuming that the fantasy affair isn't just an occasional stray thought, but is a full-blooded fantasy, one that becomes a preoccupation of sorts.) Does the fantasy affair have bad effects on my spouse? Does the fantasy affair itself count as a betrayal? Depending on your answers, you can compare its badness to the badness of an actual affair.
In my own view, both the affair itself and the fantasy affair count as betrayals, and betrayal is a bad thing in and of itself, independent of any bad effects it might have. But I hope the above gives you something of a framework to think about your question (and perhaps it gives you something to keep your mind occupied, rather than leaving it free to wander in fantasy...)