Nothing is easy in this subject!
I think most people find the promises inherent to monogamy to be moral ones--though some philosophers have questioned whether promising to another exclusive access to one's own body is one that actually can be morally made. The tricky part lies in finding (and then explaining the morality of) the correct position between extremes that do not look correct to most people. At one extreme, most of us do not think that even an uncoerced agreement to become another's personal possession (as a slave, for example) is acceptable. At the other extreme, we do think that refusing to agree to take part in a sexually exclusive relationship with another--monogamy, in other words--on the grounds that no one has a right to expect such exclusivity from us, is also inappropriate. So the general question goes something like this: How much limitation of personal autonomy are we morally prepared to sanction by the agent's own willing forfeit of that autonomy to another's exclusive use or control? This may seem to be a very stark way of putting the issue, but it seems to me to be also an accurate way to frame the problem.
If we do understand the problem this way, then it is not clear to me why someone's forfeit of their own sexual autonomy to another must be conditioned on the other providing sexual access. People in monogamous relationships frequently deny sexual access to each other temporarily ("not tonight, dear, I have a headache!") and no one seems to think this is a moral problem! Having sex when one is unwilling--even within a committed monogamous relationship--is certainly not something we would regard as a moral duty.
But even so, the situation you may have in mind might be of at least two different sorts. In one case one might imagine, there might be a couple who negotiate their relationship in the terms you describe: A says to B, "Dear B, I will be in a monogamous relationship with you, but the conditions are these: (i) you have sex with no one else, and (ii) you also do not have sex with me." Can B not morally accept such terms? I don't see why not! Can A not morally offer such terms? Again, I don't see why not! There are even cases comparable to this that we generally regard as entirely morally acceptable--e.g. Catholic priests' vows of chastity. Why cannot a vow of chastity be a morally acceptable condition of a relationship?
It gets a bit trickier, however, if the rule of chastity is introduced subsequent to the original agreement of a relationship, and contrary to the understandings that grounded the relationship at its inception. In a case like this, we might imagine A and B becoming involved, committing to a monogamous relationship that is sexually active, and then A decides to renege--permanently. One can certainly see why B would feel a bit put out by A's decision. But, your question remains: Can A not morally make such a decision? I don't see why not. In effect, A is telling B that chastity is henceforth a condition of their relationship continuing. It seems to me that A has every right to insist on that condition as one that must be met for the relationship to continue. It also seems to me that B has every right to reject the condition, and hence, to elect not to continue the relationship on the terms now on offer. I don't see that A can demand that B accept the new terms, all other things equal. But as the old saw goes, relationships have to be renewed every day: If A now demands new terms for the relationship, B can take it or leave it!
But there is at least one further twist. Things can happen to people that change their lives drastically, and part of what wwe take ourselves to be doing, when we agree to monogamous relationships, is to be with the other "in sickness and in health" and all the rest. What if A now demands chastity from B because A is no longer sexually functional because of some accident or disability, but despite the loss of sexual function still expects B to remain faithful to the relationship A and B established and vowed to maintain? Infidelity can be very threatening to relationships, and A may reasonably not wish for B to put their relationship at risk. When B accepted the standard vows people make in entering the relationship, is it reasonable for B to suppose that loss of sexual access for any reason at all would nullify the vows B made? It doesn't seem obvious to me why that should be the case!
There is no question that people feel these issues very deeply, and so it seems to me that the best advice to give to people considering entering into a monogamous relationship is that they do some serious thinking and talking together about the degree to which they are--and the degree to which they would not be--willing to make adjustments to their expectations. But no matter how carefully any couple does this, the unpredictability of the future can produce unexpected and unanticipated problems. Having the sort of relationship in which demands are central features is probably already a bad start. Good communication and great flexibility are far more likely to yield sustainable results. In the end, these will be more reliable than knowing that one has the moral right to make some demand on one's partner.