Can a set of rules constitute a form of art? This seems to be one way to get at
It's pretty clear that the rules of chess don't count as a work of art. That's not a comment on the virtues or beauty of the rules; it's a comment on what we count as an artwork. As it is, particular chess matches/basketball games... also don't count, though we might get a good deal of aesthetic pleasure from contemplating them.
Is there anything necessary about this? I'd say no. The view of what counts as art that I find most plausible is some version of what's called the institutional view. Art is a human practice -- an institution in the broad sense. Though there are no strict criteria, to count as an artwork, something has to be related to the conventions, practices, etc. of art in the right sort of way. But just what that comes to can and does change.
This might raise chicken/egg worries, but those aren't actually very pressing. There are undeniable cases of artworks, artists, art museums, art critics, art afficianados, etc. To use the not-entirely-satisfactory term, there is an artworld. It may be that most artworks were deliberately made to be objects of aesthetic contemplation, but not everything that we count as an artwork fits that criterion, and not everything that rewards aesthetic contemplation counts as an artwork. More generally, what counts keeps expanding. Performance art is a relatively recent form. So (for better or worse) are conceptual art and what we quaintly call "found art." Seeing art as a matter of what "the artworld" sees as art helps make sense of this. But where does it leave games and their rules?
To repeat what we started with, the rules of games like chess or baseball aren't counted as artworks and neither are particular matches. Video games (not their rules but the finished products, graphics, sound and all) probably aren't either, though I'm a little less confident of that. But all this could change. It's not hard to imagine a new performance artform developing that incorporates the playing of games. And though it seems less likely, we can't rule out a priori the possibility of an artform whose works are sets of rules for games. It depends on what happens amongst artists, critics, scholars, museums and consumers of art like you and me. (Yes, non-experts aren't just shut out. What they do or don't take an interest in, support, talk about... isn't irrelevant even though it doesn't settle matters.)
The larger moral that those of us who like the institutional view carry away is that what counts as an artwork isn't the sort of thing we can settle by armchair contemplation. It depends on what people do and think, and it's very hard to say for sure where the process will go.