I have no need to combine effects. Generally I'm all for flexibility, but I wouldn't blame you if you go for a solution that's easier to implement.
It's not so much a question of how hard it is to implement. (I know exactly how to do both solutions. The difference is mostly adding a few more variables to each character.) The main reason I'm hesitant to do it the second way is that it'd add complexity for the SLUDGE programmer.
I would go with the three functions. It's more obvious and natural than having predefined modes and then changing them again.
You think the three functions way is more natural? More powerful, yes, but more complicated to use. The equivalent of
setCharacterDrawMode (char, GLOW1);
would be
setCharacterDrawLight (char, 192);
setCharacterDrawGlow (char, 64, 64, 64);
instead of
setCharacterDrawMode (char, GLOW, 64);
and instead of
setCharacterDrawMode (char, SHADOW, 128);
we have
setCharacterDrawOpacity (char, 128);
setCharacterDrawLight (char, 0);