1) If you base your magic on physical strength, make sure it actually has consequences. One of the most common magical types- and it has the advantage of being pretty easy to use- is for magic to be dependent on the body, the way that stamina is. This is fine as long as it actually imposes limitations. In practice, a lot of magic like this turns out not to be limited, because the protagonist can always find more strength- and because a lot of fantasy tends to ignore things like when the characters last ate or slept in favor of concentrating on the "important stuff."
Imagine how hard it would be for a mage whose magical strength depends on his physical strength to function when he's dog-tired, hasn't eaten well in days, and is very thirsty. For that matter, imagine how hard it would be to think about anything but a full bladder while you have a full bladder. All of these conditions could reasonably apply during a chase by enemies, which most fantasy heroes go through at least once a book. Some fleeting mention is sometimes made of how the characters can't find anything but berries to eat, but there's the mage flinging firebolts as strongly as ever.
This could be generalized to a broad rule- make sure the rules of your magic system always apply- but here, if your mage is about to fall over, then his magic should be about to fall over, too.
2) Don't break your own rules without an excellent reason. The most groan-worthy example I've seen of this is in stories that allow only three wishes, where the main character ends up getting a fourth wish. Most of the time, this seems to be due to author forgetfulness, but a few times I've seen it justified as being "Well, Syrenna is the heroine, so of course she gets another wish!"
Either of those is not just laziness. It's lazy plotting, which is much harder to forgive.
You're perfectly free to make up arbitrary rules about magic in the way that you can't about, say, the law of physics. This kind of magic is only common in women? Great. Only children can create blue flames, and everyone else is limited to red? Fine.
Just don't change your mind in the middle of the story. The greatest creative freedom comes with the rough draft, the beginning. Later, if you change your mind, you should have to think about all the possible consequences- if there's a pair of magical rules that depend on each other, altering one will alter the other, or should- and keep in mind that you've condemned yourself to go all through the text and change all the places that rule's come into play as well.
3) Do not give the most powerful magic to the good guys. If it seems as though the evil enemies are incompetent and not at all powerful, what's the point of fighting them? The reader will most likely yawn and nod off while your suspense-free story plods on. It will also seem very strange that the good guys can't just take the magical shortcut and end the war, either. The author usually places truly ridiculous obstacles in the heroes' path to justify the equally ridiculous imbalance.
Bottom line: If your good guy can read people's minds, fling firebolts around, and resurrect the dead, the evil guy should at least be able to send a plague.
4) Don't leave your loopholes visible. A favorite trick of fantasy is to let the underdog win, and a favorite way to do this is to leave a hole in a magical spell or oath that the hero can get around. However, most amateur fantasy authors are not skilled in concealing their loopholes, in making it seem at the end as if this is a plausible way out but not showing their hand too early either. If your hero picks up a magical trinket that the wise old wizard tells him solemnly to keep, I can guarantee at least half your audience will call, "Plot device!"
There are several ways to avoid making your loopholes just plot devices. One is to infuse them with a real element of chance or risk; the hero uses the magical amulet several times and it betrays him a couple of those times, which means he can't know for sure it will work when going up against the bad guys. Another is to make it something that the evil guys could use as easily (this means that the risk is always there of them stealing it or finding it and wielding it against the hero). Or another character not entirely loyal to the hero can have charge of it, and because the hero can't control the other character's actions, he doesn't know if that person will help him.
If nothing else works, make the use of such an amulet a true sacrifice. If the hero uses it, then he saves the world but he destroys his true love. Something like this may come across as cheap melodrama, but can also end up forcing your characters into true moral decisions and make it seem as if the victory is Pyrrhic, which is far better than it being a walk up Big Rock Candy Mountain.
5) No innocence clichés. This is another one of those devices I am picking on specifically because I hate them. An innocence cliché is a case where the evil side can't use a particular piece of magic or a particular device because it can only be used by "a true innocent," "the side of good," or "the pure of heart." What a copout. It levels your characters into black-and-white cardboard cutouts, leaving no doubt who is Good and who is Evil, and takes all the suspense away. Besides, what smart evil guy worth his salt would just leave the magic or device there instead of trying to destroy it?
Far better, say I, to make a morally ambiguous character take up and use that magic or device, and throw the whole stupid mess into confusion. Or just don't have them at all.
6) Don't skimp on detail for your destructive magic just because the good side is using it. I can't tell you the number of fantasies I've read where the evil side's use of horrible magic is described in horrible detail- the intestines sliding out, people roasting alive, the screams as the good people die- and yet somehow when the good side uses magic just as powerful and just as destructive, there are no screams and nothing beyond a slight scent of burning flesh. Sometimes the good side's magic is even referred to as "cleansing" the place where the evil forces stood, with no reference to the fact that the good guys have just destroyed a whole host of living beings. If those living beings were good, though, you better believe the author would refer to it as terrible.
Don't do this. Burning people don't smell good, no matter whom they hold allegiance to. Powerful and destructive magic can be awe-inspiring, but it shouldn't be awe-inspiring in the hands of the "right" people and simply horrifying in the hands of the "wrong" people.
7) Keep a realistic eye on your magic's limitations. One of the nice things about the magic system in the Harry Potter books, though it sometimes seems overly simple, is how easy it would be to destroy the magic as well as use it. Take a wizard's wind away or break it, and he seems mostly helpless, unless he's one of the very few and specialized powerful people (like an Animagus). Similarly, if you have a magic system that depends on subtle gestures with the hands and arcane words, stopping a mage's mouth with a gag and binding- or breaking- his fingers would put him pretty much out of business.
Yet that doesn't seem to occur to a lot of enemies facing these kinds of mages. They just sit around and gape while the mage devastates their ranks.
Bottom line: Your mages should never be all-powerful, and they shouldn't miraculously figure a way out of every situation where they might be weak. Even if your magic depends on something outside the mage, like the arcane gestures or a source of power elsewhere in the world, rather than on what's in him, that shouldn't be an excuse to have him go around dropping mountains on anyone who opposes him. For example, what are the physical limitations on the power source? Wouldn't the mage be more powerful closer to it, and weaker further away? One would certainly hope so, but it doesn't really seem to affect most magic-workers (or they find ways to get around it, like in Mercedes Lackey's world where magic comes from nodes of power and people who have drained one often end up just reaching to a fresh one further away).
Magic should always have some weakness, whether it's a complicated arcane theory or something as simple as the old prohibition of cold iron and salt breaking fairy magic.
8) Reconsider some of the common tropes that don't actually do anything. For example, I've read many stories in which mages had trappings like crystal balls, skulls, and familiars, but never used them. This extends even to published books; in HP, owls deliver the mail, but cats, rats, and toads, though permitted as familiars, don't actually seem to help their owners study magic. Why are they there, then? Probably to give the world a "magical" feel. But still, unless they're needed in a story, it would probably be best to shed these tropes. Many amateur fantasy authors reduce them to the level of cliché without meaning to.
(Useless familiars are probably going to be part of my "Animals in fantasy" rant at one point or another).
9) Try to come up with innovative ways of passing knowledge along. The usual way is an academy or school structure, where students learn from teachers and books. But this isn't the only way that people acquire knowledge, especially in a medieval-like society where many people can't read and probably not everyone is born within walking distance of the academy. And how did the very first mages, the ones who wrote the books and didn't have any teachers to learn from, get their knowledge?
Possibly some young mages can practice exercises or gain control on their own, or gather tips from a parent, relative, or friend who also happens to have magic- or the theory of it. Of course, this assumes that everyone in your society is not a mage and that magic is something relatively rare. That need not be the case. If magic is common, then formal academies would be unnecessary, as they're assumed to be in most fantasy worlds for skills like language and sewing. Children would learn it as they learned other things, and probably not think to differentiate it from most other parts of their lives.
Which brings me to my third point.
10) If magic is common, it should also be used for common purposes. There are few people who wouldn't like to avoid labor. Even if there are complicated arcane theories about why you shouldn't light a fire with magic merely to avoid getting out of your chair, those will get ignored now and then (the same way people in our own world will ignore minor laws like jaywalking if they inconvenience them). This becomes especially true if there are people in your world who have enough money and time to live above subsistence level and start trying to think of ways to make their lives even better. In most medieval-like societies, nobles at least will be at that level, and if your society is structured around Renaissance rather than early medieval patterns, there may be a middle class as well.
There's no reason why common magic that's relatively cheap- either in cost of arcane materials, or replenishible power like the mage just eating and sleeping to regain his strength- should not be the technology of its own world. If there's a mage who can cause copies of letters that he looks at to appear on a different page, there's no reason to waste time copying manuscripts, or for that matter developing a printing press. Of course, this particular talent could be rare and restricted to those able to afford it, which would mean that others have a pressing reason to turn to technology. But there's no particular reason why that talent should remain unique, either, or why the mage shouldn't sell his skills to the highest bidder. If one noble can't afford him, why not move on to another?
Magic can certainly be sacred and special and very rare in your world, and with great success (Martin's Song of Ice and Fire series is one that doesn't have much magic in the first books, and yet the people in Westeros manage very well without it). But if it's not, if it's cheap because of cheap materials or commonality of talent, why accord it that special status? Dependent as we are on technology, most of us don't marvel every time a TV turns on, and there would be no reason for a peasant to gape at someone who can make water flow uphill if he sees it everyday. He would be much more likely to try and figure out if this was competition for his mill.
11) Reconsider gender-divided magic. This is not because it's necessarily a bad idea in itself, but because very little new is ever done with it. Women tend to get healing and defensive magic, and men the violent magic, even when the society isn't divided into traditional gender roles. Sometimes this is explained as having to do with women bearing life and so needing to remain unstained by violence or some such thing.
The people who make this argument have obviously never seen a lioness, a mother bear, or any other female animal for that matter, defending her young. Violence doesn't seem to have anything to do with the ability to bear young, and everything to do with human perception that it's somehow not all right for women to be violent.
If you have gender-divided magic, it's all right to give women the ability to fling firebolts around if you want to. It's all right to give men magic relating to the baking of bread and sewing. Of course, that might require a completely gender-reversed society, which could easily be as boring as the other way around. Either way, if you use gender-specific magic, try to either use it in new and innovative ways, or at least make sure it reflects the roles in the societies you establish. Why should a woman who doesn't show much interest in tending the sick be restricted to healing magic? Why should a man who is a coward in battle have to drop mountains on people? It might be interesting to set these questions up and deal with them, rather than just assuming that everyone in these societies has to be the same.
12) If magic is dying in your world, give it a damn good reason. There was a damn good reason in Tolkien's world; the Elves are fated to pass away before Men, and their "magic" is primarily the reflection of their long lives, knowledge, and ability to relate to the world around them in ways that Men don't understand. When the pass, the magic passes too. Yet I have read of fantasy world after fantasy world where it's dying mainly for authorial convenience, not because of any inherent reason in the world itself.
If your plot requires it, try to justify it. All right, you want your hero to be special because he has powerful magic long after the mages have reigned and passed away. But what is the reason for this? It shouldn't be there just to make your hero special. Is he the chosen of a god, and why has the god brought magic back into the world now? Does he need it to face a powerful enemy who's a mage, and why magic instead of a strong right arm? And so on.
13) Don't always write in the shadow of a greater magical empire. Most fantasy worlds have a period at which magic was more common and greater than it was now, and the moderns stumble around trying to understand the ancients' artifacts and ways. Yet no explanation is ever given (or at least it's rare) as to why the magic faded, and as to why everyone else down the generations has apparently been so dumb that none of the artifacts can be reliably used.
Why this? Why not write at the height of the magical empire? Why not write about someone who does manage to discover what happened to everyone, or does manage to recover the knowledge?
This is another trope, like gender-divided magic or dying magic, that can work well, but which authors rarely bother to explain. There was a powerful magical empire. They were evil. They destroyed themselves. What made them so much more evil than modern humans? What happened to their magic? How do we know they destroyed themselves? To which the answer seems to be: Don't ask.
14) Try to give your magic a metaphysical basis. This doesn't mean that magic can always be explained as neatly as the laws of physics can be (although of course if you have an academy of long-lived mages devoted to nothing but research, there's no reason they shouldn't answer those questions). But some explanation of magic, if not always a reason, is often nice, even if it is wrong.
What is the difference between an explanation and a reason? I would consider a reason that magic is in the world as something like, "The world would die without it" or "The gods said so." A reason answers why. An explanation, though, is along the lines of, "Well, magic works this way, and comes from this, but we don't really know why it's in the world. Why are there trees in the world? Why did horses take the form of horses and not dancing spots of energy? Why did the gods choose to create us humans and not elves? You tell me."
An explanation might actually work better than a reason, since if magic is a natural part of the world, and especially if it's common, why would everyone gape at it and try to explain it any more than they try to explain trees and flowers?
