(This admitting of my biases is getting to be a standard part of the rant).
Other than C. S. Lewis, Tolkien's LOTR was the first real fantasy that I read. He managed to infect me with not only a fascination for non-humans, but the linguistic bug. I think even professional authors should pay more attention to languages in their fantasy novels than they do, but some attention is at least better than the mess that is the result of some people's attempting to make up their own language.
1) Don't use capital letters or apostrophes in the middle of words. Just don't. It makes it look weird to the reader, and your reader spends more time wondering how to pronounce them than reading the story. Is LImya said differently than Limya or lImya? How about Li'mya? At times, authors do try to include something in the front of the story like, "The apostrophe should be pronounced as a glottal stop," but this gets old very quickly. The apostrophes and capital letters are supposedly to give the language an exotic flavor, but here exotic crosses the line into weird and distracting.
2) Don't use silly strings of letters, either. Sure, it can look "alien" to call a dragon Jwxgchiblwz. But it is only confusing to your readers, who are expecting to read a fantasy and now a book where the author's fingers apparently occasionally went spastic on the keyboard. Avoid this, just as the capital letters and apostrophes in the middles of words. If you really want an exotic flavor, construct the names of each culture on a strict rule, and when they encounter a new culture or group of people, then model the names on some other rule. They will be distinctively different and sound so to your characters without giving your readers hernias.
3) Don't unthinkingly accept the Sapir-Whorf hypothesis. In simple form (the form which most people know it in), the Sapir-Whorf hypothesis says that language structures thought. Thus, someone who has only two color words in their language, like the words for light and dark colors, supposedly cannot actually recognize dark red and dark blue as distinct colors. This has been proven manifestly untrue in simple experiments with people who speak these kinds of languages, where they were able to recognize the different colors as different and could learn the words for them in other languages with ease.
Language is not an iron chain on thought, and much thought takes place without language at all. Orwell's Newspeak would have failed. Similarly, a fantasy language that is supposedly the "language of evil" or the "language of good" would fail on strict linguistic grounds. The people who spoke that language would still be capable of thinking outside it.
Of course, Tolkien himself violates this rule with the Black Speech, but Tolkien had a fine ear for linguistic differences (to him, Welsh was pleasing, Gaelic was not, and he may have modeled a few words of the Black Speech on Gaelic- thus nazg, Black Speech for "ring," is close to nasc, Gaelic for "ring") and could get away with this kind of thing. Nor did he represent the language as the source of evil; it was evil because of who made it.
4) Don't assume that constructed languages would necessarily spread in a single generation. Again, Tolkien's Black Speech seems to be the exception to this- somewhat ironically, it is the only completely constructed language in Middle-earth; Sauron is the only character in Middle-earth who shares his creator's favorite hobby- but its creator was immortal, and it developed naturally after being released into the "wild," as it were, splitting into many Orcish tongues. (Like a lot of information I use on Tolkien's languages, this comes from Ardalambion, or Of the Tongues of Arda, the best site on Tolkien's languages, which I urge you to check out).
Most amateur fantasy authors aren't nearly so careful. A language can be created and spread and never change. The "Common Tongues" are a particularly bad example of this. Rarely is there any natural history to them, either with their being a single dominant language of the land or a pidgin, then creole, of many different tongues. Most races speak the Common Tongue in addition to their own. But why? Where did it come from? The authors don't bother to explain.
This leads me into...
5) Realize that linguistic differences will exist in a wide world. Robert Jordan is really, really bad at this. He has cultures spread across a large continent, and even a culture that has spent hundreds of years on the other side of the ocean, and yet everyone speaks the same language, with only a slight accent.
Think a little. Fantasy worlds will either not have global communication, or have it only in limited kinds (for really important messages, telepathy might be used, as it is on Marion Zimmer Bradley's Darkover, but even there, it is possible only because the mental "language" is common to all telepaths and is different from normal speech). It does not make sense for someone who lives on the other side of the world to speak the same language as someone who lives on the near side, even if it is a "Common Tongue." There would be at least regional variations, and probably much more differentiation than there is between, say, British English and American English.
Characters in fantasy with linguistic problems either live in worlds which apparently have some secret means of global communication, always meet with one person who can speak their language, or manage to miraculously learn the new tongue in a matter of days or weeks. Not possible, because...
6) Remember that linguistic facility declines with age. Children are the best learners of languages, and can grow up equally fluent in two or more with ease, as long as their parents insist they learn them. Linguistic ability like this declines, save in rare cases of polyglot geniuses, from about ten years of age or a little after. A protagonist who is sixteen and has spoken only the Common Tongue all her life is not going to be able to learn the language of the tribe who rescued her from dying in the cold just like that, particularly if there is no translator to guide her. Names of objects and some concrete actions might come easily, but imagine trying to explain the concepts of "be," "no," "yes," "yesterday," "tomorrow," or "red" without common terms. The heroine would never know, if someone yelled at her for touching a pot, whether the words really meant, "No!" or just "Don't touch that!" It would take multiple times to be sure. Meanwhile, she's trying to deal with multiple examples like this, and all the non-linguistic problems.
It's easy to see why authors want a quick fix to this problem, but unfortunately it cuts apart one of the keystones of reality in a medieval-like world, especially one filled with several different cultures and languages. If you have linguistic complexity in your world, you should accept the costs that come with that.
7) Most fantasy worlds do not have the printing press, and literacy is not mandatory. Unless you manage to explain how your protagonists learned to read, the average reader is likely to be skeptical about it. Nobles have the best excuse. Other times, something can be devised (see J. V. Jones's "Book of Words" trilogy, where the hero, Jack, learns to read over a period of five years by copying down books of pictures and words that he was originally hired to copy precisely because he could not read). But many times, heroes or heroines raised out in the middle of nowhere on a not particularly prosperous farm can still read and write. It boggles the imagination.
Another problem is the wide availability of books in many fantasy worlds outside academies, rich nobles' houses, or monasteries, all of which might be reasonably expected to contain them. If this world has the printing press, fine. Yet somehow the protagonists are always getting access to books in worlds that don't have it. This seems to happen especially if the books are older, when you'd think it would be harder.
There's never any doubt that the information in books is accurate, either, despite the propensity of scribes to make mistakes in their copying, insert little notes, or try to make "improvements" on the original text. Just once I would like to see a hero who has to read the answer in some ancient book take a moment to wonder about its accuracy.
The source of this problem, I think, is most fantasy authors' comfort with books and words in general in our own time. We can read and find books everywhere, so we don't think it's a big deal. But it is a big problem when you have a peasant hero who should never have learned how to read fluent in the philosophies of his world. (On occasion, authors even violate their own rules, as a few stories I have read that state ardently that girls in their worlds are not taught to read and then as ardently show their heroines deciphering secret messages, without explaining how she could read them).
8) Please, no English-dependent puns. I've seen this ruin a few otherwise good books. Unless your book is specifically set on modern Earth or is historical fantasy, it's a good bet that your characters aren't speaking English. Trying to have them solve riddles or crack jokes where the answer depends on the English word- or worse, the actual spelling of the English word- sends me into convulsions. It also destroys the carefully built-up suspension of disbelief (assuming the author has managed this).
9) Try to keep names consistent. If you take all your names for one country of characters from Spain, and the next from Italy, fine. If you make is so that most female characters' names end with -ian and most male characters' names with -er, fine. The mixing of names really gets to me, though, particularly when modern Earth names are mixed with made-up names and names from Earth's historical periods.
Of course, there's Tolkien on one end of the scale, who not only specifically adapted his hobbit-names from Old English sources or English words, but also gave the history of what some of those names would have been in Westron. Not everyone has to be like him. But please, please, please avoid the (unintentionally) hilarious example of Sara Douglass's The Wayfarer Redemption, which merrily uses Biblical names, Portuguese names, and names like StarDrifter. And the heroine's name is Faraday.
Most fantasy authors can settle somewhere in the middle.
