As for "mare", well, one of the meanings is a "female horse". So a nightmare should be a female horse that appears in your dream. But generally we don't dream about horses. Usually we have terrible dreams about falling off a building or mountain, appearing for an exam when we are totally unprepared or being chased by monsters and such. In Old English the word "mare" meant an evil spirit or demon. And since it was evil, it was assumed to be a "she" rather than a "he"! Golly I know Caleesto has and evil streak, hee hee, but not all females are evil......
A nightmare is not a dream about a scary horse. The origin is fairly simple, but not really obvious. The night portion is straightforward - it comes from the word night. It's the mare part that makes people think it has to do with horses.
The word is an old one. The Compact Oxford English Dictionary, 2nd Edition attests to nightmare as early as c. 1290. Night is recorded as early as c. 825, but it is such a basic word that it is likely far older than surviving manuscripts. Mare is dated as early as c. 700. Mare also has cognates in many languages, including Dutch, German, French, Polish, and Czech, and ultimately derives from the Indo-European root *mer-, meaning to rub away or to harm. *mer- is also the root of murder and mortal.