The bayonet is the foot soldier's "sword". You will probably want to purchase a bayonet and scabbard to match your musket. Pictured at left is an M1855 triangular (cross section) socket bayonet for a Springfield musket and leather scabbard with 7 rivet belt frog. Springfield and Enfield bayonets attach to the front site of the rifle. Older era muskets sometimes have short swords as bayonets, attached to a side stud on the barrel of the musket.
Some sutlers have original bayonets for $80-$100. These were generally used in India and, while contemporary with the Civil War, may not have been in use in the United States at the time. Generally they are stronger than reproduction bayonets - it is relatively easy to bend a repro bayonet if you "take a hit" and fall on it. Also, be careful when you sit down in full accoutrements - you may bend your bayonet!
Although, for safety reasons, reenactors do not go into battle with fixed bayonets, they nevertheless come in handy. Marching with bayonets fixed is a pretty sight. And they are part of the drill for "Inspection Arms" and are pretty necessary for stacking arms. Finally, a bayonet stuck into the ground makes an excellent candle holder!
If you plan to occasionally do a Federal impression, get a black leather scabbard rather than russet.
C & D Jarnigan & Co. (2000 prices)
#7032 Triangular Bayonet for Springfield musket models 1855-1864 $32.50 #7033 Triangular Bayonet for Enfield $32.50 #247 US Scabbard with Belt frog for M1855 Springfield Bayonet $29.95 #248 US Scabbard with Belt frog for British Enfield Bayonet $29.95