With proper charger (Pila IBC, Xtar WP2 II and 4Sevens Single Bay charger) and high quality protected Li-Ion cell (that is, anything based on Panasonic, Sanyo etc cells instead of TrustCrap/UltraCrap) is NOT dangerous at all. Especially if you use it in single-cell light.
When used with Solarforce XM-L dropins, lithium primaries offer very little noticeable advantage in brightness (and none in runtime), so I have switched to rechargeable 18650s and 18700s completely.
I'm keeping a remaining lot of twenty or thirty CR123A cells around for mere emergency purposes, only. I am not using them anymore, but they have a considerably long shelf life - their only advantage, if it is any, at all.
just my .02, hope that helps.
If you want a good and safe setup at a reasonable price I would recommend these batteries with this charger. This setup would be good for somebody who just wants to charge and use their lights, and could care less about voltage and amperage charts and all that stuff. This is the exact setup I would recommend to family members and coworkers that aren't tech savvy (not saying you aren't, just saying anybody could figure it out).
I'm not sure how often you use your flashlights, but cr123a's aren't really cheap unless you're buying in bulk, and even then it wouldn't take me long to make up that cost with a decent 18650 and charger setup. And cr123a's can be just as dangerous if improperly used (ie mixing a drained battery with a full one).
So in the end you're probably looking at around $30 to get a great 18650 charger and battery, that is just as safe as a cr123a combo. In the end it's up to you to decide what works financially for you.
I really appreciate you three chiming in, especially rcm with the product advice. I'm sure I will eventually make the jump over to 18650s. I know I should already have done it, if for no other reason than economics. As for now, well, I do buy the cr123s in bulk, like, usually 100+ at a clip because it is convenient, albeit wasteful, and I'm lazy.
In the end, the frugal part of my personality will win out, but I will likely still get dizzy when y'all talk science mumbo jumbo. 
Oh, and Shadowww, I'm not really scared of the danger... just the dark!