Most borers are very specific about their needs. It's kinda like worrying that an Ash borer might attack your Cherry Tree. Not impossible... not likely.
If bugs are creeping you out, figure out how to segregate your fire wood. Cover it, separate it from your living space, fumigate. Whatever...