Humans didn't like the idea of an X, hence also words such as xenophobic. Before 1806, there were no English words starting with X. Good luck creating an alphabet book without a word starting with X.
The first word was xebec, a type of small three mast ship.