From the similarity between their skin color and the color of smoke. Also from the way that they were said to be able to sneak around quietly, entering a room "like smoke".
Submit corrections to this slur
The Racial Slur Database, since 1999. Not copyrighted in any way, shape or form.
This database was created entirely from data gleaned off the 'net and via submissions from people like you and your parents. It's supposed to be funny and/or informational.