Have you ever found a tool that just makes your life a little easier? That's my goats hair brush. I know these are meant for painting but it is the best brush I've found to keep my drawings free from debris. They are soft and inexpensive so make a great addition to any art tool box (mine is my little blue Art Bin I've had since Art School)
They come in many sizes and are washable so once they've become soiled wash them gently with some brush cleaner and preserver or saddle soap, both protect the bristles.
I have a few different brushes around, one for graphite, one for coloured pencils in a couple of sizes depending on the size of the drawing.
Try them out I think you'll like this little tool.