Take a toothpicks and carefully break it in two halves. Ideally you are left with something about half as wide as the original pick. You want thin and sturdy pieces to get into the port. Now carefully scrape the lint and dirt out of the port. The dirt tends to compact on the bottom, so you have to scrape a bit to get it all out. I would recommend to do it in direct sunlight to better see the interior of the port. Torchlights usually are either to bright on the spot or to dim.

Works like a charm.