My girlfriend has the same issue. Her doctor told her she was allergic specifically to cat saliva, and since they groom themselves with their tongues their fur has little bits of saliva on them, which triggers a less intense reaction than direct cat licks.