For starters, it may be due to the fact that you cannot stand this person and hate their guts?
A pair of researchers with Kyoto University has found a possible explanation for why people sometimes have trouble maintaining eye contact when talking with someone face-to-face. In their paper published in the journal Cognition, Shogo Kajimura and Michio Nomura describe experiments they carried out with volunteers to learn more about how the phenomenon works and then discuss their findings.
Most everyone knows that maintaining eye contact with another person while speaking can sometimes be difficult—at times, the urge to look away becomes overwhelming. In some instances, it is clear that such breaks just seem natural to keep things from becoming awkward, or it signals that someone has grown bored with the conversation—but at other times, the researchers suggest, it is because we are trying to keep our brains from overloading.