Unfortunately, some people are very sensitive to caffeine. Coffee can increase the risk of anxiety, especially if someone has a tendency to be nervous, or jittery.
Caffeine works by stimulating your nervous system and anxiety can be the result of this overstimulation. Long term overstimulation can bring adrenal fatigue, a result of overuse rather than the coffee itself.
There are reports of difficulty sleeping, either broken sleep or insomnia at night, because of the energy boosts that come with caffeine. Coffee later in the day can be switched with a decaf, rather than stopping the enjoyment of the coffee.