Because we are to avoid unhealthful things. Caffeine, as a "recreational drug," is not healthful. Some (like myself) don't use caffeine except when we need a mild stimulant for health or safety reasons and prefer one which is relatively benign.

Far from being an excuse, when given the choice between pharmacueticals and caffeine, caffeine wins hands-down.