The purest enjoyment of life - be a RailFan.

You've already mentioned the usual reasons.
But overall, there are a lot of reasons for which a loco can be deemed unfit for passenger operations.
1) Age - Over-aged locomotives, with reduced reliability and increased frequency of failures can be deemed unfit for passenger or even mainline operations.
2) Defects in suspension - Locomotives with a defect in the suspension, having increased oscillations...
more...

and reduced ride quality can also be deemed unfit for passenger duties, or can be run at reduced speed limits. Note that, in many cases, the locomotive may be allowed back into regular duties once it is repaired.
3) Defects in powerpack or turbocharger - For example, a diesel loco may have a problem with its fuel injector, turbocharger or the diesel engine itself leading to reduced traction power. In these cases, the loco can be delegated to inferior duties until it is repaired.
4) Repairs and damaged parts - If a locomotive cannot be readily repaired, then it will be put into inferior duties until it can be sent to a workshop.