A transition constraint is a way of enforcing that the data does not enter an impossible state because of a previous state. For example, it should not be possible for a person to change from being "married" to being "single, never married". The only valid states after "married" might be "divorced", "widowed", or "deceased".
This is the database-centric interpretation of the term.
In formal models in computer security, a transition constraint is a property that governs every valid transition from a state of the model to a successor state. It can be viewed as complementary to the state criteria that pertain to states per se but have no bearing on transitions between successive states.