> ... these are text token prediction algorithms, underneath.

If I had to take a wild guess, my guess would be that the prediction probabilities are very dependent on context, so by changing the context, the entire slate of probabilities shift.