Part of it is very abstract and arbitrarily associative. As well, it can certainly morph. I wouldn't say there's any specific rhyme or reason to it.
If you ever watched the movie Soul—that spoke to me pretty well, even though they didn't focus specifically on synesthesia, per se.