Yeah, I think trying to take the idiom too literally reveals a lot of holes, comparing apples to oranges is a legitimate thing to do in the context of what fruit to eat, for example.

Nitpicking that particular but doesn't negate the overall meaning behind the idiom. Lots of things can be compared, but the comparison provides dubious value at best. Sure I can compare having sticky notes on a fridge to Jira, but then I'd be comparing apples to oranges.