Would disabling core modules which I don't need, like Color, speed up my site? If so, why?
In theory, the answer is yes, but in practice, I doubt it. It depends entirely on on the module though.
If you have badly written module that implements hook_boot, so it executes on even cached page requests, and it performs lots of logic in there, and you have no use for it, then sure, you would gain speed from disabling it.
Most likely though, the average module is going to be implementing logic that is only actually executed when it is used, so if you don't use it it's not going to cost much.
If it's a big module, you could see a small dent in memory usage, as all enabled modules are loaded into memory, and as pointed out by Mołot, a modules variables and menu router entries will consume some resources, but again, I think in practice, it's not going to make a big difference.
That said, if you don't use the module, why keep it enabled? It by definition doesn't provide you with any benefit if it's not used, and at least it's going to clean up the administrative pages.