After enabling CKEditor, you have to go into the filter format in the admin that you want to edit, and configure the toolbars to have the buttons and configuration that you desire. It is a bit extreme to call it unusable.
For font color, there is an add on module that provides that plugin integration, ColorButton.
For font size there is CKEditor Font Size.
For your original question, there is no way to drop-in a custom build of CKEditor and drop it into your site (that I know of). I suppose it could be possible if you use the Editor API and extend and provide it, but there is not a lot of reason to do so.
I am aware that ColorButton only has a dev release, but it is quite functional. I will work at fixing a couple of superficial things and tagging a stable release.
Where ColorButton and others expect to find the plugin:

Adding the buttons to CKEditor:

You can then override the list of colors in the option box with your own.

Also note that ColorButton requires PanelButton (a utility CKEditor plugin) as well. It also requires downloading and placing panelbutton in /libraries.