There is hook_ckeditor_settings_alter()
to change/alter CKEditor settings. But I can't find a way to prevent certain CKEditor plugins from loading.
I am trying to achieve similar behaviors as here: https://www.drupal.org/node/2546048#comment-10188390 but that is for wysiwyg module, we don't use wysiwyg module.