How should I implement a callback to a single Drupal class (in my case it's Transliteration). I don't need a full Drupal load/bootstrap just one independent class.
Taken literally, within the confines of Drupal, I don't believe you can. We rely on PSR-0 for class loading, so at the very least you must bootstrap Drupal to get the class loader running. (Though not necessarily a full bootstrupbootstrap).
To truly just load a single class, you'd need a to call a different file than index.php where you'd hard code the class loading.
I haven't looked at how Ajax requests specifically have changed from D7 => D8, but thanks to D8's DI, only classes that are used in a single request will be instantiated, so the need to try to speed optimize things out, as you seem to want to do, should be lessened.