For the time being I have solved it in the following way:

 1. I have created a separate entity browser configuration for each
    image category. Each browser has a separate id which is what makes step 2 possible.
 2. In a custom module I have implemented an ```EventSubscriber``` that listens to the ```EntitySelectionEvent``` that Entity Browser fires when an entity is selected. When this happens my subscriber checks the browser id currently used and based on that it sets the category field for any of the uncategorized entities that was selected. [Here][1] is a gist of the ```Eventsubscriber``` implementation.
 3. For each category I have created a separate view display in my "Select existing image" view. The views are very similar and the only difference between them is that they filter the entities to only show items with a specific category. 
 4. Each separate Entity Browser config created in step 1 is wired to the corresponding view display mode.

This works pretty good. When uploading files using an entity browser with the id ```image_library_browser__content``` the image will get the category ```content``` and when selecting existing items the view will only list ```content``` items.

A problem with this is if you would have alot of different image categories since you need to create one separate entity browser and display per category. On my current project I have 6 so I think it is ok. 

One nice thing with having separate entity browsers per category is that you can define separate upload paths, field descriptions and file types per category. (Currently this is not possible in any other way since Entity browser in its current form ignores the image field settings)


  [1]: https://goo.gl/w1DSu8