For the time being I have solved it in the following way:

1. I have created a separate entity browser configuration for each image category. Each config browser has a separate id to make it possible to separate in the next step.
2. In a custom module I have implemented an EventSubscriber that listens to the EntitySelectionEvent that Entity Browser fires when an entity is selected. When this happens my subscriber checks the browser id currently used and based on that it sets the category field for any of the uncategorized entities that was selected. Here is a gist of the Eventsubscriber implementation.
3. For each category I have created a separate view display in my "Select existing image" view. The views are very similar and the only difference between them is that they filter the entities to only show items with a specific category.
4. Each separate Entity Browser config created in step 1 is wired to the corresponding view display mode.

This works pretty good. When uploading files using an entity browser with the id image_library_browser__content the image will get the category content and when selecting existing items the view will only list content items.

A problem with this is if you would have alot of different image categories since you need to create one separate entity browser and display per category. On my current project I have 6 so I think it is ok.

One nice thing with having separate entity browsers per category is that you can define separate upload paths, field descriptions and file types per category. (Currently this is not possible in any other way since Entity browser in its current form ignores the image field settings)

For the time being I have solved it in the following way:

1. I have created a separate entity browser configuration for each image category. Each browser has a separate id which is what makes step 2 possible.
2. In a custom module I have implemented an EventSubscriber that listens to the EntitySelectionEvent that Entity Browser fires when an entity is selected. When this happens my subscriber checks the browser id currently used and based on that it sets the category field for any of the uncategorized entities that was selected. Here is a gist of the Eventsubscriber implementation.
3. For each category I have created a separate view display in my "Select existing image" view. The views are very similar and the only difference between them is that they filter the entities to only show items with a specific category.
4. Each separate Entity Browser config created in step 1 is wired to the corresponding view display mode.

This works pretty good. When uploading files using an entity browser with the id image_library_browser__content the image will get the category content and when selecting existing items the view will only list content items.

A problem with this is if you would have alot of different image categories since you need to create one separate entity browser and display per category. On my current project I have 6 so I think it is ok.

One nice thing with having separate entity browsers per category is that you can define separate upload paths, field descriptions and file types per category. (Currently this is not possible in any other way since Entity browser in its current form ignores the image field settings)