The cleanest solution was just going into modules/webform/modules/ and extending WebformCompositeBase to create a custom composite with an entity reference sub-element. It's super intuitive if you go off of the webform_example_composite module (included in core webforms I think).
You only really need to alter webform_example_composite/src/Element/WebformExampleComposite.php. If you create the entity reference you want to add as an individual, non-composite element in the webform ui, you can look in build->source and translate the form of it from yml to php.
A basic Group entity element looked something like this:
$elements['group_members'] = [
'#type' => 'webform_entity_select',
'#title' => t('Group Member'),
'#selection_handler' => 'default:group_content',
'#selection_settings' => [
'target_bundles' => [
'research_leader-group_membership' => 'research_leader-group_membership'
'#attributes' => ['data-webform-composite-id' => $html_id . '--group_members'],
In the webform_example_composite module, the Submission data is generated by again extending WebformCompositeBase in Plugin/WebformElement/WebformExampleComposite.php and overriding the formatHtmlItemValue and formatTextItemValue functions. If you need custom formatting for an entity reference, you'll have to look through WebformCompositeBase and figure out what to call. I was happy with the standard formatting. Just comment out the custom formatHtmlItemValue and formatTextItemValue functions in Plugin/WebformElement/WebformExampleComposite.php to see what standard formatting looks like for your composite.