This seems to be the reverse of the question remove vertical tabs, and should have a very similar answer. In the documentation for the vertical tabs module, the first code snippet is how to enable vertical tabs on any form. The second snippet looks more appropriate, as it is how to enable tabs from your custom module that creates your custom entity.
<?php
$vtabs = variable_get('vertical_tabs_forms', array());
// Enable vertical tabs on a form.
$vtabs['my_form_id'] = TRUE;
// Disable vertical tabs on a form.
$vtabs['page_node_form'] = FALSE;
// Enable vertical tabs but be selective of which fieldsets are included.
$vtabs['page_node_form'] = array('menu' => FALSE);
variable_set('vertical_tabs_forms', $vtabs);
?>
Edit:
The site variables here is a huge array that is always accessible to drupal. Some of these settings can be stored in your site's settings.php file, as this variables array is present even there. However, because of the nature of the change that you desire, I think the best choice for you is to have a hook_install in the module that defines your custom entity.
Assuming module name "your_entity", add a file "your_entity.install" to the module.
function your_entity_install() {
// Enable vertical tabs on entity form.
// this is supposed to be true as long as this module is enabled
$vtabs = variable_get('vertical_tabs_forms', array());
$vtabs['your_entity_form'] = TRUE;
variable_set('vertical_tabs_forms', $vtabs);
}
Then disable and re-enable your module. These variables can be set in the form_alter, but that would only be recommended if you needed the value to change based on other information that is not present at an earlier time.