The answer by Sharan didn't work with the current Drupal 7.34. An empty array will not create a new group,
even after disabling and uninstalling and re-enabling the module and flushing caches and so on...
I did some research and found a way that works quite well. It will create an new item (with subitems) in the 'Management' menu, that
has it's own (translatable) title and description. When the group itself is opened in a browser ('admin/config/custom' in the example),
it will show an overview page that lists all the subitems with descriptions. It also integrates as expected with the menu bar from the admin_menu module.
function hook_menu() {
// The new group
$items['admin/config/custom'] = array(
'title' => 'Custom settings',
'description' => 'All the custom settings.',
// 'position' and 'weight' can be used to place
// the group in the the parent overview (admin/config).
'position' => 'right',
'weight' => -20,
'access arguments' => array('access administration pages'),
// This renders the group overview.
'page callback' => 'system_admin_menu_block_page',
// 'file' and 'file path' are required for the
// 'system_admin_menu_block_page' callback.
'file' => 'system.admin.inc',
'file path' => drupal_get_path('module','system'),
);
// A subitem in the group
$items['admin/config/custom/module-name'] = array(
//other code here
);
}
EDIT: A cache flush should be enough to enable the group.
$items['admin/settings/custom']
menu too