I would :
Check if there's a weekend between $start and $end :
if($start_date->format('W') != $end_date->format('W'))
Then substract 48 hours for each week of difference.
$difweeks = intval($start_date->format('W') - $end_date->format('W'));
$duration = $duration - intval(48 * $difweeks);
$duration = $start_date->difference($end_date, 'hours');
// Place this :
if($start_date->format('Y-W') != $end_date->format('Y-W')) {
$difweeks = $start_date->difference($end_date, 'weeks');
$duration = $duration - intval(48 * $difweeks);
}
$entity_field[0]['value'] = $duration;