It'll be something more along the lines of:
$form['field_mytax_reference'][LANGUAGE_NONE][0]['#default_value'] = $taxTermId;
If the field is a multi-select you would wrap the default value in an array, but there's no need to set a key for it:
$form['field_mytax_reference'][LANGUAGE_NONE][0]['#default_value'] = array($taxTermId);
Have a look at Why is hook_form_alter() so messy?Why is hook_form_alter() so messy? for a bit more info.