It'll be something more along the lines of:

    $form['field_mytax_reference'][LANGUAGE_NONE][0]['#default_value'] = $taxTermId;

If the field is a multi-select you would wrap the default value in an array, but there's no need to set a key for it:

    $form['field_mytax_reference'][LANGUAGE_NONE][0]['#default_value'] = array($taxTermId);

Have a look at http://drupal.stackexchange.com/questions/27767/why-is-hook-form-alter-so-messy-in-d7/27784#27784 for a bit more info.