Trying to get form elements to appear in draggable table

Question

So I saw this table example on Drupal.org with two form elements per row. I wanted to do something similar except have the rows draggable. Below is example of the Drupal.org code.

$form['contact'] = array(
  '#type' => 'table',
  '#caption' => $this->t('Sample Table'),
  '#header' => array($this->t('Name'), $this->t('Phone')),
);

for ($i = 1; $i <= 4; $i++) {
  $form['contact'][$i]['#attributes'] = array('class' => array('foo', 'baz'));
  $form['contact'][$i]['name'] = array(
    '#type' => 'textfield',
    '#title' => $this->t('Name'),
    '#title_display' => 'invisible',
  );

  $form['contact'][$i]['phone'] = array(
    '#type' => 'tel',
    '#title' => $this->t('Phone'),
    '#title_display' => 'invisible',
  );
}

It's simple and clean, but lacks the draggle functionality I need. So following some docs I came up with the code below.

$rows = array();

for ($i = 1; $i <= 4; $i++) {

  $rows[$i] = array(
    'class' => array('draggable'),
  );

  $rows[$i]['data']['name'] = array(
    '#type' => 'textfield',
    '#title' => 'Name',
    '#attributes' => array(
      'class' => 'contact-row',
    ),
    // '#title_display' => 'invisible',
  );

  $rows[$i]['data']['phone'] = array(
    '#type' => 'tel',
    '#title' => 'Phone',
    // '#title_display' => 'invisible',
  );
}


$form['#tree'] = true;

$form['contacts'] = array(
  '#type' => 'table',
  '#caption' => $this->t('Sample Table'),
  '#header' => array($this->t('Name'), $this->t('Phone')),
  '#rows' => $rows,
  '#attributes' => array(
    'id' => 'contacts-table',
    'class' => array('field-multiple-table'),
  ),
  '#tabledrag' => array(
    array(
      'action' => 'order',
      'relationship' => 'sibling',
      'group' => 'contact-row',
    ),
  ),
);

For the most part everything is there except the table cells don't contain the form elements. It just references the name. What do I need to do to make the textfields appear.