I haven't got around to sending user emails yet.
I wonder if we are able to send HTML emails by default from the Emails section (admin/config/people/accounts), or if we need to use modules for this.
Out of the box (w/ core modules only) default account emails (and any other emails) that Drupal generates do not support HTML. Drupal converts all html emails to plain text before sending. You can install and configure MimeMail module and its dependencies. Then you'll be able to insert html markup into the account email body field at 'admin/config/people/accounts' and have those emails sent properly in html.