I posted this as an issue on the media_wysiwyg module bug tracker over at https://www.drupal.org/node/2357993 and have been working my way through it since. Here is the best result I have managed to achieve so far copied from my responses to that issue.
This issue is mostly caused by the file_entity module. To work round this I have restricted the view modes available to media_wysiwyg in /admin/config/media/wysiwyg-view-mode. I have set images to only allow the WYSIWYG view mode like so:
I have then created an override template for the file entity called
file--image--wysiwyg.tpl.php (see https://www.drupal.org/node/1793548) and I have the following code in it:
This has now removed all the annoying links, headers and wrapping
<div>s from the output EXCEPT for the media_wysiwyg wrapping
mentioned in comment #4.
As this is just one wrapping
<div> and the page is HTML this now no longer breaks the link, but it would still be preferable to be able to remove this annoying wrapping