Create node content from external HTML

Question

After a few attempts browsing the web, I don't figure out how to download HTML from a given url, and create node content from it.

What I would achieve is quite simple :

• Have a link-like field in my content type
• The source code of this page is downloaded either in the Body of the content, either in a MHT-like file
• Viewing the node displays it either in the bodyeither` in an IFrame.

The content creation would happen mostly from a dedicated script creating a new content through Drush, filling only the Title and the link field.

The main goal of this is to save short-lived webpages in a drupal website, and ideally being able to index them.

I'm open to any ideas : module, coding, whatever. Don't hesitate to give me advice !

Thanks

Answer 1

This would have to be done using a custom module. I do not know of a contributed module that would perform this. Probably because a module like this could be seen as a huge copyright infringement lawsuit waiting to happen.

But if I were to do this I would use GuzzleHttp as the engine to pull down the content. Then take that content and create a new entity using entity_create(). Where you have to focus your efforts will be sanitizing the information you "guzzle" off the web so that you do not create security issues in your site from XSS, SQLi, or other vulnerabilities. The process would be triggered on node-save.

This is just the basic idea, I cannot explain how to write the entire module as that would be pages of content.