How to scrape specific elements of the Drupal theme to a file?

Question

I want to save two divs - one with the site header and the other has the main nav menu - to a PHP file that can be used in other (even non-PHP) sites to replace their header and menu using Javascript DOM insertion and CORS with a header on the saved file that allows cross-origin resource sharing.

The idea is to update the distibution code automatically on cron or on demand, when changes in the menu are made, so they will be reflected in the satellite sites.

I have been able to create a working proof-of concept using simplehtmldom on the current version of our site. However, when I tried to adapt it to the new version in dev, using a different theme as a subtheme of Adaptive Theme, then simplehtmldom cannot read the URL properly - it seems that the blocks are missing see https://drupal.org/node/2139613

The divs are not $page regions, although the header-wrapper div contains the $page['header'] region (and the region is invisible to simplehtmldom).

I looked at the Web Widgets module, but that is for content and views - I want to get the theme elements in the header-wrapper and the nav-wrapper, two arbitrary divs at the top of the page.

It's also frustrating that I cannot access the $page array or even the $variables in my module. I had to set the module weight to 1 to be "heavier" than the simplehtmldom module, since there are other modules that require simplehtmldom module (like devel themer) and I don't want to load its library twice. But I tried the latest simplehtmldom library separately, with the same results.

I know there are scraping tools based on node.js but I'm looking for a more lightweight solution. simplehtmldom worked, but in the new site it doesn't

Answer 1

I solved the issue, and it may have been related to

// the zero here is necessary because simplehtmldom returns a collection
$topbar = $html->find('#header-wrapper', 0)

Here is my sample code - I am thinking about contributing a module for this function, with some admin settings

// get the header code and write it to a file
function module_name_global_header_write() {//dpm('raw_html');

    // there may be an issue if we are processing this function while viewing the same url as $sourceDomain
    // perhaps we will need to create a page or URL that is not viewed but can supply the HTML for the header
    $sourceDomain = 'http://site_name.com';
    $sourceMenuId = 'superfish-1';  

    // initialize simplehtmldom objects
    $html = new simple_html_dom();
    $responsive_desktop_css = new simple_html_dom();
    $global_styles_css = new simple_html_dom();

    // load files into simplehtmldom
    $html->load_file($sourceDomain); // the generated HTML
    $global_styles_css->load_file($sourceDomain . '/sites/all/themes/adaptivetheme/site_namedaptive/css/global.styles.css');
    $responsive_desktop_css->load_file($sourceDomain . '/sites/all/themes/adaptivetheme/site_namedaptive/css/responsive.desktop.css');


    // #header-wrapper and #nav-wrapper are the ids of the header and menu
    $topbar = $html->find('#header-wrapper', 0); // the zero here is necessary because simplehtmldom returns a collection
    $navbar = $html->find('#nav-wrapper', 0);

    // prepare Cross Origin Resource Sharing header
    $header = "<!-- This header necessary using CORS http://www.nczonline.net/blog/2010/05/25/cross-domain-ajax-with-cross-origin-resource-sharing/ -->
    <?php
    header('Access-Control-Allow-Origin: *');
    ?>";

    // some more CSS rules that are not in the css files
    $morestyles = '#google-appliance-block-form{width:100%;height:30px;text-align:right}#block-google-appliance-ga-block-search-form .container-inline div{display: inline;}.element-invisible, #block-google-appliance-ga-block-search-form label, .url-textfield{display:none!important;}#nav-wrapper ul li a, #menu-bar ul li a {padding: 5px 8px;}#nav-wrapper ul li.last, #menu-bar ul li.last {
    border-right: none;}';


    // prepare the CSS style tag; in case we do not want css, we only have to unset the $global_styles_css var
    $cssStyles = $global_styles_css ? '<style>' . $global_styles_css . $responsive_desktop_css . $morestyles . '</style>' : '';
    // replace relative image urls with absolute
    $cssStyles = str_ireplace("../",$sourceDomain . "/sites/all/themes/adaptivetheme/site_namedaptive/",$cssStyles);


    // assemble output from components
    $output = $header . $topbar . $navbar . $cssStyles;

        // create directory if not exists
        if (!file_exists('public://global_header')) {
                mkdir('public://global_header', 0777, true);
        }

    //write the file that will be fetched by the script tag embedded in the target site
    $filename = "public://global_header/raw_header.php";
    file_put_contents($filename,$output);
}

here is the code in the js file:

//cross-browser see http://www.nczonline.net/blog/2010/05/25/cross-domain-ajax-with-cross-origin-resource-sharing/
function createCORSRequest(method, url){
    var xhr = new XMLHttpRequest();
    if ("withCredentials" in xhr){
        xhr.open(method, url, true);
    } else if (typeof XDomainRequest != "undefined"){ //IE
        xhr = new XDomainRequest();
        xhr.open(method, url);
    } else {
        xhr = null;
    }
    return xhr;
}
var sourceURL = 'http://site.com/';
var request = createCORSRequest("get", sourceURL + "/sites/default/files/global_header/raw_header.php");

if (request){
    request.onload = function(){ 
    // do stuff if request is successful; if it fails, there is no replacement of existing HTML in target site
  // look first if a container with id #header-id exists, if not use the existing #topbar container
    document.getElementById('header-id') ? document.getElementById('header-id').innerHTML = request.responseText : document.getElementById('topbar').innerHTML = request.responseText;
    // change a relative image URL into absolute
    document.getElementById('branding').getElementsByTagName('img')[0].src = sourceURL + '/sites/default/files/image.png';
    // change relative Google Appliance Search action into absolute
    document.getElementById('google-appliance-block-form').action = sourceURL + '/';
    };
    request.send();
}

example script tag:

<script src="http://site.com/sites/all/modules/custom/module_name/module_name.js"></script>