1

I have been assigned with the task of maintaining a specific Drupal site. The problem is that the person who created the website did not actually think it through. No input filters to be heard of, no restriction for images' placement, plain HTML everywhere.

I'm now on the spot, since I've promised to tidy up the website, and I've just realized how much work will it require. There are articles that are copy+pasted from wikipedia (the WYSIWYG editor kept all the formatting), random websites or (even worse) from MS Word.

I understand that I will have to do some manual editing and check the articles one by one, but I was wondering if there is some kind of a method that someone may have used in a similar situation. Perhaps a module that will cleanup the html of the various nodes?

Note: The website is built in Drupal 6, but if you have suggestions for Drupal 7, I guess that I could upgrade.

1
  • As for cleaning up the copy/pasted Wikipedia entries — to anybody that would need that these days, I'd recommend using the Puzzler module. It allows for clean retrieval of Wikipedia data elements (via their API), and it will also automatically keep them up to date. Puzzler is the terminator of copy/pasted data from Wikipedia :-) – Vacilando Jul 17 '16 at 15:37
6

Normally, when I have encountered this, there is too much variety to do much automated cleanup. You can do something like this, though

$sql = "
  SELECT n.nid
  FROM {node} n
";

$result = db_query($sql);

while ($row = db_fetch_object($result)) {
  $node = node_load($row->nid, NULL, TRUE); // bypass internal cache
  $node->body = _filter_htmlcorrector($node->body);
  node_save($node);
}

Either stick this in a custom menu entry or make a simple script for it. Depending on how many nodes you have, you may need to use the Batch API.

_filter_htmlcorrector will fix bad HTML to an extent, and I have also used check_markup with luck, too. For check_markup, you just need to figure out the numeric filter number to use.

Other than that, you really do need to play with the processing you do in the loop. This will give yourself a head start, but you still need to edit each article and manually check/fix things.

2
  • This sounds like a really interesting solution. I'll try this in a staging setup and see how it works. Thanks a lot for your reply. – F1234k Dec 11 '11 at 11:07
  • 1
    This is a pretty awesome way to handle it. If you want even MORE control on your html, use the same concept but filter your content using php tidy extension. I have cleaned up major html static sites with this tool (thousands of pages) and once properly configured, it is quite efficient at what it does. – stefgosselin Dec 14 '11 at 12:08
2

I guess you need to use Input format in this issue. You need to create filter and set an allowed elements to avoid dirty inputs. In the wysiwyg settings admin/settings/wysiwyg/profile/ID/edit in the Cleanup and Output tab be sure to enable the Verify HTML, Apply source formatting, and Force cleanup on standard paste. If the old nodes are created using Full HTML then you need to apply the above conf in the Full HTML profile...please let us know if that helps you

1
  • 1
    I understand that for new content that will be added from now on it's pretty easy to enforce proper inputs. The problem with applying filters to the "Full Html" input format is that even though it may look OK in most of the cases, the content that is stored in my database will not be "clean". It will just display differently. Or am I missing something here? I want to "purify" the html that already exists in the website, even if that means that I will manually fix any mistakes during the purifying process. I hope that this makes sense. – F1234k Dec 8 '11 at 11:30
1
+50

If the text looks good with Full HTML, why don't just continue to go with that? At lease, you can switch to HTMLPurifier, which gives you total control of the HTML tags/attributes. Thus you could make sure that there is no security issue (no script tags).

Then I think you must manually edit each node, clear format, add custom CSS classes, semantic HTML tags... so that your content will be semantically correct. But I suggest you only do that if it is really needed.

Or, if you just want to remove extra inline CSS/JS, unnecessary class/id attributes, there are many tools to do that, and you should do directly on the database. A 3rd party tool might be faster and more mature. E.g. there is a bug in D6 HTML corrector filter.

3
  • I have a "psychological" issue with bad content being stored in my database. I guess that the chances of bad html being exploited somehow in the future are slim, but I still cannot deliver a website that will have bad content in it (with me knowing about it). The HTMLPurifier seems like a very good solution (I just took a quick look at it) so I will try it and see if it helps me out. In any case, +1 for your suggestions (and yes, I know that I will have to do a lot of manual work, but that's a common side-effect of OCDs) – F1234k Dec 11 '11 at 13:01
  • The advantage of an "input filter" is that the input could contain malicious data. However I don't know what do you want to clean: extra data (embedded CSS/JS, extra id/class that you'll never use) or non-semantic data (replace some B/STRONG tags with H4 tags for node section titles). Anyway, I'll add some tips in my above reply. – jcisio Dec 11 '11 at 14:06
  • I know that input filters are designed to do exactly that: protect you from malicious data (even if they are stored in the database, they still can't hurt you). However, the content that I'm referring to, varies a lot: html code generated from Microsoft Word, copy/pasted code from random websites and every bad practice on the book. – F1234k Dec 11 '11 at 14:12
0

I've not been in this situation, so just suggestions here! Obviously back up the db before anything. I'd try and see if you can find a filter that does a decent job of cleaning up (I know you want to clean the DB itself). I'd maybe take the worst offenders and see what markup comes out with a few different HTML correctors (eg htmLawed). If any of them consistently do an ok job, then you can use a drush script, or create an action to use with VBO, to run the filter on the bodies and teasers and then save them back to the db. Then take a look and see the damage :/ Good luck

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.