Finding nodes that have not been indexed

Question

On a Drupal 8 website I'm administering the ratio of paged indexed by search_cron stays at a value far below 100%.

New pages are indexed, but it looks as if updated pages are not indexed again. I suspect that there is some problem that keeps nodes from being (re-)indexed. However, I see no error in the log. Search_cron takes very long - several minutes even when indexing only 20 pages per cron run.

There is a page Troubleshooting cron which contains an SQL query (under "Check for problems with modules") to find nodes that haven't been indexed so one can inspect them for bad content. Unfortunately, this works only with Drupal 7, and I couldn't find a similar page for Drupal 8.

How can I find non-indexed nodes on Drupal 8?

Any other idea on how to track down the problem?

Update: I now found a way to log the nodes being indexed by adding a call to the logger to NodeSearch.php. Which admittedly is a hack.

I can now see that Drupal tries to index the same nodes again and again, without success and without error messages. However, I haven't found out the reason yet.

Answer 1

The status of node indexing is based on the search_dataset. This table stores content keyword blobs and their associated sid, the primary key for the content's keyword (i.e. the nid). When compared/joined against the node table, it should let you see which nodes aren't index.

From what it sounds like, you've already spotted problem node(s) so it's just a matter of confirmation. Removing the node(s) from being indexed (e.g. hacking NodeSearch::indexNode()) to confirm it's the problem, then finding out what content in the node is blocking indexer.

Answer 2

We have a tracker to mark nodes indexed in the Drupal database in 'search_api_item' table.

\Drupal\search_api\Plugin\search_api\tracker\Basic

You can investigate the class. It will help to understand how to check if nodes were indexed. For example, after updating a node this tracker will be called from this method

\Drupal\search_api\Entity\Index::trackItemsInsertedOrUpdated

Be care with 'indexing_order': 'fifo' => $this->t('Index items in the same order in which they were saved'),

It can make problems because we have this code in the tracker class:

    // Update the status of unindexed items only if the item order is LIFO.
    // (Otherwise, an item that's regularly being updated might never get
    // indexed.)
    if ($this->configuration['indexing_order'] === 'fifo') {
      $update->condition('status', self::STATUS_INDEXED);
    }