Search API uses flags in the
changed column of the
search_api_item table to store item-level information relating to indexing. The -1 flag (something to do with queuing items?) appears to be crucial to a problem I'm facing, where it is set on items which are then never indexed.
I've studied the code and all the documentation and relevant issues I can find, and I can't figure out in nearly enough detail what the practical implications of this flag are: what it being set causes or prevent from happening (and therefore, where I should focus next in trying to solve my problem). Since Search API is an API module which plenty of developers work with, and these flags are a core part of its logic and an important part of the contrib-accessible data it records, I'm hoping that someone will be able to help explain what a -1 flag means and what it causes to happen (or what it prevents).
For context, the specific problem spurring me to ask (with Search API using Search API Solr on Drupal 7) is that indexing stops abruptly - cron runs continue to work fine, cron requests reach the search server with no problem, but a large portion of my indexable items are never indexed. There are no error messages or relavant information in any of the appropriate logs (drupal watchdog, cron, apache solr, tomcat). It's a classic debugging nightmare: something is failing silently and leaving no trace. But there's one clue: in the
search_api_item table on the database, I can see that, when Cron reaches those items where indexing is failing, the
changed column changes to a '-1'. The notes for that column say:
Either a flag or a timestamp to indicate if or when the item was changed since it was last indexed.
After much digging in code I found what appears to be the place these flags are set ( in
search_api/includes/datasource.inc ) and there appear to be three such flags:
- -1, set in
trackItemQueued(): the one which looks to be crucial to my problem. According to the comments it means "queued" - but I can't find the crucial details of what this causes to happen (or prevents from happening) next. If I've understood this right, then this flag being set should be a sign that everything is working fine: but the items aren't indexed, and I can't see what the expected next step should be.
- 0, which apparently means already indexed (presumably causing these items to be passed over)
- 1, set in
trackItemInsert(), and I can't figure out anything about what this one means
- (and timestamps apparently signify "changed/dirty", which I imagine means "no longer matches the index data as of [timestamp], needs to be queued for re-indexing")
Search API is an API module, so there will be a good number of Drupal developers who know how it works on the inside. Do you know what the -1 flag indicates - in terms of what then happens to items flagged ("queued", or, "marked to be queued") in this way? What is the role of this flag in the process and what happens next?
(bonus points for any expert advice on the next step to take in debugging a problem like mine: but the focus of my question is general, about understanding the Search API item flags. My issue and attempts at debugging it are intended mainly as context. In case it's relevant further context, I'm also running Ultimate Cron and its dependency Background Process to handle the cron runs, to reduce search server strain by allowing lots of small indexing batches called by cron every few minutes without overdoing other drupal cron functions. This part of the process seems to work fine: the search server is reached perfectly in sync with the Ultimate Cron schedule as expected. It's what happens after the item flags are checked and updated that is leaving me stumped)
(I've also posted a related support request on the Search API issue queue, without a response. I'm judging it's okay to ask here as well because a) this meta discussion explicitly says that also posting questions that are support questions also asked in issue queues here as well is okay and also a desirable way to take the pressure off busy module maintainers, and b) this is particularly true for Search API as the maintainer has said in the past that he doesn't have much time available to help with support issues. As adivsed on Meta, I'll update both threads with any relevant information)