3

I have approximately 300 book-sized documents (50k up to 300k words each) that I'm trying to make searchable. I would like to display as many excerpts from each result as possible up to a maximum number.

It appears that excerpts are handled by Solr's highlighters, but there doesn't appear to be any parameters I can change to force it to show me more. The closest I've found is this:

hl.snippets - Specifies maximum number of highlighted snippets to generate per field. It is possible for any number of snippets from zero to this value to be generated. The default is 1.

but it seems to have no effect on the results returned. I've increased this to 50, along with increasing maxAnalyzedChars and fragsize to no avail.

EDIT: If I add hl.snippets=50 in a manually composed query it performs as expected - for some reason search_api_solr isn't applying the setting, it always returns 2 snippets. I've also noticed that it's not respecting my search:excerpt view settings.

Will keep updating as I persist.

I've been at this for nearly a week now and am about to start looking at other solutions. Hopefully I'm just being dense - any help is appreciated!


A little more background: I am loading up a database with several hundred manuals, from small to phone-book sized. I need the user to be able to:

  1. search the entire database using keywords and facets. The results must feature many large snippets which can be skimmed to determine which document is the correct one for their purpose
  2. Then they need to perform a second search within that document only, which they will refine until they find exactly what the need from it.

So I need the ability to display lots of verbose snippets and any extra control I can have over how they are generated would be a huge boon.

Thank you :)

[Update]

I've added

<str name="hl">on</str>
<str name="hl.fl">fieldname</str>
<str name="hl.snippets">20</str>

to solrconfig.xml and confirmed that it does cause solr to return the specified number of snippets, but I still can't figure out how to get search_api_solr do the same. It appears that no matter the settings I apply in the search index or search server, the module still insists on returning 3 or less snippets and generating them via search_api's highlight.php.

[Update] There is a setting in: Configuration -> Search API -> Search Server -> EDIT -> Advanced: "Retrieve result data from Solr" and "Retrieve highlighted snippets" that sounds like it makes drupal use solr's response instead of creating it via php but it appears to have no effect.

[Update] Ok, so now I guess I am going to attempt to create a simple module that uses hooks to modify the query and override the appropriate parameters, then hope that the results get processed properly.

[Update] Used devel and determined that Solr is returning the correct number of snippets according to my changed seettings in solrconfig.xml, so it's just a matter of getting drupal to take them and display them. I guess this means either hooking into highlight.php somewhere or just trying to hijack the query and write my own processor/template logic for it. Will share whatever solution I find. Tried to open an issue on drupal.org but it got flagged as spam. Oh bother :)

[Update] I am officially giving up. In order for drupal to work with the snippits that solr returns, "Retrieve result data from Solr" and "Retrieve highlighted snippets" must be enabled. When they are enabled however, Drupal asks Solr to send every field used for snippets for each result - in my case these fields are huge so this destroys performance. When those options are not enabled, Solr returns just the result data and highlights, but I cannot work with them because highlight.php generates it's own snippets from scratch. Oh well. My apologies to anyone who makes it here looking for a solution - and good luck!

0

[Update] Figured it out.Here's the solution I found. Somebody much more experienced than I will probably come along and offer better ideas but this works for now.

Leave "Retrieve result data from Solr" and "Retrieve highlighted snippets" unchecked, and just edit solrconfig.xml to define your snippets parameters like so:

<str name="hl">on</str>
<str name="hl.fl">fieldname</str>
<str name="hl.snippets">20</str>

Now solr will return the specified snippets in the results even if Drupal doesn't do anything with them. I cobbled together a simple module that implements hook_search_api_solr_search_results_alter (found in search_api_solr.api.php) to grab the results and store them in the $_SESSION variable:

excerpted from the solr_snippets.module I Frankensteined together:

function solr_snippets_search_api_solr_search_results_alter(\Drupal\search_api\Query\ResultSetInterface $result_set, \Drupal\search_api\Query\QueryInterface $query, \Solarium\QueryType\Select\Result\Result $result) {

    if(!isset($_SESSION['solr_snippets'])) { 
        $_SESSION['solr_snippets'] = array(); 
        }
    $myResults = & $_SESSION['solr_snippets'];
    $myResults = '';
    // Grab the result items
    $result_items = $result_set->getResultItems();
    // Grab the highlighted snippets
    $extra_data = $result_set->getExtraData('search_api_solr_response');
    $highlights = $extra_data['highlighting'];
    [ process the results here and load them back into $myResults ]
    }

Then they are picked up by a view field:

  public function render(ResultRow $values) {

    return [
        $build['search_results'] = [
           '#theme' => 'solr_snippets',
           '#results' => $_SESSION['solr_snippets'],
          ],
        ];    
  }

I also had to find a way to get the row count to the field template since it doesn't appear to be passed automatically. I didn't find the right way, but the method that worked for me was to implement hook_preprocess_views_view_field() and add it to the session variable:

function solr_snippets_preprocess_views_view_field(&$variables) {
    $_SESSION['solr_snippets']['rowNum'] = $variables['row']->index;

}

That's it, works great and I can use the twig template to customize the SERP results. Credit to https://drupalium.com/articles/creating-custom-view-fields-programmatically for the basic module that I hacked apart to make this. Also Drupalize.me was a Godsend.

Just to reiterate to anyone looking to do the same - what I did here was definitely not "best practice." Be careful and good luck!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.