2

I have a site in which I associate multiple Config Entities with a single Queue ie. example_queue (using the default QueueInterface).

My Config Entities produce records that are processed as queue items via that Queue.

Each of these Config Entities have the ability to enable or disable the processing of their records via that queue, so for that I am utilising the SuspendQueueException which I only throw when I know that the items from Config Entity A are disabled for processing.

Because I am using a single Queue, this now also suspends the queue processing for records from Config Entity B which is enabled.

I was thinking of either just getting rid of the SuspendQueueException, but that would then needlessly fail my queue items for Config Entity A, which I already know are failing, OR I would have a Queue per Config Entity. But then I would need to define a QueueWorker class per Queue, I think? Which could become a problem if I have let's say 1,000 Config Entities, each needing such an individual QueueWorker class?

How should I address this best? Or how can I have 1 QueueWorker utilise multiple Queues that I know need the same processing logic?

This is my ExampleQueueWorker class:

namespace Drupal\example\Plugin\QueueWorker;

use Drupal\Core\Plugin\ContainerFactoryPluginInterface;
use Drupal\Core\Queue\QueueWorkerBase;
use Drupal\Core\Queue\SuspendQueueException;

/**
 * Process a queue of items.
 *
 * @QueueWorker(
 *   id = "example_queue",
 *   title = @Translation("Example Queue worker"),
 *   cron = {"time" = 60}
 * )
 */
class ExampleQueueWorker extends QueueWorkerBase implements ContainerFactoryPluginInterface {

  /**
   * The maximum amount of times a single queue item is allowed for retries.
   */
  public const MAX_RETRY_THRESHOLD = 5;

  ...

  /**
   * {@inheritdoc}
   */
  public function processItem($data) {

    // $obj can have a parent of Config Entity A, B, C, etc...
    if (!$obj = \ExampleRecord::load($data->getId())) {
      $this->logger->error('Invalid item.');
      return;
    }

    try {
      // Process or halt depending on $obj parent settings.
      $this->ExampleService->post($obj, TRUE);
    }
    catch (\Exception $e) {
      // Log failure.
      $this->logger->error('ID: %id, Error: %error', [
        '%id' => $obj->id(),
        '%error' => $e->getMessage(),
      ]);

      // Get retry count.
      $retry_count = (int) $obj->getRetryCount ?: 0;

      // Increment and save retry count.
      $obj->setRetryCount(++$retry_count)->save();

      if ($retry_count >= self::MAX_RETRY_THRESHOLD) {
        return;
      }

      // Treat a ServiceOutageException as a reason to suspend the queue.
      if ($e instanceof \ServiceOutageException) {
        throw new SuspendQueueException($e->getMessage());
      }

      // And any other Exception for default queue error handling.
      throw new \Exception($e->getMessage());
    }

  }

}
5
  • Yeah every QueueWorker is defined by a queue ID in its annotations to later have data added to this exact queue. I guess this sounds you simply provide multiple queue workers, which of course could as extend themselves to keep things DRY if that's your concern. class MySecondQueueWorker extends MyFirstQueueWorker implements ContainerFactoryPluginInterface and then call parent::processItem($data); inside its public function processItem($data) function.
    – leymannx
    Nov 20, 2020 at 18:43
  • Thanks for your feedback. However I do potentially have 1,000 of those Config Entities, which would mean I have to create 1,000 QueueWorker classes then?
    – baikho
    Nov 20, 2020 at 18:48
  • 1
    Uh, okay. Maybe add that number to your question. Can you maybe elaborate a bit what this QueueWorker is doing exactly? And maybe add a little bit of sample code?
    – leymannx
    Nov 20, 2020 at 18:57
  • 3
    One approach would be to create derivatives, e.g. drupal.stackexchange.com/questions/293667/…. Your getDerivativeDefinitions method would query all of your config entities and you'd end up with a separate queue per entity
    – Clive
    Nov 20, 2020 at 20:34
  • Thanks @Clive, that looks promising! I'll give it a go
    – baikho
    Nov 20, 2020 at 21:15

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.