Cache max-age and varnish ESI

Question

I need to display a custom block on my homepage. This custom block retrieves some data that are not managed by Drupal. Since no event will happen in Drupal to say that those data have changed, the max-age cache property seems the way to go, and it's OK for me. The block appears exactly as expected, but the problem is thst the cache for this block seems to never expire. Actually, the block is never refreshed.

After researching, I found Setting cache max-age to 0 has no effect on Block built using BlockBase? that explains max-age cache property doesn't work as expected for anonymous requests. After adding the mentioned trigger, everything seemed to be all-right on my development environment.
Once on the production environment, it's the same as before: The block is never refreshed. Talking to the hosting provider, I was told that Varnish is used to cache the whole page, and I should use an ESI (Edge Side Include) to render this block with its own cache logic managed by Varnish.

I don't really know how to implement it in a clean way in Drupal. My first idea is to use a controller, render the block markup, and return in a response, forcing the needed cache-control header. This is the code I thought of for such controller.

$block_manager = \Drupal::service('plugin.manager.block');
$plugin_block = $block_manager->createInstance('my_block_id');
$buildedBlock = $plugin_block->build();
$renderedBlock = \Drupal::service ('renderer')->render ($buildedBlock);

$response = new Response(
    $renderedBlock,
    Response::HTTP_OK,
    array('content-type' => 'text/html')
);
$response->setSharedMaxAge(60);
return $response;

I feel it is a totally unclean way to do this. Does anybody a better approach to propose? Would my code work?

Answer 1

max-age 60s is pretty low for the frontpage. Does the data really update that frequently or is it just that it could update but usually it doesn't happen every minute?

One possible approach to solve this would be to fetch that external data on a cron job that runs every minute. If and only if you detect a change, you invalidate a cache tag, which invalidates your frontpage. This approach is interesting when the data does not actually change that often most of the time. Because your cache is valid as long as possible.

And it has the added benefit that you do not rely on the external system when users are actually accessing your frontpage. Your block simply delivers the data from state which you update in the cron job.

That said, if you do want to influence the max-age header, then you need to implement your own Response subscriber. You could also use a module like https://www.drupal.org/project/cache_control_override, but you have to carefully test different pages to see if it might disable caches in more places than you expect, e.g. a language switcher block can currently do that.

And uninstall the Internal Page Cache, it caches even uncacheable responses like this and you don't need it if you use Varnish

If and only if you have cache tag invalidation configurated and integrated with Varnish. If not, my recommendation would be to use the internal page cache module and set the default external max-age to a very short time, e.g. 1 minute. Then varnish can cache responses for that long and will revalidate against the internal page cache after that time.