I'm using the Views Data Export module for exporting Views as .csv. Standard exports work in most cases, but with larger table only batch exports work. The problem with batch exports is that the redirects don't work well with systems using load balancing between servers and external caching services. I'd like to have something like the standard exporting, but processed the same way as batch processing so that PHP memory and execution time limits aren't reached. I've been reviewing the documentation as well as the Views Data Export module, but I haven't found a way to do this yet.
1 Answer
Redirects work just fine if your infrastructure is set up correctly. I will assume your issue is that you have a mount point for your public files directory - but not for your temp directory?
If you don't want to make two mount points (three, if you have a private folder), instead have one and just have three folders inside for tmp/ public/ private - then symlink tmp and public (e.g. tmp to /tmp/drupal).
The reason being is that file processes create temporary files which then can't be used by other instances if they're not shared. The external page cache isn't a factor here as it doesn't care so much about any of this, and I'm going to assume that this is for authenticated users too - so it's actually being bypassed (further assuming it's set up correctly). The load balancer cares even less, it just bounces you between instances unless you're using sticky sessions. Every instance should have access to the same database, files and cache - there won't be any problem.
-
Making changes in infrastructure isn't an option. We have multiple servers and multiple databases. The databases are mostly the same, but things like entity IDs are different because the data comes from another database through a syncing process. It's the multiple databases causing most of the problem. When the batch progress goes to refresh sometimes it goes to another database that has no record of the batch process and it crashes. Mar 18, 2021 at 13:26
-
Do you mean, you have a multi-master configuration but the sync isn't fast enough to keep up? They're not literally separate databases right? Entity IDs aren't different, they're just not always created? ...either way, I'd say your options are to use sticky sessions or, Drupal has the option of setting master/ slave DB connections in the settings.php - you could designate one DB as master and ensure the batch only targets that one drupal.org/docs/8/api/database-api/database-configuration Mar 18, 2021 at 14:45
-
They are completely separate databases and unaware of each other. master-slave isn't an option based on our requirements. Content is created on one database and synced to others using an API. We use UUID to keep the content synced. We are currently set up to use a Network Load Balancer which is why we have the issue of the requests bouncing between databases. We are looking into updating configuration to use Application Load Balancer with which we could use session stickiness, but that's not guaranteed. Mar 18, 2021 at 15:49