I'm using the Views Data Export module for exporting Views as .csv. Standard exports work in most cases, but with larger table only batch exports work. The problem with batch exports is that the redirects don't work well with systems using load balancing between servers and external caching services. I'd like to have something like the standard exporting, but processed the same way as batch processing so that PHP memory and execution time limits aren't reached. I've been reviewing the documentation as well as the Views Data Export module, but I haven't found a way to do this yet.
Redirects work just fine if your infrastructure is set up correctly. I will assume your issue is that you have a mount point for your public files directory - but not for your temp directory?
If you don't want to make two mount points (three, if you have a private folder), instead have one and just have three folders inside for tmp/ public/ private - then symlink tmp and public (e.g. tmp to /tmp/drupal).
The reason being is that file processes create temporary files which then can't be used by other instances if they're not shared. The external page cache isn't a factor here as it doesn't care so much about any of this, and I'm going to assume that this is for authenticated users too - so it's actually being bypassed (further assuming it's set up correctly). The load balancer cares even less, it just bounces you between instances unless you're using sticky sessions. Every instance should have access to the same database, files and cache - there won't be any problem.