I ran into an issue recently, where I have a piece of code that accesses content from an external source. The content generation from this source is quite slow (3-5 seconds), but it only needs to be refreshed every 30-60 seconds.

Currently it's a bit of a Russian roulette. Every minute a user will receive a generally slower page load, while the content is refreshed and cached, and then other people will receive the cached version. Naturally, this is a headache for the users and for our apdex score.

I was trying to find a way to refresh the content in a non-obtrusive way, either automatically via scheduled action sets, or in background process when the user load the page, but I've failed with both. Normally it would not be an issue (simple enough using Elysiacron), but on Pantheon I don't think it's possible to manipulate the crontab in the fashion I require it.

Did anyone face a similar issue on Pantheon so far? Is there any solution that would solve my problem within Pantheon? Can something like this be achieved with AJAX?

I have an idea for a workaround, which would involve a third party machine requesting the part of the module that generates and caches content every 30 seconds, but that is a really ugly and inelegant way to do this.

Thanks for any input regarding this!

  • Ajax as you suggest should be doable. The visitor runs a piece of JS which calls your site, which then updates the cached content. Though I also don't think this question is really related to Drupal, so I think this will get closed as off-topic. – Letharion Jan 7 '14 at 22:42
  • 1
    @Letharion sure it is, its essentially a performance question. The easiest Drupal solution isn't meeting his needs; or requirements. See my silly 2 cache solution below. Parts of Drupal are slow when you have a bunch of stuff (like Taxonomy terms, or even just Nodes) -- imho "scaling" it or building it correctly is a valid question in the community. – tenken Jan 7 '14 at 23:54
  • Are you saving content from the external source into your drupal website?? – Sudheesh Damodaran Jan 8 '14 at 11:50
  • @tenken Sure, scaling Drupal is absolutely on topic, but the way the question is written, it's about Pantheon, and cron, and only happens to mention Drupal, as if in passing. If the question was, for example, "How to retrieve external data without impacting UX", I would have considered it on-topic. As it stands though, I don't. – Letharion Jan 8 '14 at 11:52
  • Actually, it is related to a Drupal module's development that I am working on. On Pantheon's support site it was mentioned that they monitor StackExchange so if we have any questions just ask here with a Pantheon tag, because they are monitoring it. Although I am a bit afraid of AJAX, I will try to go ahead and attempt to rewrite the module in a fashion that would work with it. – Kaz Jan 8 '14 at 17:07

Instead of updating a global cache of remote objects (which a visitor might see a stall on while data is being fetched ...)

Simply use 2 caches. 1 cache is what visitors see data from from say a Drupal cache bin. The second cache bin is another private drupal cache bin updated by your module from the remote host. You could make custom cache tables in Drupal or simply use raw MySQL tables without the Drupal API if you're not familiar with it.

Then on occasion do a database table flip (eg, truncate cache2; select cache1.* INSERT INTO cache2;, or sync from 1 Drupal cache bin to the other. This local cache store updating should be faster than the remote server.

  • I've used similar multilevel cache strategies with decent results, but you periodically do get total cache misses. – mpdonadio Jan 8 '14 at 2:27
  • Thanks for the reply! Either I expressed myself poorly, or I misunderstood something very fundamental. The basic concept here is that there is a TCP service on server A, which provides an interface on a port to query data. It only returns raw text, without caching. When the module generates data it connects to this service and retrieves data - this takes 3-5 seconds. In Drupal, you have this very clearly defined thing called a request. So unless a user (or cron) triggers the code it will not do stuff on it's own. And the request is not complete until this piece of slow code is finished. – Kaz Jan 8 '14 at 17:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.