9

I've seen a number of strategies for integrating remote data structures in Drupal. The strategies have seemed to evolve as certain modules have stabilized and use cases have been tried.

Imagine we have a data structure "Farmers Markets" that is represented by a number of data types (market, market_hours, vender, stall, produce) etc that are exposed via a REST API. The IDs for the external service would need to relate in Drupal, ie when loading a "market" we would want to grab data from the 'market_hours' and 'stall'. What would be the best way to represent that as read-only content in Drupal that is synced on a regular basis?

I'm trying to evaluate this with the following criteria:

Data Structures in Drupal:

Nodes vs Custom Entities

A number of scenarios involving web services I've seen use custom entities. It simplifies the CRUD ops. However, these items would be "content" in that they would be viewed publicly.

Storage (Local vs Remote):

I've seen a couple of examples where services are loaded as remote entities, which this module creates a library for: https://drupal.org/project/wsdata . This sounds most appealing, but haven't seen a lot of use cases. There also examples of custom code: https://drupal.org/sandbox/fago/1493180

Syncing Data:

Feeds vs Migrate vs Guzzle vs 'Web Service Client' vs 'Web Services Data'

There are a number of options. Feeds now supports entities. Migrate seems a lot cleaner than feeds, especially for custom scenarios. I've also seen folks using a guzzle client to grab sync with remote services: http://drupalcode.org/project/ckan.git/blob/refs/heads/ckan_dgu_7.x-1.x:/ckan.drush.inc#l273. I've also noticed the WS Client module https://drupal.org/project/wsclient provides an option that has been created specifically as a rest client. Web Services Data loads directly from a service and caches it locally.

Thanks for any thoughts.

  • I'm not sure anyone can give you a definitive answer on what it the most reliable and fault tolerant solution for your specific use case. – rooby Sep 16 '13 at 15:09
  • The "data" module is another possibility, which can be used in conjunction with the feeds module (currently needs the solution in this issue - drupal.org/node/1033202 ) – rooby Sep 16 '13 at 15:11
  • Using data module would just allow us to store the data in individual tables. This would be fine for creating lists through views but wouldn't allow us to use the benefits of the entity system (whether nodes or custom entities). – acouch Sep 16 '13 at 19:50
  • Yes, the data module has a submodule data_entity, which makes entities of all your data items. – rooby Sep 17 '13 at 1:33
16
+125

1. Reformulating the question

Your example suggests that the data is read only on the Drupal side, with one-way syncing only. I think this is the most important factor to consider here, because in effect whatever solution you implement will be a variant of remote storage, syncing and local caching - even if the local caching ends up being entities in the Drupal database.

So the question, rather than being "local storage vs remote storage" is going to be:

  • Should you cache the data locally at all ;
  • Should you cache the data as actual entities, and use Feeds (or similar) to synchronise the data regularly ; OR
  • Should you use some custom made module that provides the syncing and the caching.

An article you might be interested is in "Remote Entities in Drupal 7".

2. Caching the data

In general, caching the data is a good idea:

  • You are protected against outages of the other services, or timeouts in the connection ;

  • Having your data present in your Drupal database will speed up operations ;

  • Having your data present in your Drupal database will mean that you are more likely to get integration with other modules, such as Views (though this is not guaranteed).

The only advantage of not caching the data is that you never get stale data, which in some cases is preferable - sometimes it is preferable to display no data rather than stale data. I do not see this as a benefit in the example you have given, so I will focus this answer on a solution that involves local caching.

3. Local entities + Syncing

If you go for the option of having local entities and syncing them yourself, then we come back to your original questions:

  • Should you use nodes or custom entities ;

  • Which module is best for syncing.

3.1 Nodes vs Custom entities

  • The definition of what exactly is a node is one that is quite open. The documentation page on nodes suggests that nodes are "posting" that are "stored" on your site - neither of which apply to your data ;

  • As a Drupal developer I would expect that if something is a node then I should be able to manipulate it on the site itself ;

  • As Drupal user, I would similarly expect that nodes can be edited ;

  • This Drupal 8 issue https://drupal.org/node/2019031 suggests that the concept of "read only" is one that would apply at the entity level, rather than the bundle level. If this ever gets implemented you would benefit from it by having gone down this route.

To summarise: your data being read only and stored remotely it makes more sense to use a custom entity type to represent your data.

3.2 Syncing

For the second part, the two main modules for this are, as you suggest, Feeds and Migrate.

The difference between Feeds and Migrate is that Feeds is build for regular importing of content, while Migrate is build for one-off porting of content from one place to another. Migrate does support updating existing data, however given that both modules are well supported it makes more sense to use the module that was build for the task at hand - Feeds is a better match.

Having used both modules myself (Feeds for syncing, Migrating for migrating) I do not find Feeds to be more messy than Migrate. Migrate has required more custom code in my experience, though migrating entire sites is more complex than importing single content types, so it's hard to compare.

4. Custom module for remote storage, syncing + caching

There are a number of modules out there that can help with this task.

You mentioned the Web Services Data module, and others have mentioned the Data module. Another option to consider is the Remote Entity API module. Note that the only one of those I have experience with is the Data module.

  • The Web Services Data module does not yet have a release - which may indicate the code is not yet stable, the API might change and so on. It does not support Entity Field Queries (according to it's project page), and a quick browse of the code repository shows no evidence of it having Views support - so you would not be able to use Views to display your entities ;

  • The Data module is more oriented, in my experience, towards non-developers who have data in a table and want to expose it to views. I have found the Drupal 6 version to be quite frustrating to use - though that might have changed since then ;

  • The Remote Entity API module sounds quite promising - it supports fetching and caching of remote entities, and has Views support. It's only on alpha release - so API might still change. At first glance it does not seem to have Entity Field Query support either, and it only support one type of remote service so you would have to implement your own.

Conclusion

Given that none of the remote storage modules support Entity Field Queries, using actual entities + feeds is the solution that will give you the best integration with your Drupal site.

If Views support is enough, and you are not worried about potential integration with other modules via Entity Field Queries, then using the Remote Entity API might be the way to go - you would need to implement your own remote interface however.

0

If you need views, rules, tokens, cruds hooks, search api and definitly strong system integration in my opinion they can´t be considered nodes but they must be custom entities with their own idiosyncrasy storing in the database the "entity id" and the "external id" relation and with the retrieving information calls encapsulated in the "entity properties". Finally, whatever the tool you choose for syncing data, I would process it with cron Queues.

If you just need retrieve and expose data punctually I think a better choice would be create an Interface class for retrieving external data and implement this Interface with a Class that retrieve the info from your "Farmers Markets" structure.

Regards

0

There's the Field Storage API, which allows for pluggable storage back-ends other than the default database engine.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.