You should use yoursite.make to download the Drupal module and composer.json for composer.
There is the Composer module that integrates Composer with Drush.
I hope that soon .make is going to disappear, replaced by composer.json; for that to come true, some issues need to be resolved, such as version number.
Composer packages are stored in the git repository; you can't use a composer package, but you can download the code. For example, for https://packagist.org/packages/symfony/symfony you can use the following.
libraries[symfony][download][type] = "git"
libraries[symfony][download][url] = " https://github.com/symfony/symfony.git"
libraries[symfony][destination] = "lib/symfony/"
Composer can build stuff recursively; of course, this will not work here.
See http://drush.ws/docs/make.txt.
Their is also https://github.com/composer/installers than can do the invert (install drupal module with composer).