"The import is empty" errors validating config synchronization after core upgrade

Question

After Drupal 9 upgrade, I can no longer use configuration synchronization to deploy configuration to Acquia Cloud environment:

 Import the listed configuration changes? (yes/no) [yes]:
 > >  [error]  Drupal\Core\Config\ConfigImporterException: There were errors validating the config synchronization.
> This import is empty and if applied would delete all of your configuration, so has been rejected. in Drupal\Core\Config\ConfigImporter->validate() (line 750 of /mnt/www/html/d9readiness/docroot/core/lib/Drupal/Core/Config/ConfigImporter.php). 
> 
> In ConfigImportCommands.php line 360:
>                                                                                
>   The import failed due to the following reasons:                              
>   This import is empty and if applied would delete all of your configuration,  
>    so has been rejected.                                                       
>                                                                                
> 

The configuration settings are the same ones that worked before the upgrade:

$ grep -Rn '../config/default' .                  
./docroot/sites/default/settings.php:258:// $config_directories['vcs'] = '../config/default';
./docroot/sites/default/settings.php:259:// $config_directories['sync'] = '../config/default';
./docroot/sites/default/settings.php:262:$settings['config_sync_directory'] = '../config/default';
./docroot/sites/default/settings.php:263:$settings['config_vcs_directory'] = '../config/default';

My suspicion is that Acquia is overriding my settings somewhere outside of my document root, but I can't quite find it.

Local environment is Lando with the new-ish acquia recipe, so perhaps there is some legacy config that comes with acli or something.

I'm hoping to install BLT 12 as soon as I deal with the last couple of weird errors in my local; maybe that will help me to fix the config sync issue since it comes with its own configuration overrides.

This may be related to the following error reported by the Upgrade Status module before Drupal 9 upgrade: https://www.drupal.org/project/upgrade_status/issues/3229552#comment-14209685

UPDATE: via Drupal Slack

mcdruid 6 hours ago

My suspicion is that Acquia is overriding my settings somewhere

...probably not answering your question directly, but you should be able to check any of the settings with e.g.

$ drush ev 'print_r(\Drupal\Core\Site\Settings::get("config_sync_directory"));'

I tried this, and indeed it confirms that Acquia is using different configuration which is not in my settings.php.

Here is the output from running that command in my local and on Acquia Cloud:

$ lando drush ev 'print_r(\Drupal\Core\Site\Settings::get("config_sync_directory"));'     
../config/default%

$ lando drush @dev ev 'print_r(\Drupal\Core\Site\Settings::get("config_sync_directory"));'
Enter passphrase for key '/var/www/.ssh/id_rsa':
/mnt/www/html/d9readiness/docroot/sites/default/files/config_315f86d0f57aa166d80058769033a13b1b4823d0/syncConnection to d9readiness.ssh.devcloud.acquia-sites.com closed.

$ lando drush @test ev 'print_r(\Drupal\Core\Site\Settings::get("config_sync_directory"));'
Enter passphrase for key '/var/www/.ssh/id_rsa': 
/mnt/www/html/d9readiness/docroot/sites/default/files/config_72c878fc6029f4e12c65fa9d1ce2afd4c3a94575/syncConnection to d9readiness.ssh.devcloud.acquia-sites.com closed.

This proves that Acquia is overriding my configuration settings.

Just to verify this, I grepped for the Dev configuration directory in my local project and no results were returned:

$ grep -Rn 'sites/default/files/config_315f86d0f57aa166d80058769033a13b1b4823d0/sync' .
$

In case the public files directory path segment was provided by stream wrapper, variable expansion or string concatenation, I tried again with only the config identifier:

$ grep -Rn '315f86d0f57aa166d80058769033a13b1b4823d0' .
$

No results again. This config is coming from someplace weird for sure.

Could it be this require near the end of settings.php?

// On Acquia Cloud, this include file configures Drupal to use the correct
// database in each site environment (Dev, Stage, or Prod). To use this
// settings.php for development on your local workstation, set $db_url
// (Drupal 5 or 6) or $databases (Drupal 7 or 8) as described in comments above.
if (file_exists('/var/www/site-php')) {
  require('/var/www/site-php/d9readiness/d9readiness-settings.inc');
}

Note: This config is at least two years old. The project is really not called d9readiness (imagine it's called client-project-name).

Answer 1

Even though I never was able to figure out where the old D8 project config was coming from, I was able to resolve the problem by explicitly overriding the config at the very end of settings.php:

// Replace Acquia Dev Desktop settings (deprecated) with Acquia Cloud environment overrides.
// Note: This is for troubleshooting only; remove this code after configuring BLT 12.
if (isset($_ENV['AH_SITE_ENVIRONMENT'])) {
  switch ($_ENV['AH_SITE_ENVIRONMENT']) {
    case 'dev':
      // do something on dev
      $settings['config_sync_directory'] = '../config/default';
      $settings['config_vcs_directory'] = '../config/default';
      break;
    case 'test':
      // do something on staging
      $settings['config_sync_directory'] = '../config/default';
      $settings['config_vcs_directory'] = '../config/default';
      break;
    case 'prod':
      // do something on prod
      // Site Factory may require a different value depending
      // on site configuration
      $settings['config_sync_directory'] = '../config/default';
      $settings['config_vcs_directory'] = '../config/default';
      break;
    case 'ra':
      // do something on ra - necessary if a
      // Remote Administration environment is present
      $settings['config_sync_directory'] = '../config/default';
      $settings['config_vcs_directory'] = '../config/default';
      break;
    }
  }
  else {
  // do something for a non-Acquia-hosted application
  // (like a local dev install).
  $settings['config_sync_directory'] = '../config/default';
  $settings['config_vcs_directory'] = '../config/default';
}

I have set the same default config for all environments here just to get a basic config sync working again, but my intent is to remove this code after setting up BLT 12. (BLT has its own overrides in different locations, which I don't want to mess up by overriding here.)