fclose only creates file at script end when running drush command [closed]

Question

I'm really struggling with Drupal php memory/file/cache management. I've tried a lot of different ways to avoid memory errors (php 8.1) when writing a multi-megabyte file out. My observation is even though I've tried fflush() and fclose() on a file in previous attempts, the file only gets created at script end, not when I tell the script to create the file. I understand that file open/close is one of the most expensive operations to perform, and php is (most likely?) doing me a favor by waiting until the end to actually create the file, but AARGH! it runs out of memory. I've tried Drupal batch operations and they even fail with OOM errors. Anyone have a working drush script that could process 1M nodes, export data to a file without memory errors? [EDIT] Per @clive (btw you're awesome help!), here's my complete script. I have tried for the file access JsonCollectionStreamReader that used yield, but I think this is certainly a Drupal cache (one of the two hardest things in CS:) issue as this is the error: PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 20480 bytes) in /...../web/core/lib/Drupal/Core/Cache/DatabaseBackend.php on line 167

    <?php
    // Pass a premises nid as parameter to script.

    use Drupal\node\Entity\Node;
    use Drupal\Core\Cache\Cache;
    
    $premisesNid = $_SERVER['argv'][3];
    $premisesNode = Node::load($premisesNid);
    
    $premisesName = str_replace(' ', '', $premisesNode->getTitle());
    $permitsFile = 'public://PermitData-' . $premisesName . '.json';

    $fileOut = fopen($permitsFile, 'w') or die('Cant open output file!');

    $permitNids = Drupal::entityQuery('node')
      ->accessCheck()
      ->condition('status', 1)
      ->condition('type', 'permit')
      ->condition('field_property', $premisesNode->id(), '=')
      ->execute();

    $exported = 0;
    foreach ($permitNids as $permitNid) {
      $keyValueData = nodeKeyValuePairs($permitNid);
      $permit['permitData'] = $keyValueData['values'];

      // Collect the transactions for this permit.
      $transactionNids = Drupal::entityQuery('node')
        ->accessCheck()
        ->condition('status', 1)
        ->condition('type', 'transactions')
        ->condition('field_trans_parent', $permitNid, '=')
        ->execute();

      foreach ($transactionNids as $transactionNid) {
        $trans = nodeKeyValuePairs($transactionNid);
        $permit['trans'][] = $trans;
      }
      $line = json_encode($permit) . PHP_EOL;
      fwrite($fileOut, $line);
      $exported++;
      Drupal::service('entity.memory_cache')->deleteAll(); //<--Solution!
    }
    fclose($fileOut);
    echo 'Exported ' . $exported . ' permits.';

    function nodeKeyValuePairs($anyNid): ?array {
        $values = [];
        $anyNode = Node::load($anyNid);

        if (empty($anyNode)) { // Node does not exist.
          return $values;
        }

        foreach ($anyNode as $key => $value) {
          $values['values'][$key] = $anyNode->get($key)->getValue();
        }

        $tags = ['node:' . $anyNode->id()];
        Cache::invalidateTags($tags);
        return $values;
    }