Say I drag and drop files from a source site into the folder structure of a new site (destination site) that I'm migrating content to.
Dont do this. Migrate D2D has file handling classes to move the files for you. In the UI it's super easy, in code you select the correct file handing class.
Read the documentation on migrating files for D2D. There are 4 cases it supports, depending on how the remote file needs to be fetched.
In the D2D UI you just want to be sure to setup the File_class value for your migration ok. Typically the Files are migrated (moved) to Drupal as 1 of the 1st auto-generated migrations. Then in Node content that relates the migrated files to the (migrated) Node you set file class to FileFID
and select the previous Source Migration that moved the file into the new site.
EDIT
So for example with a website with existing image fields in a content type. On the destination Drupal 7 side of the migration I tell it the legacy field and tell the configuration to use MigrateFileFID
this will map just the file ID between the 2 drupal instances:

See field_ofellow_photo
above for an example of this configuration for D6 -> D7.
Then Migrate will see the information for the migration steps:

If you use MigrateFileFID
then Migrate doesnt (necessarily) move the files I believe if you leave off the destination information, it just moves FIDs. See the documentation on MigrateDesinationFile and MigrateFileFid.