You will need to create a custom module to use that hook it will allow you to run custom validation on top of the standard validation done by private msg.
A hook allows you to add functionality to a module without having to hack the module causing issues when the original module needs to be updated. When a module makes a hook available it allows other modules to perform additional processing during the execution flow in this case additional validation
more on hooks here - http://drupal.org/node/292
As a minimum your module will need a info file telling Drupal about your new module and a module file containing the hook with your extra validation code.
You can see what to put in your info file here - http://drupal.org/node/1075072
This page will guide you through setting up your module folder and naming http://drupal.org/node/1074362
Now you have your module structure set up you need to add the hook_privatemsg_message_validate() to the module file
function MYMODULENAME_privatemsg_message_validate($message) {
}
Replace MYMODULENAME with the name of your module.
From looking at this page it looks like the original message id becomes the thread id if its a reply http://api.worldempire.ch/api/privatemsg/privatemsg.api.php/group/message_hooks/7-2
array (
'mid' => 3517, // message id, identifies a message
'author' => 27, // author id
'subject' => 'Message subject',
'body' => 'Body of the message',
'timestamp' => 351287003, // unix timestamp, creation time
'is_new' => 0, // If the message has been read by the user
'thread_id' => 3341, // thread id, this is actually the mid from the first message of the thread
)
My guess is you could do something like this to test:
if($message['thread_id'] == 0 || empty($message['thread_id']) && strlen($message['body']) < 25) {
form_set_error("Message must be n chars long");
}
I'm not that familiar with the module so you might need to investigate by printing out the message array to confirm this.
Putting it all together it would look something like this
function MYMODULENAME_privatemsg_message_validate($message) {
// Uncomment next line print out the message array if you need to confirm what contiton to use for checking if first msg or not
// echo print_r($message);
if (empty($message->thread_id) && strlen($message->body) < 25 /* Change this to whatever length you want*/) {
form_set_error('body][value', t('Message must be n chars long.'));
}
}
Hopefully thats enough to get you started