To stop a node being created, you need to implement hook_node_validate instead, and raise an error. The hook is specifically to:
Perform node validation before a node is created or updated.
Which is what you want.
The example in the docs page is probably quite close to what you'll need:
function hook_node_validate($node, $form, &$form_state) {
if (isset($node->end) && isset($node->start)) {
if ($node->start > $node->end) {
form_set_error('time', t('An event may not end before it starts.'));
}
}
}
For the record, hook_node_submit is both conceptually the wrong place to try to stop a node being created, and too late in the process to do so anyway. The node hasn't been saved by that point, but you've lost any simple opportunity to stop the remaining submit handlers running and the default process completing. Even if you found a way to short out that functionality, there would be no sensible method to feed that back to the user.