If using jQuery is an option for you, you can do something like this:
$('#yourTextAreaID').keyup( function() {
$(this).val( $(this).val().replace( /\r?\n/gi, '' ) );
});
This code will replace the newline character on the fly, after the user types it. This code can be added in a custom.js file of your theme that you include in every page of your site, or you can alternatively use a hook_form_FORM_ID_alter in your custom module in conjunction with the drupal_add_js function to add the specific js file only in the specific form's edit page.
Keep in mind that if Javascript is disabled in the user's browser, he will be able to insert newline characters. In order to prevent this, you will have to add a custom validation function in your form. A very good tutorial on how to do this can be found here. The most important parts are:
1) Use a hook_form_FORM_ID_alter (in the tutorial he uses the old-school approach with hook_form_alter).
2) Add an extra validation function with something like:
array_unshift($form['#validate'], 'MYMODULE_form_validate');
3) Write your custom validation function (name it: MYMODULE_form_validate) that checks for \r?\n in the value of the textarea in the $form_state variable and return a descriptive error if there is a newline character.
Regarding the second part of your question, you should add some more information about what exactly didn't work. Did you clear the cache after following the steps? Do you get any errors in the log messages? Did you add console.log in your Javascript to verify that is being executed?
