The form values still persist in the form state, and as they're not printed, the will be submitted as empty and saved accordingly.
Do not hide form element by not outputting them in twig, render the full form there and use [#access] = FALSE; on the form elements you don't want to show. Do so in the controller, to only have that on your custom page. If you do that in hook_form_alter it will apply to the form always.
Anyway, I would use a different approach .. and create a new form from scratch, populating with the user's value, and save them back to the user entity on submit. (If you only need one "field" and not fe. the password fields - although they could also be replicated in the custom form.