The only place in core that checks for, $form_state->get('rerender')
in an if
statement is in:
/core/modules/views_ui/src/Form/Ajax/ ViewsFormBase.php
// If the form has not been submitted, or was not set for rerendering, stop.
if (!$form_state->isSubmitted() || $form_state->get('rerender')) {
return $response;
}
// Sometimes we need to re-generate the form for multi-step type operations.
if (!empty($view->stack)) {
$stack = $view->stack;
$top = array_shift($stack);
// Build the new form state for the next form in the stack.
$reflection = new \ReflectionClass($view::$forms[$top[1]]);
/** @var $form_state \Drupal\Core\Form\FormStateInterface */
$form_state = $reflection->newInstanceArgs(array_slice($top, 3, 2))->getFormState($view, $top[2], $form_state->get('ajax'));
$form_class = get_class($form_state->getFormObject());
$form_state->setUserInput([]);
$form_url = views_ui_build_form_url($form_state);
if (!$form_state->get('ajax')) {
return new RedirectResponse($form_url->setAbsolute()->toString());
}
$form_state->set('url', $form_url);
$response = $this->ajaxFormWrapper($form_class, $form_state);
}
elseif (!$form_state->get('ajax')) {
// if nothing on the stack, non-js forms just go back to the main view editor.
$display_id = $form_state->get('display_id');
return new RedirectResponse(Url::fromRoute('entity.view.edit_display_form', ['view' => $view->id(), 'display_id' => $display_id], ['absolute' => TRUE])->toString());
}
else {
$response = new AjaxResponse();
$response->addCommand(new CloseModalDialogCommand());
$response->addCommand(new ShowButtonsCommand(!empty($view->changed)));
$response->addCommand(new TriggerPreviewCommand());
if ($page_title = $form_state->get('page_title')) {
$response->addCommand(new ReplaceTitleCommand($page_title));
}
}
The if (!empty($view->stack)) {
part will only pass if the form is a multi-step form and you're not in the last step; otherwise, it will proceed to close the modal. Ex: when you click "Apply" or "Cancel" or "Remove". (Ex: when adding a field) it triggers the else:
else {
$response = new AjaxResponse();
$response->addCommand(new CloseModalDialogCommand());
$response->addCommand(new ShowButtonsCommand(!empty($view->changed)));
$response->addCommand(new TriggerPreviewCommand());
if ($page_title = $form_state->get('page_title')) {
$response->addCommand(new ReplaceTitleCommand($page_title));
}
}
Conclusion
As it only gets checked in the views_ui module, this setting is basically a way to tell the Views UI Form:
if the setting form is a multi-step form, like when you're adding a field:

to keep the modal open and build a new $form_state
for the next form

Otherwise, its like when you click apply/cancel/remove it tells it to close the modal, show save & cancel buttons, and show/refresh the live preview.