Why are field widgets not form elements?

Question

Form elements are often separate from field widgets, yet they implement very similar functionality.

Sometimes, their functionality also creep across the two apis, such as in this question: Is displaying a working Field Widget Form on its own possible?

It seems logical to me that a field widget could be a form element, that also happens to map directly to some underlying storage, especially since form api was already in place when field api came about.

I'm wondering why this is not the case.

Edit: As pointed out by kiamlaluno below, according to the official docs, widgets are Form API elements, which means I need to edit my question a little.

I came about wondering about this, because I wanted to use the Core tagging widget as a form element. To the best of my knowledge however, there is no easy way of doing that. Despite the docs claiming widgets are elements, that does not seem to necessarily be true.

If a widget was declared as

function hook_widget_info() {
  return array('my_widget' => array(
    'base element' => 'some_form_element_machine_name',
  ))
}

then the element definition would be separate from the widget, so that it could be used on both ways, but currently this is not the case.

Edit 2: Feature request opened.

Answer 1

It seems logical to me that a field widget could be a form element, that also happens to map directly to some underlying storage

Widgets are form elements; they just have additional capabilities a form element doesn't have. The documentation about the field widget API describe the widgets using the following words:

Widgets are Form API elements with additional processing capabilities. Widget hooks are typically called by the Field Attach API during the creation of the field form structure with field_attach_form().

Every widget is implemented using at least one form element, but a module implementing a widgets needs to implement some hooks that are not required from a form element:

• hook_field_widget_info()
• hook_field_widget_form()
• hook_field_widget_error()

While the first can be considered the equivalent of hook_element_info(), the other two are additional hooks necessary for the widgets to work. It's thanks to the information passed to hook_field_widget_form() that a widget is rendered.

Drupal could have implemented widgets expanding the properties returned from hook_element_info(), but the truth is that a widget and a form element are two different things that are used in different cases; consider a widget a more specialized type of form element.
If hook_element_info() were used also to return information about widgets, the code looking for widgets would need to filter out the form elements to keep the widget information; vice versa, the code looking for form elements should filter out the widget information. Considering how ofter that information is required, that is the reason why two different hooks are necessary.

Drupal could have used a "base element" property returned from hook_field_widget_info(). If such property has not been implemented, it means nobody proposed it, or it has been proposed, but it has been though to be useful on few limited cases.

As for Drupal 8, things changed a little, and form elements are objects of classes implementing FormElementInterface, while field widgets are objects of classes implementing WidgetInterface. The interface that is common between those interfaces is PluginInspectionInterface, which doesn't define any method for rendering a form element, which are instead defined from RenderElement, an abstract class.
Since now widgets are implemented with classes, the base widget is essentially the base class from which a widget class is derived; there isn't the need of a property to define the base widget of a widget.