I think you are really going to struggle to get this to work using the default taxonomy term page view. However, you can disable the view and then use the default entity full content display for terms and do whatever you'd like for it (just like a node page).
Disable the taxonomy term display that controls /taxonomy/term/%tid. This will allow the term's ...
Objective: Show upcoming events, then past events.
The simplest way to implement this entirely through the View UI is to add a second display to the view.
Create the main view display. Configure the events to display as you like.
Set to sort by upcoming events, and filter so that only upcoming events are displayed.
Add an Attachment display. ...