This is pretty simple to do with a little coding:
First create a "Display output" template inside your theme folder that will only target the relevant view (check options using "Theme: Information" bottom right under "Advanced" when you are on your view administration page). Copy the contents of "views-view.tpl.php" inside the file and clear the caches.
After that modify the original filter output:
<?php if ($exposed): ?>
<div class="view-filters">
<?php print $exposed; ?>
</div>
<?php endif; ?>
to what you need:
<?php if ($exposed): ?>
<div class="view-filters">
<?php if (!isset($_GET['something'])): ?>
<?php print $exposed; ?>
<?php else: ?>
<?php print l(t('Back to search'), $_GET['q']); ?>
<?php endif; ?>
</div>
<?php endif; ?>
You will just have to replace "something" inside that $_GET to a value that is actually used in your filters (check the URL when you have a search performed - it will look like example.com?something=value1&somethingelse=value2
). Since views adds all the filter keys to the URL when a search if performed you can pick any of them.