If I understand correct, I had a similar issue once: It's a known problem; The fact the Drush's cache isn't separate and being cleared in any general cache-clearing is problematic especially in sites where there are only tens or hundreds of users daily (and not thousands or much more) because than you need to wait everyone to rebuild the cache for you.
We can hope that in Drupal 8 the Boost's caching system will be totally or partly separated from the general caching system...
If you actually meant to ask what is the fastest way to clear cache than instead navigation to the address you posted (with or without a browser bookmark) you could install Admin_menu for Drupal 7 or Admin_toolbar for Drupal 8 and they would not only give you a cool icon to clear cache instantly on every page but a comfortable management menu on top of your browsers fold.
In Drush master you could do "
drush cr" (this stands for drush cache rebuild).
That's aside of using cache expiration mentioned in other comments.