I've also been beating my head against a wall the last while with Drupal memory issues. Here's my collected knowledge on the topic:
1. Views (can) use a lot of memory
I love me some Views (and Panels and CTools and everything merlinofchaos touches with his mighty, mighty fingers), but it's possible to create configurations with multiple relationships that use a lot of memory. If you disable your Views and the memory issue goes away, it's likely a badly-constructed View causing the issue.
What to do if it is a View, and you really need that View to work? Try putting it into code (Via Bulk Exporter or Features; see below. I've hand-coded Views-like functionality in order to improve performance with very little success) for a start. Another thought is to redo the View a different way -- if ultimately what you're getting at is taxonomy terms, make sure the type of View is a Taxonomy View when creating it; don't create a Content View that uses relationships to get at taxonomy terms.
Could also be worth mentioning here that Panels also supposedly uses a lot of memory -- I haven't really benchmarked it so can't confirm this.
2. Moving stuff from database into code is a very good practice
It took me several Drupal sites to realize this, but keeping everything that's created via a UI in the database (Views and Panels configurations especially) is a Drupal worst practice. Why? It increases load on the database and cannot be version controlled. The first point is particularly problematic in terms of memory usage -- instead of just loading the content from the View from database, the site must also load the view components itself. This is exacerbated by how Drupal uses tables: by abstracting everything to the nth degree, each bit of Drupal's functionality uses a new table, resulting in some requests joining a bajillion tables together. This gives Computer Science people hernias (caveat: link is silly), but can't really be avoided with a piece of software as modular and user-friendly as Drupal is.
The solution? Use Bulk Exporter (Included with CTools) or Features to package up bits of code currently residing in the database as module code.
3. Themes can also eat memory
Does your theme have a lot of template files (I.e., files in themename/templates/)? If so, memory is consumed each time one of those files is loaded. If you're creating templates specifically to suppress bits of Drupal from being displayed, try either:
- Changing permissions so that bit doesn't show up for specific non-admin user roles.
- Using CSS to hide elements.
The first choice is obviously what you want to do for stuff affecting security -- you don't want to use CSS to hide an "edit" button for certain users, only to have them then unhide it via Firebug or whatever.
4. Don't go overboard on contrib modules
While sometimes a site needs a lot of contrib modules, don't go overboard. Each enabled (note: enabled. Disabled modules don't use any memory) module uses memory. This is a bit obvious, but worth noting regardless.
5. The VPS is (sometimes) a lie
In my experience, some unscrupulous hosting companies love trying to push Drupal sites to VPS servers -- they're more expensive and it frees up shared hosting space for ever-more WordPress websites. The situation is worsened by the fact that webhosts often don't advertise (Or even willingly tell you if asked) what the upper memory limit is for shared hosting.
Alas, often if a site isn't under heavy traffic and is still crashing, the issue likely has something more to do with Drupal's configuration than anything else. Pushing a user to VPS just muddies the waters and adds more variables to deal with (Is it the webserver config? PHP config? VPS guest config? VPS host config, even?).
6. When all else fails, clone to localhost and beat it with a stick
This is a big reason why people use the dev-staging-production methodology with version control -- when all else fails, you can do a DB dump, git clone the site to a local testing server and then royally mess up the site on the testing server without worrying about actually breaking anything on the production server.
For memory issues, this generally means disabling modules one by one until the one causing the issue is exposed. It can also point to webhost-related issues -- if the site runs perfectly fine on a local install with memory set to something reasonable like 128MB, then you know your webhost is on crack.
My gut is that it's chain_menu_access that's causing the issue. Try disabling that and clearing the cache, see if it works.
I'll also add to this answer as I think of other things to try...