Set up multiple page views displays with the same path.
You can clone your page views display to accomplish this.
Keep their path settings the same.
In the regular user one, remove the Edit field. Be careful that you are only editing "This page (override)" and NOT "All displays."
In the admin views display, under Page Settings > Access, restrict your admin view to role=admin.
Re-order the views displays so the most restrictive (admin) views display is first.
To reorder the displays, look for the item "edit view name/description" in the upper-right corner of the screen and expand it to select "reorder displays" - this will take you to a screen where you can drag and drop the views displays. After you put yours in the right order, click "Apply" then save your view.
Hope this helps.