Both my visor mirrors fell out around 100k miles. I used a thick, all purpose glue to stick them back in.
If you wanted to get creative, you could try velcro tape (can probably be found at wal-mart) between the mirror and the cover to hold the cover in place. How often do you use the mirrors? I've never used mine!