I did the upgrade to Joomla 2.5 without updating the ext manager. I seem to recall that I was unable to uninstall it last time I tried, so I think its been quite a while since I've used it. I've been delaying my renewal until I find an answer to this question. Unfortunately with so much of this stuff buried into the database its getting harder to understand what to do.
When I click to go to it, I get an error page that says
404 - An error has occurred.
Component not found
Return to Control Panel
I see that many people had the same issues during the Joomla upgrades for 1.6 and 1.7, so I guess it is just time to start a new discussion.