Hey mojo.et,
There was no way to do this. I removed all non-existing servers once a while ago but it seems there's a small list of servers which is down now and cannot be removed by you. That's why I removed them again (~120 server favorites have been removed) and have edited the FavoriteServers Page where you can now delete records on your own with a new link that'll work always - even when the server is down.
Hope it's fixed now
/JoNny