There are bad crawlers, which crawl each page of website at very high speed which results in very high web server load.

You should probably ban all those SE IP's because they don't honor crawl delay in any fashion and those SE are useless anyways. You can probably find all those SE IP's from apache logs.