404 Errors – SEO Best Practice Snippets
Even with an infallible web development process and no errors or typos ever appearing in your own pages, third party mistakes and visitor errors are also able to cause browsers to request non-existing URLs.
It is a virtual necessity to create a custom 404 error (requested resource not found) page for any site. This custom 404 page can present your own worded apology for the error and handy suggestions for the visitor to attempt to locate the resource they wanted. This is especially important now that search engines are attempting to claw back some 404 errors into further searches, which may of course take the visitor to a competitor. The latest version of the popular Google Toolbar does precisely this if no custom 404 page with at least 512 bytes of data exists.
Incidentally, several search spiders will ‘invent’ their own URLs. This is sometimes an attempt to locate higher directories in a site, but it also seems that search engines like to test that getting a true 404 error is possible, so that they can be sure they are not crawling a potentially bottomless site, where any URL at all will return some valid page.