There are a hand full of common mistakes that pose roadblocks for spiders. A few Web design techniques actually make it nearly impossible for spiders to read and index pages. That’s not to say these techniques can’t be used, it’s just that they have to be thoughtfully used in ways that accommodate the spiders. Some things to avoid:
Frames
Most spiders are unable to navigate frames. Content within frames
is likely to be missed entirely by search engines, as they
can’t jump from frame to frame.
Forms and Drop-Down Menus
These pose a similar problem to spiders as frames. Spiders can’t
submit requests through a form or drop-down box, so the content
behind the form or drop-down won’t be indexed. One solution
is to create an alternate way for spiders to get through by offering
an alphabetical list of the content or a linked site map. By
offering both methods, you meet the needs of human visitors and
spiders.
JavaScript
JavaScript is programming language that can be embedded in Web
pages and read by browsers and add interactive or dynamic
functionality. When some spiders visit, they pretend to be
browsers with JavaScript disabled. Some sites redirect browsers
with JavaScript turned off to another page. There are ways
to effectively incorporate JavaScript in Websites to take
full advantage of its functionality, without discouraging
search engines.
Flash
Avoid welcome or splash home pages that offer little information
or that are composed entirely of Flash animation. Flash is
an animation software that makes it possible to add video
or cartoon-like graphics to Web pages. But Flash is difficult
or impossible for spiders to interpret. We’re not suggesting
throwing out Flash entirely, as it is popular and can be
used to great effect.
Bad HTML and Poor Spelling
Coding mistakes and spelling errors cause problems. Coding errors
create obstacles for a spider and a spelling error can keep
your content from showing up in search engine results. On
the other hand, attention to commonly miss-spelled words
and tactfully incorporating these words into a page can bring
incremental traffic.
Broken Links
Some search engines will penalize a page for having broken links,
which are links that point to pages that no longer exist
or have been moved. It’s another reason to stay on
top of your site, checking and updating to insure accuracy.