Imagine you’re at a party engaged in rigorous debate about the vice presidents of the United States when all of a sudden you are quizzed on who the fourteenth man to hold the office was.
“Hannibal Hamlin,” you respond.
The group of people by whom you’re surrounded collectively scoff at you.
“Incorrect,” one of them says. “Don’t believe me? Google it.”
You pull out your smartphone and type in the following question: “Who was the fourteenth vice president of the United States?” Quickly, you are greeted by the visage of John C. Breckinridge, followed by pertinent results from senate.gov and Wikipedia. Whether you’re aware, you’re seeing accurate results thanks to a virtual eight-legged arachnid that crawls across every accessible nook and cranny of the Internet, determined to deliver the best results to those surfing the Web.
That arachnid is more appropriately referred to as a search engine spider, and knowing how that technology works is integral to driving traffic to your website. One of the best ways to rack up unique visitors to your site is by getting it indexed on search engines, and understanding search engine spiders are essential to accomplishing that goal.
There are three major search engines: Google, Bing and Yahoo. With 67 percent of all Internet searches in America stemming from Google, the tech giant is clearly the search engine that decision makers focus on. (Bing and Yahoo generate 18 percent and 11.13 percent of searches, respectively.)
For starters, search engine spiders can only crawl through the Internet by following links from one site to another or from one page to another. If you publish a website that no one links to, it follows that spiders will not crawl your site. Taking this notion a step further, if you publish a website that tons of people link to, spiders will most likely find your site more frequently and the higher it will be ranked in search results. (That’s why the first results for Breckinridge come from senate.gov and Wikipedia. One could imagine how many websites about the man link back to those two sources.)
Additionally, it’s important that you make sure you don’t have anything on your website that serves as a sort of blockade that prevents spiders from crawling to your site. To do this, try using a text-based browser—like Lynx—that sees your website in a manner similar to Google’s spiders. This will help you determine if anything needs to be changed in order to grant easier access to the spiders and therefore rank your site higher.
“If fancy features such as JavaScript, cookies, session IDs, frames, DHTML or Flash keep you from seeing all of your site in a text browser, then search engine spiders may have trouble crawling to your site,” reads Google’s webmaster guidelines.
Curious as to how Google’s search engine spiders see your website? Check this out.
Increasing traffic to your website is not an impossible feat, so long as you’re familiar with the rules by which you must abide. Unfortunately—or fortunately, depending on whom you ask—those rules are always changing, which is why marketers need to make sure that they keep abreast of such fluctuations in order to direct more and more traffic to their websites.
