Search engine spiders are by far one of the most useful things to come around in the last 10 years of the internet. They are useful not only to the web sites (Google and many others) that use them, but also to people who are searching for a particular site and those who run web sites. Spiders allow your site to be seen by the millions of people who use search engines every day. In this newsletter, we will discuss what search engine spiders do, how they work, and how to set up a robots.txt file and upload that to your site to keep spiders from visiting your site.

What are spiders and what purpose do they serve?

Spiders are essentially programs that “crawl” sites and report back to their superior (Google or whatever search engine they were created for) what their findings are. Their purpose is to make it easy for sites to get listed in search engines.

You might be wondering, what does it mean to “crawl” a site? Well it means to visit and site and copy the information.

How do spiders work?

Spiders work by finding links to web sites, visiting those web sites, going through the content of a web site and then reporting the content of the site back to the database of the site which they are working for. Google spiders, thus, crawl sites and report the information back to Google’s database. From there, the information is added to Google’s search engine, and the site then shows up in Google search results. Much the same process happens with any other search engine spider.

How can I keep spiders from visiting my site?

You might be thinking, “why would I want to keep such a useful thing from visiting my site?” Well, the short answer is, sometimes site owners don’t want the spider to crawl on a particular part of their site. Some site owners don’t want spiders to crawl their site at all. The reasons for not wanting a spider to crawl a site or a particular part of a site vary, although most of the time it is because the site is either completely spam or features a page or two of spam, or perhaps there are private member-only pages the site owner does not want to appear in search listings.

If you’re one of those site owners, then you’ll want to create and upload something called a robots.txt file. We will briefly go over how to do this.

A robots.txt file

The whole purpose of a robots.txt file is to tell a search engine spider not to crawl the site or part of the site on which the robots.txt file resides.

Creating the file

Creating a robots.txt file that blocks out spiders is easy. First, open up notepad. Then, copy and paste the following:

User-agent: *
Disallow: /

Once you’ve done that, save the file as “robots” and as a .txt file.

Uploading the file

Next, you will upload the file to the part of your site which you do not want the spider to visit. So, if you don’t want them to visit yoursite.com/news/, you’ll upload robots.txt to the news folder. If you don’t want the search engine spider to visit your site as well, upload robots.txt to your index folder. That’s all there is to it.

Using the robots.txt file to make sure search engine spiders DO visit your site

Believe it or not, the robots.txt file can be used to both disallow and allow search engine spiders to crawl your site. Here’s how to create and upload such a file.

Creating the file

Open up notepad and copy and paste in the following:

User-agent: *
Disallow:

You’ll notice that the only difference between this and the earlier example is that Disallow: is not followed with /. If it were, that would tell spiders to go away. Once again, save the file as robots.txt.

Uploading the file

All you’ll do is upload the robots.txt file to the part of your site that you want the robot to pay a visit to. So if you want the robot to see the whole site, just put the robots.txt file right alongside the index file. And you’re done.

Creating and uploading a robots.txt file to help make sure spiders don’t miss your site is fast and easy. So what are you waiting for? Create and upload that file now!

Is your blog collecting dust and not getting enough traffic? It’s a common problem for bloggers. Their blogs are good, but no one is coming to visit. They want to know how to increase traffic. Increasing traffic is actually easier than it seems–provided you know how. In this newsletter, we will go over 7 specific methods of increasing traffic. These are guaranteed methods to draw more visitors to your blog.

Tip #1: Sign up for pinging services

It seems pretty obvious, but most bloggers, even those who are more experienced, don’t sign up for pinging services. If you’re new to blogging and don’t know what pinging means, pinging basically means that when your blog is updated, a “ping” is sent out by the service to let pinging sites know that your blog has been updated. People who are visiting sites with pinging services will then see your blog flash across the side of the screen as “newly updated”. Pinging services include Yahoo’s blo.gs, weblogs.com, Google Blog search, and ZingFast.

Tip #2: Sign up for Technorati

If you go to technorati.com, you’ll see an option on the left side of the screen to “Claim Your Blog”. Doing so allows you to get updated link counts, add your photo or branding to Technorati, and see your Technorati link. Having your blog claimed on Technorati helps to increase traffic because the millions of people who go to that site will have a chance to see your blog on there.

Tip #3: Sign up for FeedBurner, a RSS service

What FeedBurner basically does for your blog is it helps you to promote your blog’s content, build your audience and measure your audience. In other words, it helps you increase your blog’s traffic.

Tip #4: Bookmark your blogs at places like del.icio.us

You can set up an account and profile at del.icio.us. Then, using that, you can add links to your blog along with excerpts from your blog. Anytime someone visits your profile on del.icio.us, they will see a link to your blog and excerpts. There are other social bookmarking sites that you can use in much the same way. The bottom line is, using these social bookmarking sites can help you increase traffic.

Tip #5: Post comments at other blogs

No, this doesn’t mean you should go and spam other blogs with comments that are only there to promote a site. It means that you should visit similar blogs to your own, post legitimate comments and then sign your name(along with a link to your blog). This way, people know you aren’t there just to promote your site—you’re there to post a good piece of feedback and also say where you’re from.

Tip #6: Ask owners of other blogs to post a link to yours

Find other blogs that are similar to yours. If they seem to be decently popular(that is, they have a good amount of comments for each post), you can try to contact the owner of the blog to add a link to your blog if they have a links section. To make them really want to do this, you can offer to link back to their blog on your blog. People do this reciprocal blog linking all the time and it does help to increase traffic.

Tip #7: Use keywords in your blog

Like regular web pages, having a good keyword density on a blog is a terrific way of getting search engine spiders to notice your page. (FYI: Search engine spiders are programs from search engines that browse sites and report back their findings to the search engine database, which then lists that site. The more keywords on your site, the more likely it is for a search engine spider to pay a visit)

If your average blog posting is around 200 words, try to use 2-6 keywords per blog post. 2-6 keywords in a blog posting of 200 words is a keyword density of between 1 and 3 percent, a respectable number. Remember that you don’t want to use too many keywords, as that will make your blog look bad. Also be sure that the keywords used are relevant to your blog. Before even making a blog posting, decide what the post will be about and then select a keyword or two to use in your post. Then do it. It’s a surefire way of getting more traffic to your site!

Signing up with a service like Technorati is a quick, painless way of increasing traffic to your blog. So sign up
today!