Search engine spiders are by far one of the most useful things to come around in the last 10 years of the internet. They are useful not only to the web sites (Google and many others) that use them, but also to people who are searching for a particular site and those who run web sites. Spiders allow your site to be seen by the millions of people who use search engines every day. In this newsletter, we will discuss what search engine spiders do, how they work, and how to set up a robots.txt file and upload that to your site to keep spiders from visiting your site.

What are spiders and what purpose do they serve?

Spiders are essentially programs that “crawl” sites and report back to their superior (Google or whatever search engine they were created for) what their findings are. Their purpose is to make it easy for sites to get listed in search engines.

You might be wondering, what does it mean to “crawl” a site? Well it means to visit and site and copy the information.

How do spiders work?

Spiders work by finding links to web sites, visiting those web sites, going through the content of a web site and then reporting the content of the site back to the database of the site which they are working for. Google spiders, thus, crawl sites and report the information back to Google’s database. From there, the information is added to Google’s search engine, and the site then shows up in Google search results. Much the same process happens with any other search engine spider.

How can I keep spiders from visiting my site?

You might be thinking, “why would I want to keep such a useful thing from visiting my site?” Well, the short answer is, sometimes site owners don’t want the spider to crawl on a particular part of their site. Some site owners don’t want spiders to crawl their site at all. The reasons for not wanting a spider to crawl a site or a particular part of a site vary, although most of the time it is because the site is either completely spam or features a page or two of spam, or perhaps there are private member-only pages the site owner does not want to appear in search listings.

If you’re one of those site owners, then you’ll want to create and upload something called a robots.txt file. We will briefly go over how to do this.

A robots.txt file

The whole purpose of a robots.txt file is to tell a search engine spider not to crawl the site or part of the site on which the robots.txt file resides.

Creating the file

Creating a robots.txt file that blocks out spiders is easy. First, open up notepad. Then, copy and paste the following:

User-agent: *
Disallow: /

Once you’ve done that, save the file as “robots” and as a .txt file.

Uploading the file

Next, you will upload the file to the part of your site which you do not want the spider to visit. So, if you don’t want them to visit yoursite.com/news/, you’ll upload robots.txt to the news folder. If you don’t want the search engine spider to visit your site as well, upload robots.txt to your index folder. That’s all there is to it.

Using the robots.txt file to make sure search engine spiders DO visit your site

Believe it or not, the robots.txt file can be used to both disallow and allow search engine spiders to crawl your site. Here’s how to create and upload such a file.

Creating the file

Open up notepad and copy and paste in the following:

User-agent: *
Disallow:

You’ll notice that the only difference between this and the earlier example is that Disallow: is not followed with /. If it were, that would tell spiders to go away. Once again, save the file as robots.txt.

Uploading the file

All you’ll do is upload the robots.txt file to the part of your site that you want the robot to pay a visit to. So if you want the robot to see the whole site, just put the robots.txt file right alongside the index file. And you’re done.

Creating and uploading a robots.txt file to help make sure spiders don’t miss your site is fast and easy. So what are you waiting for? Create and upload that file now!

Getting listed quickly in search engines is something every web site owner wants to do. Lots of site owners don’t really know how to do this, though. Fortunately for them, there are several ways to getting listed quickly, all of which we will go over in this article. These ways include submitting the URL, using keywords, using links, and using blogs to get listed quick.

Tip #1: Submit your URL manually to search engines

Search engines like Google, Altavista and Yahoo all have options for you to manually submit your site. Before submitting, however, it is important to make sure of the following:

1.    All pages on your site are complete. Search engine spiders won’t touch incomplete sites.
2.    Your site is not full of spam and/or excessive use of keywords. Yet again, spiders won’t touch spam sites, which means your site will be skipped over if it contains lots of spam.

When you submit the site, make sure to submit it only once to each search engine. Submitting multiple times to the same search engine will get you no where. In fact, they might keep your site from ever having a chance at being listed on the search engine.

Tip #2: Use keywords to get spiders to come to your site

Search engine spiders absolutely love sites with good, well-placed and well-used keywords and key phrases. In other newsletters, we have covered the correct use of  keywords and key phrases, so you might already be familiar with it. If not, we’ll briefly touch on it right now.

Keywords are words that are used frequently on a site that search engines look for. The key to correct usage of keywords is to make them fit naturally throughout a piece of content and not look like spam. Before writing your content for your site, take a second to decide what the content is about. Then make a list of keywords or key phrases that seem to fit with the topic at hand. Next, write the content and try to sprinkle in the keywords or key phrases throughout the content in a natural, non-distracting way. When you are done writing the content, you should count the number of keywords or key phrases. Then divide that by the total number of words in content. If the number you get is between 1-3%, you’re good to go. If it’s below 1%, try to naturally add in keywords or key phrases. If it’s above 5%, try to take out keywords or key phrases. Aim for the perfect balance of between 1 and 3%.

Tip #3: Use links

Having another site link to you will make spiders that visit the first site come to your site as a result of the link. Many owners of sites related to yours are happy to post a link to your site on their links page. Some might require what is called “reciprocal linking”. This basically means that if they place a link to your site on their page, they want you to return the favor by posting a link to their page on your site. Doing so helps both sites better gain the attention of search engine spiders and helps them both to get listed quickly.

Tip #4: Install a blog on your site

Blogs are extremely popular and quickly listed by search engines. Places like Google are known to index (or list a site/blog) far faster than they list regular web sites. Blogs are listed by search engines like Google within days of being created, whereas it can take weeks and months for search engines to notice a site.

Installing a blog on your site is really easy. All you have to do is install a program such as WordPress on your web site and then use that to make blog postings. If you run a sports site, for instance, you could try making twice a week posts on the blog about sports news. Doing so will help your site as a whole get noticed quicker by search engines.

If you apply just one of the tips above to your web site, you will get listed more quickly than expected.

Creating a blog on your web site is quicker than ever before. Do it today and watch how quickly and effortlessly your site gets listed on search engines!

You’re a blogger and are constantly hearing about pinging. The thing is, you don’t know what it is and what it can do for your blog. Well that’s fine because a lot of bloggers don’t quite understand pinging. In this newsletter, we will explain what pinging is, how it helps your blog, and what the best pinging services are. After you’ve read this newsletter, you’ll know exactly how to use pinging to make your blog popular.

What is pinging?

When a blog is pinged, it essentially tells a web blog tracking system that the blog has been updated. Thus, pinging is the process of telling tracking services that a site has been updated.

So how does pinging work? Well, a ping service scans regularly blogs for updates. Whenever there is a new update for a particular blog, that is reported back to the pinging service’s site. You can almost think of ping services being just like search engine spiders—both scan sites looking for information, and both report that information back to their specific site. The difference is, a ping service only reports back new blog posts.

Why is pinging useful for my blog?

Pinging, in effect, tells people that there is a new blog posting available. Those who are loyal followers of your blog will see the ping and want to visit your blog. Those who haven’t read your blog before will see the ping and perhaps think “hey, there’s a new blog I haven’t been to before, I think I’ll check it out”. Pinging helps to increase your traffic because it tells people when your blog is updated. Those who visit sites with pinging services will see this information in the form of a scroll on the left or right side, or top of the page. The scroll is usually in XML format and typically includes the title and URL of your blog, as well as the date and time of the update.

OK, so now I know what pinging is. Who offers pinging?

The answer is, there are several pinging services available which you can use to have your blog pinged. We’ll go over some of them right now.

Weblogs.com

Weblogs.com is probably the most well-known pinging service. Weblogs.com monitors sites for updates and then pings whenever there is an update to report. Virtually every blog software is available to use Weblogs.

Yahoo’s Blo.gs

Another popular pinging service, this one is offered by Yahoo  and has a built-in integration with movable type and other tools. One cool thing about this service is that it offers the ability to give out updates via Instant Messenging services like AOL and ICQ. It is a direct pinging service, so those who sign up for subscriptions to your blog will be sent an email anytime your blog is updated.

Blogrolling.com

Blogrolling allows its bloggers to manage their blogrolls(updates) through a web-based interface. Recently updated blogs are also highlighted on Blogrolling’s web site. Data is automatically retrieved from other sites like blogger.com and weblogs.com every 5 minutes. Often times your blog is already being pinged by Blogrolling, so you might not have to sign up for it. Search on the site first for your URL before signing it up.

Technorati

Available at http://rpc.technorati.com/rpc/ping, Technorati is another service that pings blogs. It gets its data from a variety of places and is definitely a site you should go to for blog pinging.

These are just 4 of the many different blog pinging services available. A quick search on Yahoo or Google will reveal many other possible blog pinging services to use for your blog. It is best to sign up for as many as you possibly can to ensure that your blog is well-pinged. A well-pinged blog will definitely help your blog attract more visitors. So sign up for as many as possible and watch your blog traffic increase.

Pinging tells people when your blog is updated. It is an incredibly useful, and easy to sign up for service, so be sure
to sign your blog up for pinging!

If you run a site and are looking to get it listed quickly on search engines, there really is no easier and quicker way than to also put a blog on the site. It is a known fact that search engine spiders from sites like Google and Altavista list new blogs much faster than they list new sites. However, it’s not as simple as just creating a blog and letting it sit there. You have to also use tactics like pinging, linking and the use of keywords. We’ll go over each so that you are able to effectively use your blog for a quick site listing. We’ll also briefly go over how to host your own blog on your site.

So how do I host my own blog, anyway?

The easiest way to do this is to install and run a blogging software from another company on your web site’s server. One such software that is popular is WordPress. It’s fairly easy to install and works pretty good on your site. Other blogging software options include:

Apache Roller
Blosxom
Geeklog
Textpattern
LifeType

Whatever one you choose, be sure to follow the instructions provided by the company so that the installation goes smoothly.

Where should I put my blog on my site?

Most people who host their blog to help their site get listed quickly don’t put the blog on the main page of the site, because the site isn’t all about the blog. Instead they put it in its own folder at mysite.com/blog/. But you can really put your blog anywhere on your site, so it’s your call.

What is pinging?

It’s a question that many ask, so don’t feel silly if you don’t know what pinging is. Pinging is essentially a tracking program installed on your blog that will notify the site which it is from whenever your blog is updated. Search engine spiders usually visit newly pinged sites most often, which is why pinging is such a crucial part of the equation to get your site listed quickly. Sites that are newly “pinged” will be visited by search engine spiders. A visit from the spider equals a listing on the search engine.

How do I get pinging on my blog?

You’ll have to register with a site that offers weblog pinging. The most popular pinging program around is weblogs.com. Other programs offered include Yahoo’s Blo.gs, BlogRolling, and Technorati. Once you’ve decided on a pinging service, you’ll have to follow the instructions to install it on your blog. When you’ve done that, your blog will start getting pinged and visited by search engine spiders.

What is linking?

Linking is basically having another blog or site link to your blog. When search engine spiders visit that site and see a link to your site, they’ll visit your site. Linking is effective both for sites with blogs and those without it.

How do I get linking?

You get it by visiting sites/blogs that are relevant to what your site is about. Then you contact the owners of those sites/blogs, asking them to post a link to your site on their site. In return, you will probably have to link to their site on your site. That way, both of you are able to benefit from the linking.

Why is the use of keywords important to my blog?

Spiders love to visit sites with a good amount of keywords. People who read your site won’t mind the use of keywords if they are lightly spread throughout the content of your pages. The key to this is to make a list of keywords relevant to your blog posting and then to try to use those keywords throughout your content. It is important to make sure, however, not to overuse keywords. Spiders have been trained to ignore sites with excessive amounts of keywords, so if you use a whole lot of keywords in the hopes of getting spiders to visit your site, it’s not going to do you any good. Generally speaking, using between 2-6 keywords per 200 word piece of content is a good ratio. It’s pleasing to both readers and search engine spiders.

That’s really all there is to it. Apply the above techniques to your blog and you will net your site a quick search engine listing.

There is no quicker way to get your site listed than to use a blog. So install a blog on your site today and you’ll be listed sooner than you could have imagined!