We are living in an age where robots and spiders are crawling all over your Web site. No, this isn’t a tag line from an old 1950 horror movie, this is the way things are. Don’t be frightened though. The fact that you have robots and spiders on your Web site is a good thing. A very good thing if you care about your own success Online. How can you make the most out of the robots and spiders? it all starts with a little file called “robots.txt”.

Before I get into what the robots.txt file is all about, there is something I have to cover. If you have been around the proverbial Web master block a few times, you have heard about search engine spiders. They are small “robots” that search engines send out across the Internet to look for content. Just about every major search engine uses them.

Now let us start with what it is. The robots.txt file is a small text file that sits in your root directory. When search engines send out spiders to roam the Internet looking for content to pick up, they read the robots.txt file first. Think of it as your way to talk directly to the search engines.

This is how your Web site ends up on a search engine, like Google. When you “submit” your Web site to a search engine you are putting your domain on a list of Web sites for them to spider over. Now which is best? Is it better for the search engine spider to find you by itself or with you submitting yourself ot the search engine? There is debate for both sides, so I will not get too deep into that.

So you know now that a search engine sends out spiders to pick up content on the Internet. You know you can talk to the spider with including something within your robots.txt file in your root directory. Now comes the fun stuff.

Now that you have a robots.txt file in your root directory, you can figure out what you want to tell the search engine spiders. This time, “Hey, how you doing?!” isn’t going to cut it. You have to learn how to speak their language. First there is the User-agent code. The User-agent code specifies the specific search engine you wish to speak to. Each search engine spider has a name. For example, Google’s search engine spider’s name is “googlebot”. Other search engines have other names.

Here is a good Web site to check out if you are curious about what names certain search engines are using.

Search Engine Dictionary – Spider Names

The Web Robots Database

To use the User-agent code to call for a specific search engine spiders to read, do this:

User-agent: googlebot

This tells Google’s spider that you want them to follow the rules you set in your robots.txt file.

To use the User-agent code to call for all search engine spiders to read, do this:

User-agent: *

This tells all search engine spiders that you want them to follow the rules you set in your robots.txt file.

Now instead of telling them, “It is okay for you to get content from here, here and here” it is much easier to tell the spiders where not to go. That is where the robots.txt file is most helpful. That is where the Disallow: command comes in handy. Using it, you can tell a search engine spider not to get anything inside your “photos” folder for an example. How does it look in the robots.txt file?

Lets make this command for all search engines spiders to stay out of my “photos” folder located in my root directory.

User-agent: *

That is it! Now I don’t have to worry about any search engine spiders looking inside my “photos” folder and indexing what is inside. The thing to remember is to keep your paths relative to where your root directory is. What does that mean?

If your domain name is (http://www.mitchkeeler.com) then in the above example, I just told the spider to stay out of my folder here (http://www.mitchkeeler.com/photos/). If my “photos” folder was inside my “images” folder (http://www.mitchkeeler.com/images/photos/, then the above example wouldn’t have worked.

It would have had to of been changed to this:

User-agent: *

Disallow: /images/photos/

For an example of what a robots.txt file looks like in action, lets take a trip to the White House! Here is the White House’s Web site’s robots.txt file:

http://www.whitehouse.gov/robots.txt

Now you are ready to get into your own favorite text editor and create a robots.txt file for yourself. If you still have any questions feel free to shoot me an E-mail or check out these handy links:

Search Engine World – Robots.txt Exclusion Standard Information

robotstxt.org

I’m bored, so when I am bored I do random Google searches. Nothing I am too proud off, because I know it honestly sounds as dull as watching paint dry. After going through my usual name searches though, I noticed that I am the number two and three entries for free mac mini in Google.

Not that this has helped me any at all. I still only have one person who has signed up and completed an offer. Brandon was right though, the easiest thing to do is to sign up for the free efax trial, no costs at all.

I was going to cancel it, but the customer support person I was talking to gave me ten dollars worth of more credit to stay around for the rest of the 30 days. How could I say no to that?

Speaking of which I got my free DVDs in the mail today for finishing up my original joining of the Columbia House DVD club for my first offer. That one I guess never went through, but the efax one did. Go figure. Still want that free Mac Mini? Then please click my link and sign up! I promise it is not evil!

Now we have all heard the debate over how evil ads are. Some say they are good, others say they are evil things that should be banished off of the Web. I was once in the camp who wanted to banish them, but over the years I have reconsidered how important ads are to Web sites.

Many of the greatest resources Online are supported by ads. They hope a few people click on them, so they can get some revenue to keep themselves up and running. What the public has to realize is that not every Web site is run by an unemployed geek who has an endless supply of money and types out everything he writes in his parent’s basement.

Web sites cost money to keep up. The more popular the Web site, the more money it costs. Every single time somebody goes to that Web site it takes away from the bandwidth. Bandwidth costs money. Every time a new article or new content in general gets posted, it takes away from disc space. Disc space costs money.

There is a reason why most free Web hosting places make you put up with ads on your Web sites. That is how they earn a return on offering you free Web hosting. Over the different Web sites I have worked for and worked with I always see a common complaint.

“Why are there so many ads around here! I’m going to another Web site!”

The reason for the ads, nine times out of ten, is because the Web site has become a success and now must cover it’s proverbial butt when it comes to paying for their bandwidth and disc space. Another thing to take into consideration is that this is how a lot of people Online earn money.

I know several people who do writing Online or other projects and their income comes from ad revenue. Maybe not all of it, but at least some of it that is left over, after paying for Web hosting, they are able to stick in their pocket. You might work a nine to five job in an office. Would you like it if I came to your house and took money from your pocket? I think not.

Now I am not out there to bastardize the society we live in today. I just want to make people think twice before they say that ads are annoying, and ads are evil. Not all ads are created equal.

I dislike pop-up ads. Ads shouldn’t get in the way of enjoying the content on a Web site. A few banner ads, horizonal or vertical though do not bug me at all. If clicking a few banner ads means I don’t have to pay ten or twenty bucks a month for a service, I am all for it. That brings up another topic though, what about paid content?

The idea of having paid for content hardly ever works. The reason is, people say they would love to look at things without any ads, but hardly ever pony up the cash to make it work. Even if they do, that means you now have to publish two versions of your content. One with ads, and one that doesn’t have ads. This brings up your bandwidth, disc space and the amount of time you waste double posting your product. Right now it just does not add up.

So next time you think about moaning about an ad or two on a Web site, think about the folks behind the HTML and what you can do to support them. If you like what you see, and you want to see it keep coming. Click an ad. Donate via PayPal. Support these people before they loose their shirts.

I have just posted my follow up article on my last series of articles reviewing the various control panels there are out there today. There are many control panels out there to choose from, so it is a good thing to keep in mind when you are shopping for Web hosting.

Over the past week or so I have received lots of feedback, questions, and suggestions about my three part series on control panels. Control panels can be confusing things. Knowing what to do with them and all the good things they have in them can take a while to learn and master. Most people seemed to have been able to get a lot of good information, but Paul had an interesting question.

This was a really fun series to do just because I got so much feedback and so many questions after it was all said and done. I will have to do another one of these types of articles soon. Read the rest of the article at Lockergnome!

Every so often I have more than a few thoughts dangling through my mind. So every so often, I figured instead of dedicating an entry to each I would just dump them here for you to sort out as you wish.

The Web hosting book is still going well, and is still being passed through a few pairs of hands before the public release. Really not much more I can tell you, so that is the reason why not much has been said about it the last few days.

I plan on having a rough draft or beta version of the Web hosting radio show (or podcast.. which ever you wish to call it for now) by next weekend. I just ordered the same mic Brandon has been using. It sounds great, and it is cheap! As for format of the show, I am not near ready to decide. The two ideas I have in my head right now are either doing an audio version of one of my standard articles or doing a “Hosting Headlines” type format.

I hope to get a few more articles pumped out over the next few days. I have been in a creative surge for the past week or so. Trying to take advantage of it before I loose my inspiration and get sucked back into playing Bejeweled and Astro Pop.

Been talking with Jonathan Wold of The Startup Blog the last couple of days through E-mail. He’s got some very interesting projects lined up and an interesting fellow to talk with. You’d be surprised by the number of interesting folks that lurk behind the walls of the Web hosting world.

My Web site layout has changed again, this time for the last time. I wanted something that looked a little more different than the blog layout. Tweaking away at CSS is a passion of mine for sure. Also, I have added a few new things to this blog. The biggest one is the “Ask Mitch!” at the top of the right hand column. I want to start getting more Web hosting questions from the public. Those seem to make for the best inspiration for articles and tutorials. If you know anybody who is looking for help, send them my way!

After hearing what is out there today with podcasts and radio shows, I’ve had an idea bouncing around in the back of my head. After listening to such things as The Chris Pirillo Show, Brandon’s Podcast, Eric Rice’s show and many more, I think I might be able to do this too.

I wonder though if there would be an audience for a Web hosting podcast? That is my main area of expertise, so if I was going to jump on the bandwagon, that would be the place. Step one, I’d have to go buy a mic. I haven’t recorded anything myself on here for many years. Not since my days of messing with folks with the mic chat was new in Yahoo! Chat.

So what do you think? Is there an audience for an audio version of what I do? Now that I am finished with the Web hosting book I am looking for my next big project.

There are many tools of the trade in the world of Web site development, but very few have as many good uses as your .htaccess file. What, you don’t have one? I’ll admit, information on these little darlings are confusing to some and hard to find. Hence the reason why we need an explanation of what it is and what it can do for you.

Inside your Web hosting space, you know you have folders, HTML documents, and other things that make your Web site work. Think of this file as just another addition to the many thing you have inside there. It usually sits inside your root directory, such as “/public_html/” and has more than a few good uses. You might already have a .htaccess file in your root directory. If so, all you need to do is edit that one. Just remember that if it is in your root directory, that it will effect all folders and documents inside that directory. If you want to over ride that, then you will have to make a new .htaccess file inside the directory in question.

For an example, if you want what is in the .htaccess file to effect everything but your “photos” folder then you would have to create a new .htaccess file inside your “photos” folder.

How can you create one? Well, depending on the FTP program you are using it should be as easy as creating a new document, then renaming it .htaccess. Note that it has nothing coming before the dot. If you have anything in front of the dot, then it will not be read by the Web hosting server as a .htaccess file. The server will not know what to do with it.

If you want to create the file on your computer first, you should be able to right-click your desktop and select “New” then “Text Document”. Then right-click that document on your desktop, and go to “Rename”. Type in the new name as “.htaccess”. Some programs might put the .txt extension on it automatically. All you have to do is go in to rename it again, and take that part off.

Another thing to remember is if you are creating the file on your own computer or uploading it from your own computer, make sure you are uploading it in ASCII mode, not binary. Most FTP programs do a pretty good job at auto-detecting how it should be uploaded, but if you run into a problem that is the first place I would check.

Also, .htaccess files can only be created on a Linux based server running Apache. If you are not sure if you fall into that category login to your control panel your Web host provided you and snoop around in there. If you are still not sure, feel free to ask your Web host. That is what they are there for! You may need to CHMOD the htaccess file to 644 so the sever can execute the commands inside it.

Now you know how to create an empty .htaccess file. Now it is time to start filling it up with some easy to learn and useful commands. Think of each line you put into the .htaccess file as a command for the Web hosting server to do something. Here is an example of what I mean. Think of this as the content of your .htaccess file:

• Take out the trash
• Clean the sink
• Mow the yard
• Rake the leaves

Each are short commands you might do during your own day. A .htaccess file is much like giving the Web hosting server a to do list. Most commands in the .htaccess file are suppose to be on one line only. This is just the way that the server reads the information.

Another warning you might take in consideration is the ability to use the .htaccess file at all. Some Web hosts have banned it or banned it for certain uses because it causes stress and strain on the Web hosting server itself. Make sure you read through your Web host’s frequently asked questions and acceptable use policy before moving forward. If they have something against it, then it should be in one of those two places.

Now that you have it, what can you do with it? That is another few articles all together. Some of the coolest things are password protection, error page redirects, and deny/allow IP address to access your Web site.

Well, after seeing a few folks sign up to get a free Mac Mini, how could I say no? Well as far as my own deal, it is so far so good. I’ve had a few people signing up but don’t know if they have gone through with the deal or not. They say not to start asking questions about your order till a person has been signed up for 15 days.

The reason being is that some of the deals take a little longer than others. I was speaking with somebody earlier today who said they signed up for the credit card, but you won’t get credit added for the referral it till you make your first purchase with the card in question.

So I am guessing I have more than a few folks like that signed up. One person’s referral went through very quickly though. I’d like to know which offer they picked. I’ve done some searching around and have found a few other folks talking about it as well.

It might have been the way Brandon describes in his blog:

I signed up for the free eFax trial to qualify, and it didn’t cost me one dime to do. In fact, I’ve already terminated my eFax account because I have no need for the service. Now all I have to do is refer ten people and my end of the deal will be complete. Sign up to get your own free Mac mini with this link, participate in one of the offers, and then refer ten of your own friends or contacts to do the same. It’s a cycle that can yield beautiful results.

I might have to try that out, since it is the cheapest way I have heard about getting around the problem. Matt has also written a great article on the Mac Mini deal:

All our lives, we hear that ‘nothing in life is free’. Well apparently, those folks have not heard of the wonders coming from the Gratis Network. What, is the name not ringing any bells for ya? I am sure that you have seen their work on TV shows like The Screen Savers. You remember, when Alex got his free iPod!

I’ll keep everybody updated on the process! If all goes well, I might try a few of the other, “Win a Free ____” offers or pimp out a few other folks’ referral codes on here.

So what are you waiting for, go sign up to get a free Mac Mini!

Over the past three days I have written up a trilogy of articles looking at all the difference Web hosting control panels there are out there today. I also stacked them up next to each other, one by one to see how they compare. Some are better than others, but each got their own score. Curious to see which one was better than the other? Here are a few teasers and links to the articles themselves.

Every Web host provides you with one, and if you do a lot of work on your Web site it could be something you curse or something you get to adore every single day. A control panel is like the brain behind your Web site’s body. It gives you one place to look at to do everything you would ever need to do to your Web hosting account. You can create E-mail accounts, create FTP accounts, track statistical information and so much more.

Read the rest of part one!

In yesterday’s article I took a look at cPanel, Plesk, and H-Sphere control panels. All of these got good scores, and I would recommend any of them for you to look for when shopping for Web hosting. Have we found the king of kings just yet? That has yet to be seen. Today we will look at Ensim, DirectAdmin, and Interworx. Can they beat yesterday’s selections?

Read the rest of part two!

Over the past two days I have looked at many different control panels. Some offered less, and some offered more. Which one will be the best? That has yet to be seen. By the end of this article though, you will at least know my Web hosting opinion on the matter. Today we will finalize this control panel round up with Hosting Accelerator, InsPanel, CWIPanel and my final thoughts on control panels in general.

Read the rest of part three!

Have anything else to add? Be sure to comment and let me know! Depending on how much feedback I get, I plan on doing one more article on this topic covering everybody’s thoughts, questions, arguments and suggestions.

I saw this on Kevin Rose’s Web site and throught I would give it a shot. You can now get a free Mac Mini, from the same folks who had the free iPod deal a few months back. It sounds harmless enough. If you are intrested, please use my refferal code:

http://www.FreeMiniMacs.com/?r=14056999

I signed up for the Columbia House DVD deal, because it seemed to be the cheapest one. I’ve been a Windows user all my life, so I thought it might be nice to see how the other side lives.

If I can get at least 10 folks to sign up through here and do the same, I will let you know how legit this deal is. Worth a shot right? I can’t be the only one who loves free stuff.

Update! 1 down 9 to go! Keep signing up! If this all does go through I’ll promise to give you pictures and a full review!

Monday Update! Monday morning and still one definate and two maybes. Once I get 10, I’ll be more than willing to post somebody else’s link here on this entry.