James John – Software Engineer

[WAPKA TUT] HOW TO EDIT YOUR WAPKA ROBOT.TXT FILE

PLEASE SORRY I HAVE NOW GONE TRU EVERY WAPKA SITE AND FOUND OUT THAT THEIR SITE CANNOT BE DISPLAYED IN YAHOO SEARCH

WHY?

THE REASON IS FROM DEFAULT WAPKA ROBOT.TXT
EVERY CRAWLER/SPIDER, BEFORE IT CRAWS A SITE IT MUST FIRST OF ALL GOTO TO THE ROBOT.TXT AND CHECK THE AREA IT MUST AND MUST NOT CRAWL

NOW DEFAULT WAPKA ROBOT.TXT GOES LIKE THIS

NOW
user agent means the name of the crawler….slurp is yahoo, googlebot is google and so on

now user agent: * means all the spiders but user agent: slurp or user agent: googlebot specifies the particular spider you are referring to
and
Disallow: /
means that the crawler should not touch or craw any of your site page

Disallow:
means the spider is free to access all your page

NOW WITH THAT
User Agent: Slurp
Disallow: /

MEANS THAT YAHOO BOT SHOULD NOT TOUCH YOUR SITE, AND THAT WHY WAPKA SITES DON’T APPEAR IN YAHOO SEARCH
THAT IS ALL, IF YOU WANT YOUR SITE TO BE CRAWLED AND APPEAR IN ALL SEARCH ENGINE

PUT THIS IN YOUR ROBOT.TXT FILE VIA EDIT SITE=>GLOBAL SETTINGS=>HEAD TAGS=>ROBOT.TXT 

James John

Software Engineer