[WAPKA TUT] HOW TO EDIT YOUR WAPKA ROBOT.TXT FILE
WHY?
THE REASON IS FROM DEFAULT WAPKA ROBOT.TXT
EVERY CRAWLER/SPIDER, BEFORE IT CRAWS A SITE IT MUST FIRST OF ALL GOTO TO THE ROBOT.TXT AND CHECK THE AREA IT MUST AND MUST NOT CRAWL
NOW DEFAULT WAPKA ROBOT.TXT GOES LIKE THIS
1 |
User-agent: Slurp <br /> Disallow: / <br /> User-agent: * <br /> Disallow: <br /> Crawl-delay: 60 <br /> |
NOW
user agent means the name of the crawler….slurp is yahoo, googlebot is google and so on
now user agent: * means all the spiders but user agent: slurp or user agent: googlebot specifies the particular spider you are referring to
and
Disallow: /
means that the crawler should not touch or craw any of your site page
Disallow:
means the spider is free to access all your page
NOW WITH THAT
User Agent: Slurp
Disallow: /
MEANS THAT YAHOO BOT SHOULD NOT TOUCH YOUR SITE, AND THAT WHY WAPKA SITES DON’T APPEAR IN YAHOO SEARCH
THAT IS ALL, IF YOU WANT YOUR SITE TO BE CRAWLED AND APPEAR IN ALL SEARCH ENGINE
PUT THIS IN YOUR ROBOT.TXT FILE VIA EDIT SITE=>GLOBAL SETTINGS=>HEAD TAGS=>ROBOT.TXT
1 |
User Agent: * <br /> Disallow: <br /> |