robots.txt

[Jerry McBride]

On Sun, 09 Feb 2003 10:16:40 -0800 (PST) Kevin O'Gorman <kevin at kosmanor.com>
wrote:

> On Sun, 9 Feb 2003, Ken Moffat wrote:
> 
> > Anyone know the function of robots.txt? I have seen attempted access to 
> > it in my apache logs.
> > 
> 
> Yes.  That is where you can limit the browsing of well-behaved robots.

It's also used to control other things besides web-crawlers... for instance wget
honors it.



-- 

******************************************************************************
                     Registered Linux User Number 185956
          http://groups.google.com/groups?hl=en&safe=off&group=linux
             Join me in chat at #linux-users on irc.freenode.net
     5:41pm  up 32 days, 23:15,  2 users,  load average: 0.15, 0.13, 0.09