crxop at tiscali.it
Tue Nov 9 06:54:23 EST 2004
Hello. I just want something to make /robots.txt available with get method. I already make a directory robots.txt and /.htaccess : #this is in the main dir SetHandler mod_python PythonHandler mod_python.publisher /robots.txt/index.py def index(req): req.content_type="text/plain" return """ User-agent: * Disallow: """ This works for client but I think a spider will get a 404 (telnet does). Also this make great confution and apache replace the index function for / too, so request "GET /" return the same as "GET /robots.txt/", I guess this is a know problem. I've tried "RemoveHandler .txt" but it don't works (with SetHandler I suppose). So, there is a method for making robots.txt available? Thank you, Daniele.