|
Daniele Cruciani
crxop at tiscali.it
Tue Nov 9 06:54:23 EST 2004
Hello.
I just want something to make /robots.txt available with get method. I
already make a directory robots.txt and
/.htaccess : #this is in the main dir
SetHandler mod_python
PythonHandler mod_python.publisher
/robots.txt/index.py
def index(req):
req.content_type="text/plain"
return """
User-agent: *
Disallow:
"""
This works for client but I think a spider will get a 404 (telnet
does). Also this make great confution and apache replace the index
function for / too, so request "GET /" return the same as "GET
/robots.txt/", I guess this is a know problem.
I've tried "RemoveHandler .txt" but it don't works (with SetHandler I
suppose).
So, there is a method for making robots.txt available?
Thank you,
Daniele.
|