Advanced search
Log In | New Account   
Home My Page Projects Code Snippets gURLChecker
Summary Activity Forums Tracker Lists Docs News SCM Files

Feature Requests: Browse | Download .csv

[#512] Should use robots.txt

Date:
2005-08-15 23:44
Priority:
1
State:
Open
Submitted by:
Nobody
Assigned to:
Nobody (None)
Summary:
Should use robots.txt

Detailed description
While first-testing gURLChecker on a website, it appeared that the root robots.txt had not been taken in count. I think
that adding support for the robots file could be a good idea.

(If you want a verbose description, please read my initial mail about that:
http://lists.labs.libre-entreprise.org/pipermail/gurlchecker-devel/2005-August/000075.html)

-- 
Cyril Brulebois

Followup

No Followups Have Been Posted

Attached Files:

No Files Currently Attached

Changes:

Field Old Value Date By
priority32009-02-06 22:44esaracco

Powered By FusionForge