perl-WWW-RobotRules - Database of robots.txt-derived permissions
| Website: | https://metacpan.org/release/WWW-RobotRules |
|---|---|
| License: | GPL-1.0-or-later OR Artistic-1.0-Perl |
| Vendor: | Alcance Libre, Inc. |
- Description:
This module parses /robots.txt files as specified in "A Standard for Robot Exclusion", at <https://www.robotstxt.org/robotstxt.html>. Webmasters can use the /robots.txt file to forbid conforming robots from accessing parts of their web site.
Packages
| perl-WWW-RobotRules-6.02-19.aldos.noarch [17 KiB] |
Changelog
by Joel Barrios (2023-08-23):
- Clean spec file. |