rubygem-robotex - Ruby library to obey robots.txt
| Website: | http://www.github.com/chriskite/robotex |
|---|---|
| License: | MIT |
| Vendor: | Alcance Libre, Inc. |
- Description:
With one line of code, Robotex (pronounced like “robotics”) will download and parse the robots.txt file and let you know if your program is allowed to visit a given link.
Packages
| rubygem-robotex-1.0.0-13.fc14.al.noarch [7 KiB] |
Changelog
by Miroslav Suchý (2013-03-18):
- use correct path (msuchy@redhat.com) |