Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, crawler, scrapers, ...) are allowed to access specific
resources on a domain.
| Version: |
0.7.15 |
| Depends: |
R (≥ 3.0.0) |
| Imports: |
stringr (≥ 1.0.0), httr (≥ 1.0.0), spiderbar (≥ 0.2.0), future.apply (≥ 1.0.0), magrittr, utils |
| Suggests: |
knitr, rmarkdown, dplyr, testthat, covr, curl |
| Published: |
2024-08-29 |
| DOI: |
10.32614/CRAN.package.robotstxt |
| Author: |
Pedro Baltazar [aut, cre],
Peter Meissner [aut],
Kun Ren [aut, cph] (Author and copyright holder of list_merge.R.),
Oliver Keys [ctb] (original release code review),
Rich Fitz John [ctb] (original release code review) |
| Maintainer: |
Pedro Baltazar <pedrobtz at gmail.com> |
| BugReports: |
https://github.com/ropensci/robotstxt/issues |
| License: |
MIT + file LICENSE |
| URL: |
https://docs.ropensci.org/robotstxt/,
https://github.com/ropensci/robotstxt |
| NeedsCompilation: |
no |
| Materials: |
NEWS |
| In views: |
WebTechnologies |
| CRAN checks: |
robotstxt results |