Robot guidelines

The web is a place for robots and humans alike. We crawl other sites for our own purposes and we don't mind being crawled for your own purposes, but we endeavour to and expect a certain level of civility when crawling someone else's website.

  • Do not hammer (making a large number of requests within a short time). Try to spread the requests as much as possible.

  • When spidering (fetching urls found in previously crawled pages), respect the rules in robots.txt

  • Do no try to fetch a ton of random urls just because you suspect they might exist. For example, don't fetch all urls from anime.php?id=1 to anime.php?id=100000 because most of those are 404s and thus waste resources
    • bookmark/share with:

    this article has been modified since it was originally posted; see change history

    Misc Page homepage / archives