×
  • remind me tomorrow
  • remind me next week
  • never remind me
Subscribe to the ANN Newsletter • Wake up every Sunday to a curated list of ANN's most interesting posts of the week. read more

Robot guidelines

The web is a place for robots and humans alike. We crawl other sites for our own purposes and we don't mind being crawled for your own purposes, but we endeavour to and expect a certain level of civility when crawling someone else's website.

  • Do not hammer (making a large number of requests within a short time). Try to spread the requests as much as possible.

  • When spidering (fetching urls found in previously crawled pages), respect the rules in robots.txt

  • Do no try to fetch a ton of random urls just because you suspect they might exist. For example, don't fetch all urls from anime.php?id=1 to anime.php?id=100000 because most of those are 404s and thus waste resources
    • bookmark/share with: short url

    this article has been modified since it was originally posted; see change history