How we check

If you've found PageKiwi in your server logs, someone used one of PageKiwi's checks, such as Link Check, on a page of your site, or on a page that links to yours. Here's what it does.

What it fetches

  • Usually one page

    The page someone asked us to check, with its links and pictures. Facts Check also reads up to 10 pages that page links to on the same site. A whole-site Link Check reads up to 10 pages under the address given (200 when we test it ourselves). We never read pages beyond that site, or beyond the part of it we were given, though we test each link on them once.

  • Each link and picture once

    Every link on a checked page is fetched once to see whether it works, the way a visitor clicking it would. Picture Check fetches each picture once to measure it.

  • Site Watch, once a night

    If the site's owner (or someone with their email) signed up to Site Watch, we check its home page and links once a night, about 4 a.m. Eastern.

  • Site health and previews

    Site health fetches your home page, its http and www versions, and reads the security certificate. Google and share preview fetches the one page asked for and its share picture. Once each, when someone asks.

  • One "not found" test

    One request to a made-up address on the checked site (it starts /pagekiwi-check-), to learn what its "page not found" looks like.

How politely

Speed
At most one request a second to any one site.
Size
At most 10 links from one page for a free check (300 for our own tests). We read up to 2 MB of the page we were asked to check (3 MB for Site health's home page), up to 2 MB of each page Facts Check reads, the first 120 KB of each page Link Check follows a link to, and up to 30 MB of each picture (15 MB for a share picture). We stop reading there.
Per day
Your site can be checked at most 30 times a day, by everyone together.
robots.txt
Read before we fetch the page, and for every link on the same site. We follow rules for PageKiwi, or for * if there are none for us.
User agent
PageKiwi/0.1 (+https://pagekiwi.com/about/checker/)
From
162.0.217.226
What we keep
The report: the checked page's address and title, and the text and address of each link with a problem. Kept 90 days for whoever ran the check. We don't keep copies of your pages. Picture Check also keeps a small copy of each problem picture, and any smaller versions it makes, for 90 days.

To stop it

Add this to your site's robots.txt and we won't check pages on your site:

User-agent: PageKiwi
Disallow: /

One exception: if another site links to yours and someone checks that other site, we fetch your linked page once to see whether the link works, as anyone clicking it would.