Crawl policy

How PeptigrityBot behaves.

If you are looking at your server logs and want to know who we are, this page is the answer. Our crawler identifies itself on every request as:

PeptigrityBot/1.0 (+https://peptigrity.com/crawl-policy)

What we collect

  • Product and variant names, SKUs and product URLs
  • Listed prices, currency, and whether an item shows as in stock
  • CAS numbers and synonym lists where a product page publishes them

That is all. We do not collect customer data, order data, or anything behind a login, and we do not submit forms or add items to carts.

What we never do

  • We do not create accounts on your store, and we store no credentials.
  • We do not bypass a login, a paywall, or a cookie gate. If prices require an account, we record your shop as gated and stop.
  • We do not ignore robots.txt. If it disallows our user agent from your product paths, we do not fetch them.
  • We do not run per-shop custom scrapers. We read three standard public formats or nothing.

How often, and how gently

  • Once per day, off-peak.
  • One request at a time per domain — never parallel connections to the same store.
  • A pause between page requests, and a longer pause between shops.
  • If your server returns a rate-limit response we slow down and retry later rather than pressing on.
  • After five consecutive failed visits we stop crawling your shop and raise an alert rather than continuing to hit a site that is not responding.

How your prices are shown

Prices appear on our price comparison pages with your shop named and linked. Links to your store carry no affiliate tracking — we earn nothing from them. No shop pays for placement or position, and being listed is not an endorsement. Full details are on the methodology page.

Removal, correction, or a feed

If you want your shop excluded, tell us and we will remove it. You do not need to give a reason, and we will not ask for one. You can also block PeptigrityBot in your robots.txt, which we will honour on the next visit either way.

If a price we show is wrong, tell us and we will fix it. If you would rather send us your pricing directly than have us read your storefront, we are building support for that — get in touch and we will set it up.

All three go to the same place: contact us.

All prices·Methodology