AOCrawler

AOCrawler is operated by AO (ao-stack.com). It visits public business websites to record which actions a customer can perform there (for example: search a catalog, check stock, get a price, request a quote) so that AI agents can find and use those sites. Site owners can review and control their listing.

Identity

User-Agent: Mozilla/5.0 (compatible; AOCrawler/0.1; +https://crawler.ao-stack.com/policy)

Requests the unattended browser makes to the site it is visiting are signed with Web Bot Auth (HTTP Message Signatures, RFC 9421). Requests the page itself makes to other domains (third-party resources, which are never stored) pass through unsigned. The public key is published at /.well-known/http-message-signatures-directory. AOCrawler never hides its identity, never evades fingerprinting, and never solves CAPTCHAs or challenges automatically.

What it does

• Honours robots.txt (token AOCrawler, and the default rules).
• Reads a site's terms of service first. Where the terms prohibit automated access, AOCrawler limits itself to public files (robots.txt, sitemaps, .well-known) and does not browse.
• To tell a real "page not found" from a site's custom error page, it requests one random path that does not exist (/ao-nonce-<random>) once per site per visit. This request reads nothing and changes nothing.
• Requests at most one page per second per host and backs off on 403, 429, 503 or any challenge page; it does not retry around blocks.
• Reads and searches only. It does not create accounts, submit orders, make payments or create records. Any step beyond a read is only performed at the request of the site owner.
• Records first-party requests only, with authentication headers, cookies and tokens removed before storage. Third-party requests are not stored.

Opting out

• Add to robots.txt: User-agent: AOCrawler / Disallow: /. This is honoured on the next cycle and any existing listing is withdrawn.
• Or email crawler@ao-stack.com from the site's domain. Requests are actioned within 7 days; stored crawl data is deleted on request.

Contact

crawler@ao-stack.com