Crawls web content that the Allen Institute for AI uses to train open language models.
Source: allenai.orgOperator
Allen Institute for AI
Job
Model training
Collects pages for training future models. Slower to pay off, and separate from search and citations.
robots.txt
Vendor does not say
Ai2's page does not mention robots.txt. It publishes the user agent so site owners can filter or reject its traffic.
Source: allenai.orgMozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)
Add one of these groups to the robots.txt at the root of your domain.
A crawler obeys only the most specific group that names it (RFC 9309), so a named group does not inherit the rules under User-agent: *. Repeat any paths you keep private, as the example does with /admin/.
User-agent: AI2Bot Allow: / Disallow: /admin/
User-agent: AI2Bot Disallow: /
Read from the robots.txt we serve. We want AI engines to read and cite this site, so every AI crawler is allowed everywhere except the app, the API and the sign-in pages (/dashboard/, /api/, /login, /signup, /sign-in, /sign-up). The file also declares Content-Signal: ai-train=yes, search=yes, ai-input=yes.
User-agent: AI2Bot Allow: / Allow: /api/markdown/ Disallow: /dashboard/ Disallow: /api/ Disallow: /login Disallow: /signup Disallow: /sign-in Disallow: /sign-up
See the whole file. The same rules repeat in every group, because a crawler that matches a named group ignores the wildcard one.
Sources, checked 2026-09-24