How to recognise us
- User-Agent
Mozilla/5.0 (compatible; neuralindex-ingest/0.1; +https://neuralindex.ai/bot)- robots.txt token
neuralindex-ingest
Challenge pages
We stop at a challenge page. We do not try to get past it.
Letting us in
To let us in, allow requests whose User-Agent contains the token above in your firewall or bot protection, and leave the token allowed in robots.txt.
Allow us by User-Agent. Do not rely on an address list.
Contact
Questions about the crawler? Email [email protected].