JobClayBot

JobClayBot is the crawler of JobClay, a platform for job boards. It reads job feeds, applicant tracking system job boards and careers pages that a job board's operator added on JobClay, and brings the jobs it finds onto that board.

How it reads

• Each source is read at most once an hour.
• It pauses between requests to one site: at least a second, or the Crawl-delay in the site's robots.txt, up to ten seconds.
• Careers pages are read only where the site's robots.txt allows the token JobClayBot.

How to recognise it

Every request carries this user agent:

Mozilla/5.0 (compatible; JobClayBot/1.0; +https://jobclay.com/bot/)

Keeping it off your careers pages

Add a group for JobClayBot to your site's robots.txt:

User-agent: JobClayBot
Disallow: /

Contact

For questions about JobClayBot, or a problem with how it reads your site, write to bot@jobclay.com.