# spiderpic.com robots.txt file User-agent: * Disallow: /redirect Disallow: /api Disallow: /thumbnail Disallow: /captcha Disallow: /error Sitemap: http://www.spiderpic.com/sitemap-index.xml