User-Agent: * Allow: / Crawl-delay: 10 User-agent: SentiBot Disallow: / User-agent: crawler4j Disallow: /