User-agent: * Disallow: / # [WK-2551]. User-agent: Scrapy/2.5.1 (+https://scrapy.org) Allow: /