Check the rules for the specific crawler and path, remembering that each subdomain has its own file.
All guides
Русский