User-agent: * Disallow: /Customer/* Disallow: /DownloadFile/ Disallow: /DownloadDocument/ Disallow: /Portal/Register Disallow: /Portal/Login Disallow: /Account/Login Disallow: /Account/LogIn Disallow: /Account/Register Disallow: /PortalHub/ # Meta's crawler followed every sign-up and language variant of the portal # register/login pages, and each visit fired an uncached hub search. User-agent: meta-externalagent Disallow: /Customer/* Disallow: /DownloadFile/ Disallow: /DownloadDocument/ Disallow: /Portal/Register Disallow: /Portal/Login Disallow: /Account/Login Disallow: /Account/LogIn Disallow: /Account/Register Disallow: /PortalHub/ # A SharePoint search crawler on a shared state network crawls us twice a day at # about 850 requests a minute and slows public pages for everyone (AB#11584). # Crawl-delay is best effort. The enforced limit is the Front Door rate rule. User-agent: MS Search 6.0 Robot Crawl-delay: 2 Disallow: /Customer/* Disallow: /DownloadFile/ Disallow: /DownloadDocument/ Disallow: /Portal/Register Disallow: /Portal/Login Disallow: /Account/Login Disallow: /Account/LogIn Disallow: /Account/Register Disallow: /PortalHub/ Sitemap: https://publicinput.com/sitemap.xml