# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file # # To ban all spiders from the entire site uncomment the next two lines: User-agent: * Disallow: /agent_tools/ Disallow: /admin/ Disallow: /*==$ Disallow: /api/v1/leads Disallow: /*?replytocom Disallow: /*/sign_in?referrer Disallow: /*/facebook?*referrer Disallow: /*/google_oauth2?*referrer Disallow: /*/posts?*query= Disallow: /api/v1/listings* User-agent: AhrefsBot Disallow: / Sitemap: https://aws.hawaiilife.com/sitemap.xml.gz User-agent: SeekportBot Crawl-delay: 2