aaP_peter0 I am trying to index a sitemap.xml. When I go to it from the browser it shows with no issues, however when google goes to index it I get an error 403. Domain is running through cloud flare Any idea what could be blocking it?
aaP_peter0 idflorin Yes but that should not be stopping google. Where would it even be blocking google on the firewall ?
idflorin Maybe your rules for the UA filter are broken. these are my UA filter rules: (HTTrack|Apache-HttpClient|harvest|dirbuster|pangolin|nmap|sqln|hydra|Parser|libwww|BBBike|sqlmap|w3af|owasp|Nikto|fimap|havij|zmeu|BabyKrokodil|netsparker|httperf|BLEXBot|barkrowler|MJ12Bot|Nutch|MBCrawler|MegaIndex|GrapeshotCrawler|CriteoBot|admantx|newspaper|CF-UC|comscore|Zoominfobot|SeznamBot|DataForSeoBot|Seekport|Java|HonoluluBot|Go-http-client|python-requests|Bytedance|Bytespider) for Nginx free firewall 5.0