Open Nginx free firewall->Global->UA filter
and
edit original rules and replace original rules with:
(HTTrack|Apache-HttpClient|harvest|dirbuster|pangolin|nmap|sqln|hydra|Parser|libwww|BBBike|sqlmap|w3af|owasp|Nikto|fimap|havij|zmeu|BabyKrokodil|netsparker|httperf|BLEXBot|DotBot|Rogerbot|barkrowler|SemrushBot|MJ12Bot|Nutch|MBCrawler| SF/)
Some explanations:
+Nginx free firewall->Global->UA filter: edit rule and add “BLEXBot”
+Nginx free firewall->Global->UA filter: edit rule and add “DotBot” ->is moz.com bot
+Nginx free firewall->Global->UA filter: edit rule and add “Rogerbot” -> https://moz.com/help/moz-procedures/crawlers/rogerbot#crawl-delay-to-slow-down-rogerbot
+Nginx free firewall->Global->UA filter: edit rule and add “barkrowler” via https://www.babbar.tech/crawler
+Nginx free firewall->Global->UA filter: edit rule and add “SemrushBot”
+Nginx free firewall->Global->UA filter: edit rule and add “MJ12Bot”
+Nginx free firewall->Global->UA filter: edit rule and add “Nutch” -> http://nutch.apache.org/bot.html
+Nginx free firewall->Global->UA filter: edit rule and add “MBCrawler” ->
-Nginx free firewall->Global->UA filter: edit rule and remove “audit” -> I user ahrefs service and this rules blocked ahrefs bot
Reference - original rules:
(HTTrack|Apache-HttpClient|harvest|audit|dirbuster|pangolin|nmap|sqln|hydra|Parser|libwww|BBBike|sqlmap|w3af|owasp|Nikto|fimap|havij|zmeu|BabyKrokodil|netsparker|httperf| SF/)
Do you know other useless bots?
