• Discussion

  • Nginx Free Firewall modified UA filter saved me 1000 requests in 12

Open Nginx free firewall->Global->UA filter
and

edit original rules and replace original rules with:
(HTTrack|Apache-HttpClient|harvest|dirbuster|pangolin|nmap|sqln|hydra|Parser|libwww|BBBike|sqlmap|w3af|owasp|Nikto|fimap|havij|zmeu|BabyKrokodil|netsparker|httperf|BLEXBot|DotBot|Rogerbot|barkrowler|SemrushBot|MJ12Bot|Nutch|MBCrawler| SF/)
Some explanations:
+Nginx free firewall->Global->UA filter: edit rule and add “BLEXBot”
+Nginx free firewall->Global->UA filter: edit rule and add “DotBot” ->is moz.com bot
+Nginx free firewall->Global->UA filter: edit rule and add “Rogerbot” -> https://moz.com/help/moz-procedures/crawlers/rogerbot#crawl-delay-to-slow-down-rogerbot
+Nginx free firewall->Global->UA filter: edit rule and add “barkrowler” via https://www.babbar.tech/crawler
+Nginx free firewall->Global->UA filter: edit rule and add “SemrushBot”
+Nginx free firewall->Global->UA filter: edit rule and add “MJ12Bot”
+Nginx free firewall->Global->UA filter: edit rule and add “Nutch” -> http://nutch.apache.org/bot.html
+Nginx free firewall->Global->UA filter: edit rule and add “MBCrawler” ->

-Nginx free firewall->Global->UA filter: edit rule and remove “audit” -> I user ahrefs service and this rules blocked ahrefs bot

Reference - original rules:
(HTTrack|Apache-HttpClient|harvest|audit|dirbuster|pangolin|nmap|sqln|hydra|Parser|libwww|BBBike|sqlmap|w3af|owasp|Nikto|fimap|havij|zmeu|BabyKrokodil|netsparker|httperf| SF/)

Do you know other useless bots?

2 months later

Does the bot rule case sensitive or insensitive ?

For example, u block DotBot there, but in moz site, their user agent is dotbot
If it's case sensitive, ur rule doesn't work.

6 months later
a month later
Connect with us: 📨 Telegram 💬 Discord Email: support@aapanel.com