Let’s get it started.
Nginx
Nginx power millions of sites and very popular among web hosting. If you are using Nginx, then here is how you can stop them. Let’s say you are getting lots of automated requests with the following user-agent and you have decided to block them.
- java
- curl
- python
if ($http_user_agent ~* "java|curl|python") {
return 403;
}
If you would you like those to redirect somewhere, then:
if ($http_user_agent ~* "java|curl|python") {
return 301 https://yoursite.com;
}
The above configuration must be under the server block.
And the following to block by referrers. The following example which should go under the location block for blocking requests from semalt.com, badsite.net, example.com.
if ($http_referer ~ "semalt\.com|badsite\.net|example\.com") {
return 403;
}
After making necessary changes, you need to save the file and restart Nginx to take effects.
To restart Nginx, you can use:
service nginx restart
Nginx is a powerful web server and if you are interested in learning, then check out this online course.
Apache HTTP
To block user-agent in Apache, you can use the mod_rewrite module. Ensure the module is enabled and then add the following in either .htaccess file or respective .conf file.
If you are having multiple sites configured and want to block for a specific URL, then you may want to put them in respective VirtualHost section.
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} badcrawler [NC,OR]
RewriteCond %{HTTP_USER_AGENT} badbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} badspider [NC]
RewriteRule . - [R=403,L]
The above rule will block any request containing user-agent as badcrawler, badbot, and badspider.
And, the below example to block by the referrer name BlowFish, CatchBot, BecomeBot.
RewriteEngine on
RewriteCond %{HTTP_REFERER} blowfish|CatchBot|BecomeBot [NC]
RewriteRule . - [R=403,L]
As usual, restart the Apache server and to test the results.
WordPress
If you are using WordPress on shared hosting or don’t have access to web server configuration or not comfortable in modifying the file, then you can use the WP plugin. There are many WP security plugins, and one of the popular one for blocking bad bots are Blackhole for Bad Bots.
Want to Block Unwanted User-Agent & Referrers in Apache, Nginx and WordPress? No worries, our dedicated admins are available!!
