Blocking User Agents, Bots, Crawlers, and Scrapers via .htaccess Print

Your website's .htaccess file can be used to prevent specific user agents from reaching your site. This is especially useful when dealing with an influx of bots, scrapers, or crawlers performing nefarious and unwanted activity towards your website.

Common examples of user agents as they would appear in your access logs:

HTTP/2" 200 14389 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"

HTTP/2" 302 0 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"

HTTP/2" 404 11644 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36"

HTTP/2" 302 25 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"

To block one specific user agent, add the following to your site's .htaccess file:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} UserAgentName [NC]
RewriteRule .* - [F,L]

In the above example, replace UserAgentName with the actual user agent string identified from your access logs.

To block multiple user agents, add the following to your site's .htaccess file:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*(UserAgent1|UserAgent2|UserAgent3).*$ [NC]
RewriteRule .* - [F,L]

In the above example, replace UserAgent1|UserAgent2|UserAgent3 with the actual user agent string identified from your access logs.