Blocking User Agents, Bots, Crawlers, and Scrapers via .htaccess Print

Your website's .htaccess file can be used to prevent specific user agents from reaching your site. This is especially useful when dealing with an influx of bots, scrapers, or crawlers performing nefarious and unwanted activity towards your website.

Common examples of user agents as they would appear in your access logs:

HTTP/2" 200 14389 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"

HTTP/2" 302 0 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"

HTTP/2" 404 11644 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36"

HTTP/2" 302 25 "-" "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)"

To block one specific user agent, add the following to your site's .htaccess file:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} UserAgentName [NC]
RewriteRule .* - [F,L]

In the above example, replace UserAgentName with the actual user agent string identified from your access logs.

To block multiple user agents, add the following to your site's .htaccess file:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*(UserAgent1|UserAgent2|UserAgent3).*$ [NC]
RewriteRule .* - [F,L]

In the above example, replace UserAgent1|UserAgent2|UserAgent3 with the actual user agent string identified from your access logs.

Known User Agent names:

User Agent Name: Googlebot
String: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

User Agent Name: bingbot
String: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

User Agent Name: Slurp
String: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

User Agent Name: Baiduspider
String: Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)  

User Agent Name: YandexBot
String: Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)

User Agent Name: AhrefsBot
String: Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)

User Agent Name: SemrushBot
String: Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)

User Agent Name: MJ12Bot
String: MJ12bot/v1.4.0 (http://www.majestic12.co.uk/bot.php?+)  

User Agent Name: facebookexternalhit
String: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

User Agent Name: Twitterbot
String: Twitterbot/1.0 Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) QtWebEngine/5.12.3 Chrome/69.0.3497.128 Safari/537.36

User Agent Name: LinkedInBot
String: LinkedInBot/1.0 (compatible; Mozilla/5.0; +http://www.linkedin.com)

User Agent Name: Slackbot
String: Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)

User Agent Name: GPTBot
String: GPTBot/1.0 (+https://openai.com/gptbot)

User Agent Name: ChatGPT-User
String: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

User Agent Name: OAI-SearchBot
String: Mozilla/5.0 (compatible; OAI-SearchBot +https://www.openarchives.org/Register/BrowseSites)

User Agent Name: ClaudeBot
String: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected])

User Agent Name: anthropic-ai
String: Mozilla/5.0 (compatible; anthropic-ai/1.0; +http://www.anthropic.com/bot.html)

User Agent Name: PerplexityBot
String: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://docs.perplexity.ai/docs/perplexity-bot)

User Agent Name: Sogou web spider
String: Sogou web spider/4.0 (+http://www.sogou.com/docs/help/webmasters.htm#07)