基于User-Agent头的HTTP流量重定向

基于User-Agent头的HTTP流量重定向

今天，我们将谈论一个100%真实的案例。在一次事件中我们发现了大量的流量导致网站不可访问。从报告中，我们开始检查网站流量，发现了这样的情况。

66.249.65.120 - - [23/Jan/2023:07:10:02 -0500] "GET /?0694lobc28030csi187359.html HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5414.101 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.120 - - [23/Jan/2023:07:11:22 -0500] "GET /?9453wyca61460akg1320522.html HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5414.101 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.118 - - [23/Jan/2023:07:11:22 -0500] "GET /robots.txt HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
114.119.142.37 - - [23/Jan/2023:07:11:54 -0500] "GET /?8655nqcp42237pkk88280.html HTTP/1.1" 500 7309 "https://mysamplesite.com/?8655nqcp40361pkk211402.html" "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)"
66.249.65.120 - - [23/Jan/2023:07:12:41 -0500] "GET /?3598dhdz10101zsq1935112.html HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5414.101 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.118 - - [23/Jan/2023:07:13:33 -0500] "GET /?5116oavv41120vcm970162.html HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.122 - - [23/Jan/2023:07:13:33 -0500] "GET /?7537fatm18056mgo189475.html HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5414.101 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.118 - - [23/Jan/2023:07:14:01 -0500] "GET /?9928nbbl-23612leq-1793-634.html HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5414.101 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.122 - - [23/Jan/2023:07:15:20 -0500] "GET /robots.txt HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.120 - - [23/Jan/2023:07:15:20 -0500] "GET /?8425etst44656tek508701.html HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5414.101 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.118 - - [23/Jan/2023:07:16:40 -0500] "GET /?2288eqbi3960iqq1791964.html HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5414.101 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
114.119.138.65 - - [23/Jan/2023:07:17:41 -0500] "GET /?8655nqcp42236pkk87279.html HTTP/1.1" 500 7309 "https://mysamplesite.com/?8655nqcp39690pkk1539730.html" "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)"
66.249.65.120 - - [23/Jan/2023:07:17:59 -0500] "GET /?2288eqbi3799iqq1630803.html HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5414.101 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.120 - - [23/Jan/2023:07:19:19 -0500] "GET /robots.txt HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.118 - - [23/Jan/2023:07:19:19 -0500] "GET /?8270bwnl15589loa1426605.html HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5414.101 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.120 - - [23/Jan/2023:07:19:33 -0500] "GET /?8270bwnl13152loa988166.html HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.118 - - [23/Jan/2023:07:19:34 -0500] "GET /?7537fatm5165mgo997171.html HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5414.101 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.122 - - [23/Jan/2023:07:20:38 -0500] "GET /?8270bwnl14918loa755933.html HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5414.101 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.118 - - [23/Jan/2023:07:21:58 -0500] "GET /?0898iiiy-31607ysq-1792-637.html HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5414.101 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.122 - - [23/Jan/2023:07:22:33 -0500] "GET /?8041kjhl-24025lic-207-48.html HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5414.101 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.118 - - [23/Jan/2023:07:22:33 -0500] "GET /robots.txt HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
114.119.158.196 - - [23/Jan/2023:07:23:00 -0500] "GET /?8655nqcp42202pkk53245.html HTTP/1.1" 500 7309 "https://mysamplesite.com/?8655nqcp39522pkk1371562.html" "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)"
66.249.65.120 - - [23/Jan/2023:07:23:17 -0500] "GET /?9166yewp-6999pmm-1172-5.html HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5414.101 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.118 - - [23/Jan/2023:07:24:37 -0500] "GET /?9453wyca28952akg796981.html HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5414.101 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
114.119.138.127 - - [23/Jan/2023:07:24:38 -0500] "GET /robots.txt HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (compatible;PetalBot;+https://webmaster.petalsearch.com/site/petalbot)"
66.249.65.120 - - [23/Jan/2023:07:25:33 -0500] "GET /?2288eqbi27556iqq1399584.html HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.122 - - [23/Jan/2023:07:25:33 -0500] "GET /?8731mhos8493sgc327502.html HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5414.101 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.120 - - [23/Jan/2023:07:25:56 -0500] "GET /robots.txt HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.118 - - [23/Jan/2023:07:25:56 -0500] "GET /?9453wyca28420akg264449.html HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5414.101 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.118 - - [23/Jan/2023:07:27:16 -0500] "GET /?8917ijnl21806lco1646828.html HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5414.101 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.120 - - [23/Jan/2023:07:28:35 -0500] "GET /?8316ktkj7656jcm1489664.html HTTP/1.1" 500 7309 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5414.101 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
114.119.159.197 - - [23/Jan/2023:07:28:40 -0500] "GET /?8655nqcp42195pkk46238.html HTTP/1.1" 500 7309 "https://mysamplesite.com/?8655nqcp37974pkk182213.html" "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)"

在这一点上，我们有几个选择来解决这个事件。

例如，通过防火墙，我们可以生成一个源IP的列表，并将某些范围添加到你的阻止列表中，这样做效率很低，而且不能从根本上解决问题，因为他们使用谷歌机器人作为请求的来源。

使用重定向规则以阻止或重定向流量。通过这种方式，在响应请求时传递的信息量减少了，传输消耗也因此减少了，我们可以把攻击者送到我们想象中的地方。

实施这些规则的步骤如下。

1. 分析异常request的特征。
2. 基于所使用的网络服务器类型的重定向规则设计。
3. 规则的实施。
4. 服务器跟踪流量监控。

你在上一节看到的日志样本，属于Apache httpd，对应于组合日志格式或CLF（组合日志格式）的网站访问日志，它包含以下字段。

1. IP来源。请求源IP

66.249.65.120

1. 日期和时间：这个字段显示......它是正确的，是请求的日期和时间。

[23/Jan/2023:07:10:02 -0500]

1. 日期和时间：这个字段显示......它是正确的，是请求的日期和时间。

"GET /?0694lobc28030csi187359.html HTTP/1.1"

1. HTTP响应代码。服务器的响应代码通常是200，如果成功，500，如果有一个错误，等等。
2. 服务器回复的字节大小。非常重要，大小越大，你的服务中的传输消耗就越大。
3. 用户代理"。这个字段也将是相关的，它向服务器报告执行请求的设备、操作系统、应用程序等的类型。请记住，这个字段可以被攻击者操纵（欺骗），但为了这个事件的目的，我们确认我们请求的源IP确实属于日志中这个字段所示的机器人。

"Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5414.101 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

从这些字段中的信息，我们将分析规则的类型以及请求的哪一部分我们可以用来过滤、阻止或重定向流量。

对我们来说，要重定向的机器人将是那些在流量样本中发现的机器人。

1. Googlebot
2. petalbot

Apache HTTP服务器

对于这些情况，Apache的配置相对容易，只需要激活mod_rewrite模块和访问被攻击网站的.htaccess文件即可。

而我们的配置将如下。

.htaccess文件

<IfModule mod_rewrite.c>
    RewriteEngine On
      RewriteCond %{HTTP_USER_AGENT} (googlebot) [NC,OR]
  RewriteCond %{HTTP_USER_AGENT} (petalbot) [NC]
  RewriteRule .* "http://0.0.0.0/" [R=301,L]
</IfModule>

将这些行添加到所请求的资源或文件所在的目录中的.htaccess文件的开头。如果没有.htaccess文件，你可以创建它。

我们的规则有一行用于激活路由覆盖模块，还有尽可能多的 "RewriteCond "行，用于从 "User-Agent "中检测要重定向的机器人的名称，从头文件"%{HTTP_USER_AGENT}"中捕获它。(一段话中有太多复杂的术语，对吗？，你最后会相信我是个专家)

"RewriteCond "语句用于定义 "RewriteRule "将被应用的条件，也就是说，如果满足其中一个条件，将执行重定向。"[NC]"表示对术语或文本的搜索是不分大小写的，",OR]"表示至少要满足其中一个条件才能重定向执行。

最后一条规则 "RewriteRule "定义了要采取的行动，而我们要采取的行动是。

1. 要永久重定向"[R=301"
2. 停止执行后续规则",L]"
3. 来自任何来源的".*"
4. 对一个我非常喜欢的地址 "http://0.0.0.0/"，这在网络方面意味着不可路由的。

文件中的这一变化是即时的，所以你可以再次检查访问日志，你会发现请求变成了这样的内容。

66.249.65.54 - - [26/Jan/2023:07:12:19 -0500] "GET /?3529sryf20359fes199380.html HTTP/1.1" 301 251 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.58 - - [26/Jan/2023:07:15:12 -0500] "GET /?9166yewp17868pmm1706886.html HTTP/1.1" 301 252 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.56 - - [26/Jan/2023:07:15:58 -0500] "GET /?5408qvgw19728waq1567748.html HTTP/1.1" 301 252 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5414.101 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.58 - - [26/Jan/2023:07:19:14 -0500] "GET /?7758aoex11194xkq1029206.html HTTP/1.1" 301 252 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.56 - - [26/Jan/2023:07:21:17 -0500] "GET /?7537fatm6174mgo7181.html HTTP/1.1" 301 248 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.58 - - [26/Jan/2023:07:24:59 -0500] "GET /?4338szji18409igq248428.html HTTP/1.1" 301 251 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5414.101 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.58 - - [26/Jan/2023:07:26:09 -0500] "GET /?8425etst23249tek1090273.html HTTP/1.1" 301 252 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.56 - - [26/Jan/2023:07:28:15 -0500] "GET /?5408qvgw16062waq189979.html HTTP/1.1" 301 251 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.54 - - [26/Jan/2023:07:28:33 -0500] "GET /?7537fatm21027mgo86749.html HTTP/1.1" 301 250 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.58 - - [26/Jan/2023:07:33:05 -0500] "GET /?3529sryf4892fes724897.html HTTP/1.1" 301 250 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.56 - - [26/Jan/2023:07:46:55 -0500] "GET /?7537fatm16622mgo460639.html HTTP/1.1" 301 251 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.54 - - [26/Jan/2023:07:48:56 -0500] "GET /?1079ssfb2579bos410582.html HTTP/1.1" 301 250 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.56 - - [26/Jan/2023:07:53:51 -0500] "GET /?7537fatm10862mgo697873.html HTTP/1.1" 301 251 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.56 - - [26/Jan/2023:07:54:07 -0500] "GET /?5116oavv14746vcm583761.html HTTP/1.1" 301 251 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

文件中的这一变化是即时的，所以你可以再次检查访问日志，你会发现请求变成了这样的内容。

尽可能保持这些规则的活性，如果它们改变了机器人的类型，就相应地调整它们，然后打个盹。

Nginx

为了应用这些规则，我们将进入nginx.conf文件（在你的服务器上保存的路径），并在与被攻击网站的域名/子域名相关的服务器块中添加以下几行。

if ($http_user_agent ~* 'googlebot|petalbot') {
      return 301 http://0.0.0.0;
}

Nginx在定义规则方面更优雅一些。使用"~*"将在$http_user_agent中搜索与机器人名称相匹配的文本字符串，而不考虑大小写的敏感性，如果有匹配的，它将执行一个永久的301重定向到http://0.0.0.0。

这里有一个完整的Nginx的基本配置，这样你就可以参考应该在哪里添加重定向的行，你必须在受影响的域的服务器块中添加这些行，就在server_name下面。

events {
      worker_connections  4096;  ## Default: 1024
}

http {
      index index.html;
      server {
                server_name mysamplesite.com;
                
                #User-Agente rule here!
                          if ($http_user_agent ~* 'googlebot|petalbot') {
                                        return 301 http://0.0.0.0;
                          }
                          
                                  location / {
                                                root /var/www/mysamplesite.com/htdocs;
                                  }
                          }
         
            server {
                      listen 80 default_server;
                      server_name _; # This is just an invalid value which will never trigger on a real hostname.
                              server_name_in_redirect off;
                      location / {
                                    root /var/www/default/htdocs;
                      }
            }
      }