分类：反爬虫策略

2020
10-10

Nginx反爬虫策略，防止UA抓取网站

新增反爬虫策略文件：vim/usr/www/server/nginx/conf/anti_spider.conf文件内容#禁止Scrapy等工具的抓取if($http_user_agent~*(Scrapy|Curl|HttpClient)){return403;}#禁止指定UA及UA为空的访问if($http_user_agent~"WinHttp|WebZIP|FetchURL|node-superagent|java/|FeedDemon|Jullo|JikeSpider|IndyLibrary|AlexaToolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|Java|Feedly|Apache-HttpAsyncClient|UniversalFeedPar... 继续阅读 >