The list of very uncommon spiders is as follows. It is recommended to block them to reduce traffic consumption.

ahrefsbot
barkrowler
dotbot
mj12bot
semrushbot
DataForSeoBot
oai-searchbot
petalbot
iaskspider
meta-externalagent
Amazonbot

【Description of Each Spider】

AhrefsBot:

A web crawler operated by Ahrefs, primarily used to provide data support for Ahrefs' SEO toolset. In other words, it offers no benefit to your website at all!

Barkrowler:

An SEO web crawler operated by Babbar, used to analyze website structure, content relevance, and authority metrics to support SEO and competitive analysis research.

Dotbot:

Primarily used to collect web data for Moz's link indexing service. It is a marketing-type crawler tool.

MJ12Bot:

A web crawler from Majestic, a well-known SEO company in the UK. It has little value in China.

SemrushBot:

A web crawler software sent by SEMrush, an online SEO analysis platform, primarily used to automatically discover and collect new and updated web page data on the internet for SEMrush's use.

DataForSeoBot:

A web crawler operated by DataForSEO, primarily used to collect public online data to support its SEO tools, keyword research, and market analysis services.

OAI-SearchBot:

A web crawler developed and operated by OpenAI specifically for search functionality. (Handle with caution as needed.)

PetalBot:

A search engine crawler independently developed by Huawei, primarily used to crawl and index web page content for Huawei's Petal Search. (Handle with caution as needed.)

iaskSpider:

A crawler program for the iAsk search engine from Sina. (Handle with caution as needed.)

Meta-ExternalAgent:

A web crawler operated by Facebook, primarily used to collect and index web page content to train its artificial intelligence models and build independent search capabilities. (Handle with caution as needed.)

Amazonbot:

A web crawler operated by Amazon, primarily used to collect public web page content to improve its products and services, and may be used to train Amazon's AI models. (Handle with caution as needed.)

Others：

The following are UA signatures of vulnerability scanning tools. Please add them to the blacklist as needed.

acunetix
antsword
appscan
audit
babykrokodil
bbbike
dirbuster
fimap
harvest
havij
httperf
httrack
hydra
libwww
nessus
netsparker
nikto
nmap
owasp
pangolin
parser
sf
sqlmap
sqln
w3af
wpscan
zmeu

How to Block?

Please go to "MagiAegis Defense System - WAF - Black and White Lists", copy the spider names above into the "UA Black and White List", as shown in the following image: