Quantcast
Channel: phpBB.com
Viewing all articles
Browse latest Browse all 2255

[3.3.x] Support Forum • Re: Crawler requesting 40,000 pages when only 49 topics

$
0
0
I noticed from my logs that while I see a variety of IP addresses, each IP is making multiple requests within a short time frame.

So if IP can can be checked and limited to a number a page requests in say 10-60 seconds, and if they are not whitelisted as being part of Google's network, etc, then drop the request.

But again this will require support for listing netblocks instead of IP ranges.

Statistics: Posted by KenDouglas — Sat Jul 26, 2025 5:22 am



Viewing all articles
Browse latest Browse all 2255

Trending Articles