If the bot has not been added to the bot list it is certainly an issue but problems being encountered by most are from unnamed bots. They are just using a browsers user agent so there is not much you can do within phpBB to fix the issue. There was comment from phpBB's lead developer Marc in one ticket stating SID's are removed in 4.0 so this particular problem where you are creating an infinite amount of URL's to crawl will go away.
There is still going to be a lot of links as guests, specifically the posts links which could be as many as 15 links to same page assuming you have not changed the default posts per page.
Glad there is 301 but Realistically they should not be attempting to crawl the page again because there is canonical URL tag. Same thing with the archive.org bot on it's initial query.
There is still going to be a lot of links as guests, specifically the posts links which could be as many as 15 links to same page assuming you have not changed the default posts per page.
Meta and PetalBot were the top abusers but as soon as I added them to the list, they started getting 301's for any url they requested that had a sid:
Glad there is 301 but Realistically they should not be attempting to crawl the page again because there is canonical URL tag. Same thing with the archive.org bot on it's initial query.
Statistics: Posted by thecoalman — Tue Jun 17, 2025 5:19 am