First of all, than you for the quick action on this. It's a great to see.
Yeah, I totally agree that a huge part of the issue is the bots. I've already taken some action there. Specifically, I've set up the "Ultimate AI Block List" from
https://perishablepress.com/ultimate-ai-block-list/.
As far as who the culprit is, well... Here's the count of hits from each user agent in this month's access log, just showing the top 10.
781080 meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
45746 meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
4115 Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)
1542 Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.7680.177 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
786 Mozilla/5.0 (compatible; SERankingBacklinksBot/1.0; +https://seranking.com/backlinks-crawler)
690 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36
642 Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)
327 Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler)
180 Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.7680.177 Mobile Safari/537.36 (compatible; GoogleOther)
129 Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.7680.164 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Why is Facebook hammering my site so hard? Their bot seems to be getting confused by the tags. Maybe searching every combination of tags in every possible order? I'm not sure what it thinks it's doing.
2a03:2880:f814:3a:: - - [01/Apr/2026:00:02:19 -0500] "GET /trivia/gallery/search.php?tag_id=9,17,18,22,30,36,37,40,42,43,44,52,55,56,65 HTTP/1.1" 302 3999 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
2a03:2880:f814:31:: - - [01/Apr/2026:00:02:20 -0500] "GET /trivia/gallery/search.php?tag_id=3,5,9,12,17,22,30,36,37,40,42,43,44,46,57,58,63,65,69 HTTP/1.1" 302 3999 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
2a03:2880:f814:1b:: - - [01/Apr/2026:00:04:42 -0500] "GET /trivia/gallery/search.php?tag_id=3,5,12,22,30,36,40,42,43,44,53,55,56,57,58,60,67 HTTP/1.1" 302 3999 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
2a03:2880:f814:32:: - - [01/Apr/2026:00:04:43 -0500] "GET /trivia/gallery/search.php?tag_id=3,8,9,17,22,30,37,40,41,42,43,44,53,57,61,63 HTTP/1.1" 302 3999 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
2a03:2880:f814:18:: - - [01/Apr/2026:00:04:43 -0500] "GET /trivia/gallery/search.php?tag_id=3,9,12,20,30,31,36,37,40,42,43,55,61,66,69 HTTP/1.1" 302 3999 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
2a03:2880:f814:2b:: - - [01/Apr/2026:00:06:04 -0500] "GET /trivia/gallery/search.php?tag_id=3,9,23,29,30,36,37,40,42,43,44,46,61,63,64,66,67 HTTP/1.1" 302 3999 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
2a03:2880:f814:c:: - - [01/Apr/2026:00:07:14 -0500] "GET /trivia/gallery/index.php?/search/psk-20260103-gv2O1X6gqy&display=medium HTTP/1.1" 200 56772 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
2a03:2880:f814:33:: - - [01/Apr/2026:00:08:20 -0500] "GET /trivia/gallery/search.php?tag_id=9,17,22,23,30,36,37,42,43,44,55,56,57,64,66 HTTP/1.1" 302 3999 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
2a03:2880:f814:23:: - - [01/Apr/2026:00:11:31 -0500] "GET /trivia/gallery/index.php?/search/psk-20250909-ZhJA61KgyH&display=medium HTTP/1.1" 200 52748 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
2a03:2880:f814:d:: - - [01/Apr/2026:00:11:50 -0500] "GET /trivia/gallery/search.php?tag_id=3,12,22,33,36,37,42,43,44,46,57,58,63,65,68 HTTP/1.1" 302 3999 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
2a03:2880:f814:2c:: - - [01/Apr/2026:00:12:07 -0500] "GET /trivia/gallery/index.php?/search/psk-20260219-zKERMG2aHl&image_order=9 HTTP/1.1" 302 3999 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
2a03:2880:f814:8:: - - [01/Apr/2026:00:16:27 -0500] "GET /trivia/gallery/index.php?/search/psk-20251126-pf91XbqtmP&display=medium HTTP/1.1" 200 56752 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
2a03:2880:f814:19:: - - [01/Apr/2026:00:16:55 -0500] "GET /trivia/gallery/index.php?/search/psk-20251222-OmGNUiAuBW&display=medium HTTP/1.1" 200 57458 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
2a03:2880:f814:24:: - - [01/Apr/2026:00:36:44 -0500] "GET /trivia/gallery/search.php?tag_id=3,9,19,30,31,33,36,37,40,42,43,44,55,58,61,63,66,81 HTTP/1.1" 302 3999 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
2a03:2880:f814:36:: - - [01/Apr/2026:00:38:14 -0500] "GET /trivia/gallery/search.php?tag_id=9,12,17,22,30,33,36,37,42,43,44,55,57,58,68,69,81 HTTP/1.1" 302 3999 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
2a03:2880:f814:12:: - - [01/Apr/2026:00:40:00 -0500] "GET /trivia/gallery/search.php?tag_id=9,17,22,30,37,40,41,42,43,46,53,56,57,65 HTTP/1.1" 302 3999 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
2a03:2880:f814:24:: - - [01/Apr/2026:00:46:35 -0500] "GET /trivia/gallery/search.php?tag_id=3,5,12,17,30,36,37,40,41,42,43,44,52,55,56,57,58 HTTP/1.1" 302 3999 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
2a03:2880:f814:17:: - - [01/Apr/2026:00:48:48 -0500] "GET /trivia/gallery/search.php?tag_id=3,9,15,19,23,29,30,33,36,40,42,43,44,46,52,55,56,57,60,65,68 HTTP/1.1" 302 3999 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
2a03:2880:f814:31:: - - [01/Apr/2026:00:49:07 -0500] "GET /trivia/gallery/search.php?tag_id=9,15,18,22,30,37,40,41,42,43,44,53,58,61,63,66,81 HTTP/1.1" 302 3999 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
2a03:2880:f814:8:: - - [01/Apr/2026:00:49:43 -0500] "GET /trivia/gallery/search.php?tag_id=3,9,12,22,23,30,36,40,41,42,43,44,53,55,56,58 HTTP/1.1" 302 3999 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
I got 827,092 hits on search.php from Facebook so far this month alone. Doesn't take long to hit millions of searches at that rate.