Announcement

 

Pages: 1

#1 2019-12-24 10:10:05

HHawk
Member
2012-01-24
15

Huge amounts of traffic caused by picture.php (53 GB and counting)

abuse bandwith traffic

Hi fellow Piwigo users,

Since this month I am getting a massive spike in traffic / bandwith being used. Currently it's already 53 GB and still counting. I have no clue how to solve it.

I already blocked the IP's causing this (every time) by nullrouting their IP with /24. But they just find a new IP and start over again. I think they are trying to upload crap or similar. As the error is 503 with a few MB's in size. This is obviously causing a massive increase in traffic.

Simply keeping banning the IP's is not a solution. Also the website is not very popular and is only for my friends and myself (as it has over 12.000 pictures in several galleries from several years back). I really want to keep it online for them, however this traffic issue is getting really annoying.

Ofcourse I can put a password on the website, but I rather not do that. Is there some other way to decrease the traffic caused by this? I searched the forum already and Google, but not really results on this? Am I the only one?

Below you can find the necessary information. If needed more, I will gladly share.

Piwigo version: 2.10.1
PHP version: 7.2
MySQL version: MariaDB 10.1
Piwigo URL: http://is-this-really-needed?

Plugins used: LocalFiles Editor, meta, RV Thumb Scroller (all latest versions)
Theme used: Bootstrap Darkroom (latest available version)

Offline

 

#2 2019-12-24 10:43:48

executive
Member
2017-08-16
1214

Re: Huge amounts of traffic caused by picture.php (53 GB and counting)

strange that you would be targeted with so much traffic. (is your URL really similar to another popular site?)

spammers can't just choose ANY ip they like. It must be within certain range. If its just you and friends/family using the gallery then it should be easy enough to filter everything else out.

Last edited by executive (2019-12-24 10:44:38)

Offline

 

#3 2019-12-24 11:25:46

erAck
Only trying to help
2015-09-06
2029

Re: Huge amounts of traffic caused by picture.php (53 GB and counting)

Maybe your site was discovered by some search engine. Especially a Chinese one tends to query all possible URIs with ever changing IPs. Though it can be annoying it will be eventually over. Maybe it's also some referrer spam. As you didn't indicate how log entries look like (why do you think they're trying to upload something?) or what IP ranges are involved, there isn't much that can be said. If you're concerned about the download data size (53GB with 12000 photos isn't that much, it's just each photo downloaded once) maybe some .htaccess rules taking a shortcut to status 403 could help.

Running Piwigo at https://erack.net/gallery/

Offline

 

#4 2019-12-24 11:31:11

HHawk
Member
2012-01-24
15

Re: Huge amounts of traffic caused by picture.php (53 GB and counting)

As mentioned, it's not downloaded. They are abusing the picture.php which results in a 503.
So both answers are not really helpful. I already removed my website from indexing ages ago. So it must be something else.

And "spammers" can select any IP they want. Unsafe routers, VPN tunnels, etc.

The only solution I can think of is by blocking countries, but that will affect my other sites on my server as well. So that's not really a solution either.

Sure, I will ask 100+ people for their IP so I can white-list them. ;-)

The main issue is that they are clearly targeting picture.php (nothing else).

Offline

 

#5 2019-12-24 12:01:36

erAck
Only trying to help
2015-09-06
2029

Re: Huge amounts of traffic caused by picture.php (53 GB and counting)

picture.php is involved in every call to view an image. Removing a website from indexing doesn't help anything against certain search engines. Referrer spammers do not select any IP, there tend to be patterns. Status code 503 is just "service unavailable" and can be anything your server doesn't handle a load. But if you prefer to keep silence about details then good luck and happy holidays.

Running Piwigo at https://erack.net/gallery/

Offline

 

#6 2019-12-24 12:59:11

HHawk
Member
2012-01-24
15

Re: Huge amounts of traffic caused by picture.php (53 GB and counting)

Apparently my 3rd picture never got uploaded which shows more information.
Maybe this proves to be more useful?

My server is more than adequate to handle any kind of load. Quad-Core Intel X5560 CPU with hyperthreading and 32 GB of RAM. So really that is NOT the issue.

Furthermore; with older versions of Piwigo I never had this issue before.

Offline

 

#7 2019-12-24 13:01:17

HHawk
Member
2012-01-24
15

Re: Huge amounts of traffic caused by picture.php (53 GB and counting)

Pffft... Apparently hit the maximum size.
So doing it now like this:

https://i.imgur.com/7E4Vn1E.jpg

Or link: https://imgur.com/7E4Vn1E

Now it should be visible. I hope.

Offline

 

#8 2019-12-24 14:48:33

erAck
Only trying to help
2015-09-06
2029

Re: Huge amounts of traffic caused by picture.php (53 GB and counting)

As assumed, the request is just viewing a picture page of your recent photos album (recent_pics), hence it is downloading the image. The AH01068 seems to be some timing problem as can be searched for example https://www.qwant.com/?q=%22AH01068%20G … version%22 and AH01075 seems to be related, https://www.qwant.com/?q=%22AH01075%20E … st%20to%22

The IP in the sample screenshot is one of an Amazon net range (34.192.0.0/10). Inspecting the raw access_log entries might reveal whether it's a search bot or not and if so and it doesn't obey robots.txt you could add a BrowserMatch and Deny or RewriteCond %{HTTP_USER_AGENT} and RewriteRule to bail out in Piwigo's .htaccess

Running Piwigo at https://erack.net/gallery/

Offline

 

#9 2019-12-24 15:19:33

HHawk
Member
2012-01-24
15

Re: Huge amounts of traffic caused by picture.php (53 GB and counting)

Well one time it's Amazon, like now. The other time it was "216.244.66.245" which is from Wowrack.com (of which I never heard).

Furthermore this is my robots.txt:

User-agent: *
Disallow: /

Why does robots.txt exist while they aren't obeyed like you mentioned? Kinda sucks.


Other than that I use in my .htaccess:
SetEnvIfNoCase User-Agent "crawl" bad_bot
SetEnvIfNoCase User-Agent "mj12bot" bad_bot
SetEnvIfNoCase User-Agent "360Spider" bad_bot
SetEnvIfNoCase User-Agent "80legs" bad_bot
SetEnvIfNoCase User-Agent "Ahrefs" bad_bot
SetEnvIfNoCase User-Agent "AhrefsBot" bad_bot
SetEnvIfNoCase User-Agent "alltheweb" bad_bot
SetEnvIfNoCase User-Agent "AlphaBot" bad_bot
SetEnvIfNoCase User-Agent "AndroidDownloadManager" bad_bot
SetEnvIfNoCase User-Agent "AnonymizerAttributor" bad_bot
SetEnvIfNoCase User-Agent "AtOPvMzpDosdPDlkm3ZmPzxoP" bad_bot
SetEnvIfNoCase User-Agent "Baidu" bad_bot
SetEnvIfNoCase User-Agent "BLEXBot" bad_bot
SetEnvIfNoCase User-Agent "Bork-edition" bad_bot
SetEnvIfNoCase User-Agent "BOT for JCE" bad_bot
SetEnvIfNoCase User-Agent "Brutus/AET" bad_bot
SetEnvIfNoCase User-Agent "BUbiNG" bad_bot
SetEnvIfNoCase User-Agent "CCBot" bad_bot
SetEnvIfNoCase User-Agent "crawler4j" bad_bot
SetEnvIfNoCase User-Agent "DataCha0s" bad_bot
SetEnvIfNoCase User-Agent "Dataprovider.com" bad_bot
SetEnvIfNoCase User-Agent "deepcrawl" bad_bot
SetEnvIfNoCase User-Agent "Deepnet Explorer" bad_bot
SetEnvIfNoCase User-Agent "desktopsmiley" bad_bot
SetEnvIfNoCase User-Agent "DigExt" bad_bot
SetEnvIfNoCase User-Agent "DomainCrawler" bad_bot
SetEnvIfNoCase User-Agent "DotBot" bad_bot
SetEnvIfNoCase User-Agent "DTS Agent" bad_bot
SetEnvIfNoCase User-Agent "feedfinder" bad_bot
SetEnvIfNoCase User-Agent "gamingharbor" bad_bot
SetEnvIfNoCase User-Agent "GigablastOpenSource" bad_bot
SetEnvIfNoCase User-Agent "GuzzleHttp" bad_bot
SetEnvIfNoCase User-Agent "Harverster" bad_bot
SetEnvIfNoCase User-Agent "Havij" bad_bot
SetEnvIfNoCase User-Agent "heritrix" bad_bot
SetEnvIfNoCase User-Agent "HubSpot" bad_bot
SetEnvIfNoCase User-Agent "ia_archiver" bad_bot
SetEnvIfNoCase User-Agent "Indy Library" bad_bot
SetEnvIfNoCase User-Agent "JDatabaseDriverMysqli" bad_bot
SetEnvIfNoCase User-Agent "juicyaccess" bad_bot
SetEnvIfNoCase User-Agent "larbin" bad_bot
SetEnvIfNoCase User-Agent "Linguee Bot" bad_bot
SetEnvIfNoCase User-Agent "LinkChecker" bad_bot
SetEnvIfNoCase User-Agent "linkdex" bad_bot
SetEnvIfNoCase User-Agent "MauiBot" bad_bot
SetEnvIfNoCase User-Agent "Missigua" bad_bot
SetEnvIfNoCase User-Agent "MJ12bot" bad_bot
SetEnvIfNoCase User-Agent "MRSPUTNIK" bad_bot
SetEnvIfNoCase User-Agent "msnbot" bad_bot
SetEnvIfNoCase User-Agent "Nutch" bad_bot
SetEnvIfNoCase User-Agent "OutclicksBot" bad_bot
SetEnvIfNoCase User-Agent "panscient" bad_bot
SetEnvIfNoCase User-Agent "Pcore-HTTP" bad_bot
SetEnvIfNoCase User-Agent "PHPCrawl" bad_bot
SetEnvIfNoCase User-Agent "plaNETWORK" bad_bot
SetEnvIfNoCase User-Agent "Prospectpup" bad_bot
SetEnvIfNoCase User-Agent "Python-urllib" bad_bot
SetEnvIfNoCase User-Agent "Scrapy" bad_bot
SetEnvIfNoCase User-Agent "Screaming Frog" bad_bot
SetEnvIfNoCase User-Agent "SemrushBot" bad_bot
SetEnvIfNoCase User-Agent "SEOkicks" bad_bot
SetEnvIfNoCase User-Agent "SEOkicks-Robot" bad_bot
SetEnvIfNoCase User-Agent "SISTRIX" bad_bot
SetEnvIfNoCase User-Agent "SiteSucker" bad_bot
SetEnvIfNoCase User-Agent "slurp" bad_bot
SetEnvIfNoCase User-Agent "SMTBot" bad_bot
SetEnvIfNoCase User-Agent "Snapbot" bad_bot
SetEnvIfNoCase User-Agent "Sogou" bad_bot
SetEnvIfNoCase User-Agent "spbot" bad_bot
SetEnvIfNoCase User-Agent "Spinn3r" bad_bot
SetEnvIfNoCase User-Agent "techleadzbot" bad_bot
SetEnvIfNoCase User-Agent "TinEye" bad_bot
SetEnvIfNoCase User-Agent "TwengaBot" bad_bot
SetEnvIfNoCase User-Agent "Twitturly" bad_bot
SetEnvIfNoCase User-Agent "User-Agent" bad_bot
SetEnvIfNoCase User-Agent "Viewzi" bad_bot
SetEnvIfNoCase User-Agent "WebCapture" bad_bot
SetEnvIfNoCase User-Agent "www.deadlinkchecker.com" bad_bot
SetEnvIfNoCase User-Agent "XoviBot" bad_bot
SetEnvIfNoCase User-Agent "Yandex" bad_bot
SetEnvIfNoCase User-Agent "YandexBot" bad_bot
SetEnvIfNoCase User-Agent "YandexImages" bad_bot
SetEnvIfNoCase User-Agent "YebolBot" bad_bot
SetEnvIfNoCase User-Agent "Zauba Crawler" bad_bot
SetEnvIfNoCase Referer "site.ru" bad_bot
SetEnvIfNoCase User-Agent "Windows NT 6.0; rv:34.0" bad_bot
Deny from env=bad_bot

But apparently that's not enough either.

Furthermore; I checked the log as you stated, but I doubt it's a bot:

34.240.10.6 - - [24/Dec/2019:14:50:36 +0100] "GET /picture.php?/6759/recent_pics HTTP/1.0" 503 3624537 "https://www.websitename.nl/index.php?/recent_pics" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9"
34.240.10.6 - - [24/Dec/2019:14:51:28 +0100] "GET /picture.php?/611/recent_pics HTTP/1.0" 503 4762369 "https://www.websitename.nl/index.php?/recent_pics" "Mozilla/5.0 (iPad; CPU OS 9_2_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13D15 Safari/601.1"
34.240.10.6 - - [24/Dec/2019:14:52:16 +0100] "GET /picture.php?/8027/recent_pics HTTP/1.0" 503 5257601 "https://www.websitename.nl/index.php?/recent_pics" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0"
34.240.10.6 - - [24/Dec/2019:14:52:53 +0100] "GET /picture.php?/1598/recent_pics HTTP/1.0" 503 2990195 "https://www.websitename.nl/index.php?/recent_pics" "Mozilla/5.0 (iPad; CPU OS 9_2_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13D15 Safari/601.1"
34.240.10.6 - - [24/Dec/2019:14:53:37 +0100] "GET /picture.php?/10140/recent_pics HTTP/1.0" 503 3513033 "https://www.websitename.nl/index.php?/recent_pics" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0"
34.240.10.6 - - [24/Dec/2019:14:54:39 +0100] "GET /picture.php?/5811/recent_pics HTTP/1.0" 503 5243105 "https://www.websitename.nl/index.php?/recent_pics" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"
34.240.10.6 - - [24/Dec/2019:14:55:32 +0100] "GET /picture.php?/5319/recent_pics HTTP/1.0" 503 5152143 "https://www.websitename.nl/index.php?/recent_pics" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"
34.240.10.6 - - [24/Dec/2019:14:56:11 +0100] "GET /picture.php?/1867/recent_pics HTTP/1.0" 503 3849467 "https://www.websitename.nl/index.php?/recent_pics" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.109 Safari/537.36"
34.240.10.6 - - [24/Dec/2019:14:56:47 +0100] "GET /picture.php?/7031/recent_pics HTTP/1.0" 503 3386131 "https://www.websitename.nl/index.php?/recent_pics" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"
34.240.10.6 - - [24/Dec/2019:14:57:45 +0100] "GET /picture.php?/6988/recent_pics HTTP/1.0" 503 4747515 "https://www.websitename.nl/index.php?/recent_pics" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0"
34.240.10.6 - - [24/Dec/2019:14:58:29 +0100] "GET /picture.php?/1343/recent_pics HTTP/1.0" 503 3529927 "https://www.websitename.nl/index.php?/recent_pics" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0"
34.240.10.6 - - [24/Dec/2019:14:59:13 +0100] "GET /picture.php?/5310/recent_pics HTTP/1.0" 503 3572401 "https://www.websitename.nl/index.php?/recent_pics" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:44.0) Gecko/20100101 Firefox/44.0"
34.240.10.6 - - [24/Dec/2019:15:00:40 +0100] "GET /picture.php?/6888/recent_pics HTTP/1.0" 503 4580193 "https://www.websitename.nl/index.php?/recent_pics" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"
34.240.10.6 - - [24/Dec/2019:15:01:24 +0100] "GET /picture.php?/1822/recent_pics HTTP/1.0" 503 2796163 "https://www.websitename.nl/index.php?/recent_pics" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0"
34.240.10.6 - - [24/Dec/2019:15:02:05 +0100] "GET /picture.php?/2286/recent_pics HTTP/1.0" 503 4327948 "https://www.websitename.nl/index.php?/recent_pics" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"
34.240.10.6 - - [24/Dec/2019:15:02:43 +0100] "GET /picture.php?/2291/recent_pics HTTP/1.0" 503 3304101 "https://www.websitename.nl/index.php?/recent_pics" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"
34.240.10.6 - - [24/Dec/2019:15:03:34 +0100] "GET /picture.php?/7102/recent_pics HTTP/1.0" 503 2700595 "https://www.websitename.nl/index.php?/recent_pics" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"
34.240.10.6 - - [24/Dec/2019:15:04:27 +0100] "GET /picture.php?/10929/recent_pics HTTP/1.0" 503 4612673 "https://www.websitename.nl/index.php?/recent_pics" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36"

At least it's not advertising as a bot.

I went haywire on this now and blocked the complete USA. I used this website: https://www.countryipblocks.net/acl.php

I have no clue if it will help, but I am kinda getting fed up with abuse on traffic. Not that it hurts me, I just cannot stand it.

Offline

 

#10 2019-12-24 15:23:19

HHawk
Member
2012-01-24
15

Re: Huge amounts of traffic caused by picture.php (53 GB and counting)

I also blocked IE obviously. Sorry forgot to mention that.
Which maked the total lines of my .htaccess: 72.000 (!)

Yes, that's not good, but I am really fed up with this to be honest.

Last edited by HHawk (2019-12-24 15:25:47)

Offline

 

#11 2019-12-24 15:57:15

erAck
Only trying to help
2015-09-06
2029

Re: Huge amounts of traffic caused by picture.php (53 GB and counting)

> Why does robots.txt exist while they aren't obeyed like you mentioned

Good robots obey, bad robots don't. It has always been like that.

Looking at the timestamps it seems to be some bot but disguising user agents. Nothing to solve there.

Shrug.. if you want to chase that by banning countries.. I'd just let it keep going until it disappears again unless it continues for ~ever. Maybe something is just scraping all your site and once done it'll be over.

Running Piwigo at https://erack.net/gallery/

Offline

 

#12 2019-12-24 19:55:23

Zentalquabula
Member
2014-05-10
217

Re: Huge amounts of traffic caused by picture.php (53 GB and counting)

34.240.10.6 is a Google cloud customer. You need to start using iptables or another firewall to block ranges from cloud and VPS providers.

sudo /sbin/iptables -I INPUT -s 34.64.0.0/10 -j DROP

Google how to do it and then make it load the rules at startup. Block ranges from Amazon, Rackspace, FDC, OVH, Keyweb, Leaseweb, Plusserver, Linode and anything else that is not a human user.

Learn how to country block with GeoIP as an additional feature. Lot of poop is coming from RU, UA, RO and BR.

Offline

 

Pages: 1

 

Board footer

Powered by FluxBB

github twitter newsletter Donate Piwigo.org © 2002-2024 · Contact