Announcement

#1 2016-04-20 23:21:30

PolyWogg
Member
2014-08-17
59

High I/O usage revisited...

Just over 2 years ago, Tom posted at http://piwigo.org/forum/viewtopic.php?id=23987 a post about the same problem I am having now. Since it's 2 years old, and partially resolved, I am posting new one.

I'm with GreenGeeks.com. Have basic WP install plus separate install for Piwigo. Nothing fancy, no huge number of plugins, relatively vanilla install. I do, however, have a lot of photos (>7000) at http://www.polywogg.ca/photos/

However, suddenly my images stopped rendering -- I upload and display the full res image -- this isn't a high traffic site, just a few friends, etc. and they frequently want to copy photos of themselves or their kids, so it's easy to do so.  It used to be no issue, suddenly my site is not rendering at all. Checked, yep, my server is telling me I'm timing out. Here is their message:

>> Your site has been limited within the past 24 hours

>> You have reached entry processes (number of simultaneously running php and cgi
>> scripts, as well as cron jobs and shell sessions) limit 94 times

>> I/O usage resources were limited for your site

>> Your site might hit resource limits soon

>> Your Physical Memory usage was at 429.11 out of 256

>> You had 938 I/O operations per second out of 1024 max I/O operations
>> per second allowed

THey suggested I add CloudShare and _mod_Deflate to speed up the site but didn't seem to make any difference. I uploaded new pics a couple of days ago, but nothing in the last 36 hours so it likely isn't still processing my big images down to little ones, is it? It was maybe 50 new pics?

Running 2.8.0, rest is relatively vanilla.

According to site, it's using a lot of CPU:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
434543 polywogg 20 0 293m 40m 10m R 34.3 0.1 0:05.84 /usr/bin/php
434497 polywogg 20 0 305m 52m 10m R 32.6 0.1 0:07.54 /usr/bin/php
434544 polywogg 20 0 300m 47m 10m R 32.6 0.1 0:06.04 /usr/bin/php

Their suggestion, of course, was that it was the software's problem, not their silly limited hosting site. It is a relatively basic plan, but it is not THAT basic. THeir official diagnosis was "Presumably there is some issue with your software, which cause high resource usage of your account and problem with displaying image."

At this point, if I follow the previous post from two years ago that Flop25 helped solve, it looked like he was going to upload smaller images instead. Which I would do if it would likely help, as loading album pages overwhelms the site, but even a SINGLE page like

http://www.polywogg.ca/photos/picture.p … tegory/285

won't load.

I confess that makes no sense to me -- a single page with a single image can't be generating too many requests to the server, can it?

I confess the above is about the limit of my technical abilities, so I hope I understand any replies (Flop25 said something at the end of the previous thread about _data/i, and I have no idea what that is). Either way, thanks for any help people can offer...

About the only thing that has changed in the last two months was the upgrade to the new version AND I had to change to a different theme than I had been using (clear instead of pure).

Paul
aka a very confused PolyWogg

Environment

        Operating system: Linux
        PHP: 5.6.20 (Show info) [2016-04-20 21:17:16]
        MySQL: 10.0.24-MariaDB [2016-04-20 17:17:16]
        Graphics Library: External ImageMagick 6.7.2-7

Database

        7126 photos (first photo added on Sunday 13 September 2015)
        279 albums including 0 physical and 279 virtual (7126 associations)
        19 tags (257 associations)
        2 users
        0 groups
        0 comments
        no rate

Offline

 

#2 2016-04-21 17:39:53

plg
Piwigo Team
Nantes, France, Europe
2002-04-05
13177

Re: High I/O usage revisited...

Hi PolyWogg,

PolyWogg wrote:

[...] I do, however, have a lot of photos (>7000) [...]

Sorry but that made me laugh :-) A "lot" of photos is more 500k than 7k. For Piwigo I mean.

From what I see, your hosting is pushing the limit of HTTP requests crazy low. Piwigo is already efficient to avoid too many HTTP requests. Of course we can't avoid as many HTTP requests as there are thumbnails on a page!

My only recommendation is to find a better hosting. Of course I know one that works perfectly with Piwigo ;-) but there are others.


Latest blog post (November 9th 2018) Why Flickr could not remain free for ever

Offline

 

#3 2016-04-21 18:34:38

deheme
Member
IDF
2014-12-12
92

Re: High I/O usage revisited...

Hi PolyWogg,

I do not consider to have a big set :
Database
27756 photos (first photo added on Friday 7 March 2014)
984 albums included 475 physical and 509 virtual (56809 associations)
164 tags (19093 associations)
29 users
6 groups

Shared on 2 independant Piwigo's

All my originals are full size (16Mpx and some 36Mpx) transmitted thru ftp.
Never had problem during synchronisation excepted when I synchronize the full set simultaneously on my two Piwigo bases on the same directory (symlink) ...
Rendering thumbnails is sometime a bit long but never as long as on your site...

DéHème

Offline

 

#4 2016-04-21 23:22:00

PolyWogg
Member
2014-08-17
59

Re: High I/O usage revisited...

plg wrote:

Sorry but that made me laugh :-) A "lot" of photos is more 500k than 7k. For Piwigo I mean.

Hehehe That's not a lot, that's an "insane" amount :) Most sites I clicked on had a few hundred, or maybe that was all that was visible to me.


plg wrote:

From what I see, your hosting is pushing the limit of HTTP requests crazy low. Piwigo is already efficient to avoid too many HTTP requests. Of course we can't avoid as many HTTP requests as there are thumbnails on a page!

Do you mean the 1024 I/O requests per second? For $2 extra a month, I can triple that.

I played with settings today, and *might* have found a fix, just don't know which one. I switched themes, deactivated all my plugins, upgraded site to PHP 7 instead of 5.5, and in doing the last one, noticed that image_magick and GD were not activated. So activated them. However, I didn't have the option ticked to generate all the images anyway, so now I have. And it's loading for other people. Not on my phone, but that's another issue I think.

Anything else to consider before I slowly turn everything back on to see if it is messing things up?

Paul
aka PolyWogg

Offline

 

#5 2016-04-23 10:47:03

plg
Piwigo Team
Nantes, France, Europe
2002-04-05
13177

Re: High I/O usage revisited...

PolyWogg wrote:

Do you mean the 1024 I/O requests per second?

I have no idea. I just see that you don't have many photos. But actually that doesn't seem to be the real issue. The problem seems to be the number of HTTP requests. If you only have 10 photos but 1 million visits each day, then your hosting provider says "it's too much". If you have 500k photos but 10 visits each day, you will generate much less HTTP requests.

So the question is "how many millions visits your Piwigo receives each day?".

PolyWogg wrote:

Anything else to consider before I slowly turn everything back on to see if it is messing things up?

I keep my recommendation: find a better hosting provider (or ask this one to solve your problem, that's their job, they can even contact us to find a solution)


Latest blog post (November 9th 2018) Why Flickr could not remain free for ever

Offline

 

#6 2016-04-23 14:48:12

PolyWogg
Member
2014-08-17
59

Re: High I/O usage revisited...

Thanks, it shouldn't be that busy. This is a personal hobby site, and I'm not a social butterfly with an active photo fetish :) Which means I should be hitting single digits on a good day i.e. I've uploaded an album, I share it on Facebook, maybe 15 people over two days might stop by. That's about it.

I'm locked in for my supplier for a bit, so not switching right away and they are about tapped out in finding solutions. It seems to work for most people for viewing on desktop again, and links to the images through WP don't seem to have a problem loading, but if I load it on a mobile device, no pics. Not consistent on the desktop either, but most of the time doesn't overwhelm.

But sometimes, even a single pic page is enough to generate too many HTTP requests. I've boosted from 1K I/O reqs per sec to 3K, and it helped, but once in awhile, it seems to overwhelm. They think I must have PIWIGO mis-configured, which is entirely possible, I just have no idea where to look to see what I might have messed up. I don't have enough knowledge to have gone mucking with code or anything, it's a vanilla install from Softaculous originally, and I just added pics and plugins and themes after that. If I try to load a Piwigo page, say a single image, it won't load. If however I go directly to the image itself i.e. COPY IMAGE LOCATION and load that page? No problem. It's definitely Piwigo config causing the problem.

Regrettably, I seem to have no way forward to fix whatever isn't tweaked right in Piwigo, so I think I'm going to have to start over for the fifth time with a different product. Gallery 1, Gallery 2, Gallery 3, and Piwigo seem not to be good solutions for me, so I may just go with a WordPress product. My server doesn't seem to mind the load for it...I could blow off Piwigo and do a backup of the files and reinstall them, but not sure to what end...

Sigh.

Paul

Offline

 

#7 2016-04-24 00:07:08

flop25
Piwigo Team
2006-07-06
6872

Re: High I/O usage revisited...

everyone here including non team members is saying that your web hosting is a rubbish but you still what to change the script again? In 2016, you have a Moskvitch 408 and you want the latest 5.1 surround sound : not enough power but you are still changing the hifi modules
well jokes apart:

noticed that image_magick and GD were not activated

so now what exactly is the status? because I don't see anything better
http://www.polywogg.ca/photos/index.php?/category/287 I get sometimes "GET 20160418223524-5b50cc55-th.jpg 508 Loop Detected" but usually I get what's in the screenshot attached


Uploaded Images


To get a better help : Politeness like Hello-A link-Your past actions precisely described
Check my extensions : more than 30 available
who I am and what I do : http://fr.gravatar.com/flop25
My gallery : an illustration of how to integrate Piwigo in your website

Offline

 

#8 2016-04-24 00:09:39

flop25
Piwigo Team
2006-07-06
6872

Re: High I/O usage revisited...

wow after of dozen refresh pics are appearing one by one!
do they use a cache on their server? if yes it's quite a bit aggressive


Uploaded Images


To get a better help : Politeness like Hello-A link-Your past actions precisely described
Check my extensions : more than 30 available
who I am and what I do : http://fr.gravatar.com/flop25
My gallery : an illustration of how to integrate Piwigo in your website

Offline

 

#9 2016-04-24 02:45:37

PolyWogg
Member
2014-08-17
59

Re: High I/O usage revisited...

Thanks Flop & Tom et al,

I do understand about the server, but to use your stereo example, it's like I have three components:

a. Receiver (say controlling I/O outputs per second, 1MB/second originally, with 3MB and 5MB higher end options);
b. CD player (say being the physical memory originally at 256MB); and,
c. Speakers (say being the virtual memory at 1GB).

While we all may agree that the output is rubbish, I was trying to determine which of those three was the variable I needed to fix either with them (higher package) or with others even (unlikely, but possible). I didn't get an answer from here or the tech support which of the three was the biggest limiting factor, but not sure I am framing the question correctly.

On top of that, I am also not sure if I have some other problem besides the stereo setup:

d. Crappy wiring (i.e. my Piwigo setup such as the image handlers -- for setup, both GD and IMagick are now both available, and I have Piwigo using GD right now);
e. Faulty remote (i.e. PHP 5.6 vs. 7.0);
f. High end recordings with lots of range (i.e. high-res full size images vs. thumbnails).

Part of my ongoing challenge is that I am maxing out my technical abilities here. I posted a link to the site for friends on FB to test, and some of them had problems earlier, but when I upgraded to 3MB/s and 384MB physical memory, while keeping virtual at 1GB, they seemed to be able to see it okay. At the same time, though, my mobile app could not. Desktop working, tablet yes, phone no. But it's not consistent. Which argues as you said above, it's the server.

However, then I took a fresh WordPress install, dumped a set of the photos into a page at full-size resolution. If it's the server, it shouldn't matter much between WordPress and Piwigo, rendering the same page more or less with 30 thumbnails on it or 30 large pics should be comparable. Yet my phone / tablet / desktop has no problem with any of it. It's not super fast to load, but it's just load-size slow i.e. rendering 30 full-size images. Nowhere near capacity on the server.

Which then puts me back to thinking it's not the server or the load, it's something I screwed up in Piwigo. I even did another WordPress page where I pulled the images DIRECTLY from my Piwigo install. Displays them no problem at all. Had a bunch of people check, worked fine for them too. Same images, works in WP but not on the Piwigo page for that album.

Caching could be part of the issue -- i.e. I load it, works, then the next 5 people get the cache too and it works for them. But why does it work without it in WP but not in Piwigo?

As I said, I'm over my technical capacity, so I worry I'm wasting your time. I don't know enough to say, "Hey, could it be this?" so I'm kind of fumbling in the dark. I can tell you the variables above (a,b,c,d,e,f), but where to tweak to isolate the cause and be sure, I have no clue.

I'll check out piwigo.com. Perhaps if it's configured correctly? :)

PolyWogg

Offline

 

#10 2016-04-24 18:21:11

PolyWogg
Member
2014-08-17
59

Re: High I/O usage revisited...

So I might have found some of the problem doing a deep dive on the logs...in analyzing them, I have 11K "requests" to the server between 5:04 PM on 23rd and 10 AM this morning.

Of those 11K requests -->


****************************************************************
- 2597 completed with code 200
     - most were GETs however there were a couple of CRON requests:
             POST /wp-cron.php?doing_wp_cron=# which are not Piwigo, look okay;
     - several HEAD requests, not sure what they are yet;

****************************************************************


All of them look relatively normal...

HOWEVER over 7000 of the remaining 8500 were Piwigo so it does look like I have something messed up in my install or configuration:


****************************************************************

- 3902 were 301 redirects -- ALL of these were in Piwigo, not sure what they are, all look like:
     - GET /photos/_data/i/upload/2016/04/12/20160412215041-35125533-me.jpg HTTP/1.1

- 3593 were 404s -- these are very similar to the 301s in format, almost all GETs
     - GET /photos/_data/i/upload/2015/09/20/20150920070427-612fa7f4-th.jpg HTTP/1.1

****************************************************************

OTHERS:

- 18 were 302s

- 75 were 304s

- 1322 were 403s -- with a lot of Posts? and even robots.txt? Could be an attack I guess

- 230 in the 500 range

****************************************************************

It looks like both the 301s and 404s are pointing to alternate size versions of large images...does that give a clue to what I have mis-configured?

Paul

Offline

 

#11 2016-04-24 23:18:22

flop25
Piwigo Team
2006-07-06
6872

Re: High I/O usage revisited...

Hi
I've investigated but my knowledge and degree of latitude are limited ; in fact that's all graphics under /photos which are impacted. Icons, thumbnails, large pics... are crazy : 301, 404, loaded but then 404 ... Js, html, css are ok
So that means you probably have a directive somewhere (php.ini, htacess, apache/nginx ...) which messed at the server level, not at the script level (hotlink protection matching url with 'photos'?) ; that's definitely clear for me that Piwigo runs great since you enabled gd and IM to generate pictures. And according to the fast reply of the server to requests other than graphics your server might be very good in fact.


see that image It exists, it's sometimes displayed but refresh it and it disappears http://polywogg.ca/photos/_data/i/upload/2015/11/22/20151122172056-f4aa36fa-me.jpg


To get a better help : Politeness like Hello-A link-Your past actions precisely described
Check my extensions : more than 30 available
who I am and what I do : http://fr.gravatar.com/flop25
My gallery : an illustration of how to integrate Piwigo in your website

Offline

 

#12 2016-04-25 02:02:10

PolyWogg
Member
2014-08-17
59

Re: High I/O usage revisited...

So at one point they auto upgraded me from PHP 5.6 to 7.0, I had a conflict with some other software installs I was doing (to do list, a calendar), so we went back down to 5.6. Then I blew off the other pieces and upgraded again to 7.0. With the problems on the weekend, plus Piwigo was giving me a series of redundant errors (forget the wording, deprecated in 7.0 i.e. things in Advanced Menu Manager for example had calls that wouldn't work in future PHP, that sort of thing), the people put me back down to 5.6.

At least one of those changes messed up my defaults and installs / options, etc. for PHP, maybe more...here is what I show in my install:

OFF:
Language Options     asp_tags     Allow ASP-style <% %> tags
Language Options     safe_mode
Data Handling, register_globals

ON:
File Uploads

RESOURCE LIMITS:
max_execution_time -- scripts -- 30 seconds -- set to 90 seconds in options
max_input_time -- script parsing data -- 60 seconds -- set to 120 in options
memory_limit -- per script -- 32M -- set to 256M in options
upload_max_filesize     -- Maximum allowed size for uploaded files -- 16M (I'm sure I had that changed to 64MB previously) -- yes changed in PHP options
post_max_size -- 64M **** wait a minute, I wonder...on pages where I have full-size images, if they were all 5-7MB, and there was more than 10 or 11 of them...not sure what it would do.

For my PHP extensions, I have checked:

bcmath, dom, enchant, fileinfo, gd, http, imagick, imap, intl, ioncube_loader, json, mbstring, mcrypt, mysql, mysqli, mysqlnd, pdo, pdo_mysql, pdo_sqlite, phar, posix, propro, pspell, raphf, sockets, tidy, wddx, xmlreader, xmlrpc, xmlwriter, zip

I do NOT have these checked but none of them seem required:

apcu, apm, bit_int, bitset, bz2_filter, dba, dbx, doublemetaphone, eio, gender, geoip, gnupg, htscanner, inotify, interbase, jsmin, ldap, libevent, lzf, mailparse, memcache, memcached, mongo, msgpack, mssql, ncurses, nd_mysql, nd_mysqli, nd_pdo_mysqli, oauth, oci8, odbc, opcache, pdf, pdo_dblib, pdo_firebird, pdo_odbc, pdo_pgsql, pgsql, phalcon, quickhash, radius, recode, redis, rsync, snmp, soap, sourceguardian, spl_types, ssh2, stats, stem, stomp, suhosin, sybase_ct, sysvmsg, sysvsem, sysvshm, timzonedb, trader, translit, uploadprogress, uri_template, uuid, weakref, xdebug, xrange, yaz, zend_guard_loader

If anyone sees anything that is drastically amiss, let me know!

PolyWogg

Offline

 

#13 2016-04-25 14:02:27

JAK
Member
2013-07-07
48

Re: High I/O usage revisited...

Just a thought, if the photos being loaded are straight out of the camera originals, the photo dimensions (in pixels) may be high and Piwigo is certainly having to scale them down. This does take a lot of processing time on a low power server (such as mine!) If the photos are scaled down before uploading to Piwigo in a program such as Irfanview, Faststone Photo Resizer  or Elements this processing requirement is eliminated or much reduced.  Reducing the pixel dimensions to say, 1200 pixels max for the width or 800 pixels max for the height vastly reduces the server processing requirements.  This wouldn't have been a problem for photos taken with an older camera as 1200 x 800 pixels was about the norm for digital images but now the pixel dimensions can be above 7000 x 5000, hence the possible problem.

Another advantage of pre-preparing the photos before uploading offers the possibility of correcting exposure errors or doing judicial cropping.  Friends can always request copies of the full size originals if you are happy for them to have them.

If you are already doing these things then disregard my suggestions and look for some other server issue!

Offline

 

#14 2016-04-25 14:14:02

flop25
Piwigo Team
2006-07-06
6872

Re: High I/O usage revisited...

PolyWogg wrote:

I do NOT have these checked but none of them seem required:

apcu, apm, bit_int, bitset, bz2_filter, dba, dbx, doublemetaphone, eio, gender, geoip, gnupg, htscanner, inotify, interbase, jsmin, ldap, libevent, lzf, mailparse, memcache, memcached, mongo, msgpack, mssql, ncurses, nd_mysql, nd_mysqli, nd_pdo_mysqli, oauth, oci8, odbc, opcache, pdf, pdo_dblib, pdo_firebird, pdo_odbc, pdo_pgsql, pgsql, phalcon, quickhash, radius, recode, redis, rsync, snmp, soap, sourceguardian, spl_types, ssh2, stats, stem, stomp, suhosin, sybase_ct, sysvmsg, sysvsem, sysvshm, timzonedb, trader, translit, uploadprogress, uri_template, uuid, weakref, xdebug, xrange, yaz, zend_guard_loader

that's ok for me ; geoip memcache pdf for instance might be usefull in the futur but nothing mandatory


To get a better help : Politeness like Hello-A link-Your past actions precisely described
Check my extensions : more than 30 available
who I am and what I do : http://fr.gravatar.com/flop25
My gallery : an illustration of how to integrate Piwigo in your website

Offline

 

#15 2016-04-26 06:36:49

PolyWogg
Member
2014-08-17
59

Re: High I/O usage revisited...

Well, I've made progress, just not as much as I thought I had.

a. I found out that one of the changes from PHP 5.6 to PHP 7.0 didn't keep my original settings, so that triggered some fun that might have messed up my install earlier.

b. I changed a security plugin in my WordPress area that might have been a bit overzealous in protecting my site, and messed up "lower" installations in the hierarchy.

c. Back when I first set up my website, they had a "sandbox / staging area" where you could get your site all set up, then remove a DNS record entry or something and it would go live when it was ready. But when I went live, www.polywogg.ca wouldn't work, only polywogg.ca would work. They adjusted some stuff, but every once in awhile, it doesn't like the lack of www or the reverse, it doesn't like when it's there. So I've put in a temporary overarching redirect to take www.polywogg.ca and map to polywogg.ca because the site is configured that way.

d. I was getting a LOT of traffic from China that might have been an attack, couldn't tell. Either way, I blocked it. I'm starting to think though that they might have just been looking at some of my clip art that's on there, and I thought was "hidden" but was not apparently. Google was still finding it. I've set it to private now, don't know if it will make a difference or not.

I may indeed start uploading pre-processed images. I really liked having the full image up there. Not sure if I will follow plg over to piwigo.com yet though.

However, I am not out of the woods yet. One problem needs more work, and I can solve that, but here's the wonky thing that is still happening. Someone requests a specific image with a get message, and here's what happens all one second apart:

POST /photos/picture.php?/3556/categories HTTP/1.1
GET /photos/picture.php?/3556 HTTP/1.0
POST /photos/picture.php?/3556/categories HTTP/1.1
POST /photos/picture.php?/3556/categories HTTP/1.1
POST /photos/picture.php?/3556/categories HTTP/1.1

The four posts are returned as 403 codes, the single get completes with 200 code. But it looks to me like the single request generated five separate requests to the server. Is that normal? Or is it the first POST command came back 403, so it tried it 3 more times while the GET command completed the first time?

Another came up:
GET /photos/picture.php?/1120/categories&metadata HTTP/1.1
GET /photos/picture.php?/1120 HTTP/1.1
GET /photos/picture.php?/1120 HTTP/1.1

Two 200 completions and a 302 but again, multiple requests for the same file coming from a legitimate IP, all with the same time stamp down to a second. I don't know if it issued the request 3 times, or my server interrupted a click as demanding three things?

I've reduced the load, but not sure I have "most" of it fixed yet.

Paul

Offline

 

Board footer

Powered by FluxBB

github twitter facebook google+ newsletter Donate Piwigo.org © 2002-2018 · Contact