Piwigo.org

JT · 2014-10-30 21:12:12

Well as I explained, soon as I went to the Recent Photos the hard drive started crunching , before I had a chance to choose select whole set and generate M medium ....so i left it like that for quite some time.
But even now the browser thing was still spinning and not all the images had an actual 'preview' on that Recent Photos page (first/page1 didn't check rest), so I had assumed the hard drive crunching was it trying to create previews for the images, previews that show up on that "Recent Photos" page.

Anyway, during this time I had tried to load the main gallery page also but was too slow so i just closed the tab. At least my putty/ssh shell is still working and hasn't frozen like before.

So now I finally decided to go ahead and run the rest of the STEP 4 you mentioned and seleted 'whole set' (85 photos) and chose "M" and clicked to go and now it says:
Generate multiple size images
0/0 0/0

I just checked in the _data/i/galleries/currentgallery/
and there's like 14 or 15 total files mostly -th and one -me and two -sq
Could it be spinning its wheels for nothing? after some time i still see:
Generate multiple size images
0/0 0/0
And still says only 15 images in _data/i/galleries/currentgallery

Now I should mention that this particular install is the SLOWEST of the systems i've tried piwigo on, it is basically a WD My Cloud Mirror device which can also run Wordpress etc. The other systems were real datacenter-grade powerful servers though and still had similar issues.

Can you make a simple OPTION in the configuration somewhere so that we can choose:

1.) TURN ON/OFF Image Resizing to be restricted to ONLY one file at a time.
2.) TURN ON/OFF dynamic Resizing, ie the type that happens when a USER loads an image or a page view showing thumnails so that if Medium sized images or thumnails don't exist we can choose to either let the system create them or just show placeholder instead.

I think these two options shouldn't be hard to add and will make it a whole lot easier for many people to avoid server crashes etc when doing batch uploads of many images or FTP sync etc...

I'm also interested in any write-up wiki-page etc that may explain how to pre-create all the necessary multiple-sized images/thumnails and upload those to piwigo server so it doesn't have to generate anything itself. Of course this isn't very feasible for Community Plugin users.

I would propose an UPLOAD mechanism for BATCH UPLOADING that works even on Community Plugin, user uploads allowing non-flash HTML5 etc batch upload for community plugin usage so that perhaps multiple sized images are created on the CLIENT/Browser side during the upload process so that the server will not need to create such images. I'm thinking that NODE.JS or other browser-side technology may help with this, or create a JAVA APP that can be run from the browser that will handle all this if its not possible with just HTML5/node.js or similar.

Do you have a PLUGIN SDK Tutorial for Piwigo? What are the licensing terms, what license must the plugins created be, any kind of open source license or exactly the same as piwigo etc?

To me it makes the most sense to "AUTOmatically" make the uploader create the needed thumnails, OR at least have them create just the first "M"/medium one if not everything. It should be unnoticed by the end-user uploader, they should just select files they want to upload and click upload, and the piwigo plugin etc should take care of the rest. Then on the server side, if not 'everything' was created by the Client/Uploader then it will just create thumbnails etc from the M/medium and the others on demand/request if so allowed in Album preferences. thanks

JT · 2014-10-30 23:30:04

Hours have passed and uptime on server is still 9.7 or so and on the firefox browser tab with
http://192.168.11.7/photogallery/admin. … ast_import
the circle thing is still spinning/busy while at the bottom it still says:
Generate multiple size images
0/0 0/0

At this point I think i will Reboot again and re-try your Steps 1-5 yet again and see how it goes.
the ls | wc -l on _data/i/galleries/currentgallery still shows same 15 files only...

JT · 2014-10-30 23:56:38

Rebooted and tried your 5 steps again.

But again at STEP 4 i don't think things are going as its supposed to, I only see 4 preview images on teh Recent Photos page out of the 20 on first page, i even think previous time before reboot was the same or even a couple more. And it SEEMS that piwigo starts to try and CREATE these preview images even BEFORE i click submit to create Medium sized images...and that might be the current problem.

Is there a way to DISABLE the creation of these PREVIEW images on the Batch Manager and the "Recent Photos" pages? thanks

JT · 2014-10-31 00:48:35

Well seems it did freeze/crash the server again. I had simply gone to http://serverip/photogallery to see the main galelry page and after a while, it did finally load but the album had no preview image and at the lower right of the page there was some kind of floating bar with apparent movement as if it was trying to load something or do something. Anyway after a while with that bar not going away i just closed the tab.

But now my putty/ssh to the server is unresponsive/frozen again and I am forced to reboot the server again. At this point I will wait for Plg to see what suggestions he may have. thanks!

flop25 · 2014-10-31 09:35:57

what's the server cpu etc? for instance i have a very small server 1.80GHz 4go ram which runs 2 piwigo 1 WP and other stuff; the galleries are faster than wp and the picture generation doesn't crash the server just a hight load of the cpu
piwigo won't change the technology because the script is meant to be used on shared server too

JT · 2014-10-31 21:59:09

Thanks for the reply, as I've mentioned I have this problem on several different servers, some quite powerful, but this current one is the smallest, I mentioned it was the WD My Cloud Mirror 8TB and it should have a 1.2Ghz CPU, it has wordpress on it also. This piwigo installation is not meant to be used my multiple users, just ONE user at a time and the only problem is this image resizing generation, I think that without this issue piwigo would run just fine on it.

I was hoping PLG had some suggestions on how I can get piwigo running without killing the server since I already tried his STEPS 1-5 suggestions but got stuck at Step 4 which seemed to be because piwigo was trying to create "PREVIEW" images on the "Recent Photos" page. (As it also does on the Batch Manager page)

So one of my important questions is how can i DISABLE preview image generation on the Recent Photos page and the Batch Manager page so when i go to those pages piwigo doesn't try to generate 'previews' of the images there.

I think an uploader that handles generating resized images would be a huge improvement.

JT · 2014-10-31 22:03:55

The other piwigo install that gave me similar problems with image resizing when uploading lots of images, ftp sync etc has powerful Xeon server cpu:
Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz

flop25 · 2014-10-31 22:05:34

are you using Imagemagik and a recent one?

the ultimate solution would be to prepare your resized pics locally and upload them after the sync/upload

plg · 2014-10-31 22:07:12

JT, are you running PHP with suPHP?

JT · 2014-10-31 22:37:23

Hi guys,

On the weakest current system I'm dealing with I'm not sure what its running, its an off-the-shelf appliance, but I did not see 'convert' so I'm assuming piwigo is using "GD", I didn't mess with piwigo defaults in any way on this system.

On the other system, I think I had 'some' improvement maybe when changing piwigo config file to go with ext_imagick and pointing it to /usr/bin where the convert binary is located. And that setup is debian with Fast-CGI mode not SuPHP. But honestly I think you shouldn't be pushing for some kind of strict specific install to this problem, the better solution would be for piwigo to control these image resizing processes better.

One recommendation I made was for some options, could be global but even per album maybe, where we could simply "Disable/Enable" dynamic image resizing completely. It seems to be that even after doing those STEPS 1-5 you mentioned, that didn't save me from the actual "Preview Images" that are embedded inside the "Recent Photos" page or the "Batch Manager" page from trying to spawn some kind of processes to resize originals and provide a preview image. In my case, if you simply used the 'entire' FULL resolution image for these previews (of course with html width=100 or something) it would probably be much faster since the bandwidth is not a problem.

What would happen if I rename i.php or what's the best way to cripple resizing? In the piwigo config file?

Also, as I've mentioned, is there a wiki or some writeup with good info such that I can pre-create all the necessary image sizes along with proper naming and where to upload each etc? Ideally there should be a plugin to handle all that since it seems very time consuming otherwise, that's why I was interested in your PLUGIN licensing terms since I'd consider creating a tool myself if none really exists.
thanks!

plg · 2014-10-31 22:53:06

JT wrote:
But honestly I think you shouldn't be pushing for some kind of strict specific install to this problem, the better solution would be for piwigo to control these image resizing processes better.

Well... honestly if your problem was so frequent for sure we would work on it stronger. Actually it's not a common issue and instead of thinking to improve Piwigo, I wonder what can be different in your environment compared to the thousands (or much more actually) where Piwigo runs perfectly fine :-)

There is no "guide" for pre-generating the resized photos, because this problem is very uncommon. Let's see what need to be done : for all photos in "galleries", you have to create 9 files in _data/i/galleries. Here is an example :

* galleries/album_for_jt/a-flower-in-a-field.jpg (the original file)
* _data/i/galleries/album_for_jt/a-flower-in-a-field-sq.jpg (size Square)
* _data/i/galleries/album_for_jt/a-flower-in-a-field-th.jpg (size Thumbnail)
* _data/i/galleries/album_for_jt/a-flower-in-a-field-2s.jpg (size XXS)
* _data/i/galleries/album_for_jt/a-flower-in-a-field-xs.jpg (size XS)
* _data/i/galleries/album_for_jt/a-flower-in-a-field-sm.jpg (size Small)
* _data/i/galleries/album_for_jt/a-flower-in-a-field-me.jpg (size Medium)
* _data/i/galleries/album_for_jt/a-flower-in-a-field-la.jpg (size Large)
* _data/i/galleries/album_for_jt/a-flower-in-a-field-xl.jpg (size XLarge)
* _data/i/galleries/album_for_jt/a-flower-in-a-field-xx.jpg (size XXLarge)

JT · 2014-10-31 23:58:41

Hi Plg,

Many thanks for the example file names you gave for image generation, do you have the specific 'resolution widthx height' to go with each of those so as to keep piwigo templates happy etc?

I'm very surprised users have not come up on this issue since as I've said I can duplicate it at will on many different servers, so my only conclusion is that majority of piwigo users run very small setups.

Do you have any example installations where you have piwigo galleries with "original" photos that are of the resolution I am using who are uploading many images at one time either by FTP sync or community users batch uploading "large" files where the gallery and server is still usable during this upload/sync of batches of large high resolution images?

I would assume that you do want piwigo to be used on large website and for high resolution images and I think perhaps due to many users using shared hosting which throttles cpu and has other safeguards to limit processes, they haven't seen servers crippled by piwigo during large batch upload/sync. And likely others who may run their own server either VPS or dedicated perhaps are not using FTP sync much and just uploading a "few" photos at a time only using the piwigo web interface.

If you have a fresh piwigo install somewhere, I can probably duplicate some of my photos and do a test on one of your own server setups. I've tried it on at least 3 setups where the resize/sync has become an issue and I think it would help piwigo in the long-run to fix this problem even if not too many people are experiencing it 'yet'.

There perhaps are others who tested piwigo, noticed the issues with terrible slowdown/freezing on large batch uploads and simply dropped piwigo completely without bothering to post here.

I'd love to hear from piwigo site administrators and see if they have piwigo sites that get at least a hundred or couple hundred high resolution batch images uploaded at one time and see how their servers handle the sudden image resizing for all those images while at the same time not freezing the gallery for the many people just viewing albums on the server.

Another question is, does the Community Plugin have ability to upload BATCH Images without Flash? ie HTML5? I think it does not currently right? Will it soon?
thanks!

JT · 2014-11-01 00:17:45

Also I did post already in this thread example of others who posted on your forums with similar performance issues:
http://piwigo.org/forum/viewtopic.php?id=22348
http://piwigo.org/forum/viewtopic.php?id=23686

So is there an easy way for me to completely disable (in a reversible way) dyanmic image resizing so that even Recent Photos page or Batch Manager doesn't try to run image resizing to create the preview images on those pages?
thanks!

JT · 2014-11-01 00:43:09

Also keep in mind that 48Megapixel and 75megapixel or more cameras are coming out soon so as resolution increases more people will notice this issue when they upload a batch of 500 or 1000 large images and the poor server has to try and create several image sizes for each one, it would actually be a miracle if it could do it without hiccups unless on some super-grid computer :D

Currently my images are about 7k by 5K resolution (jpeg size 7MB to 20MB) and batch uploads vary from 100 to 500 images at once so far.

I have not looked under the hood at piwigo code yet, other than having fixed some bugs with the EXIF plugin, but I'd be curious if piwigo does try to create Medium format file first by default and then uses that version to create the smaller file and the smaller file to create the smaller thumbnail etc...

If there's room for optimization I'm all for it, I don't mind donating towards a specific FIX or optimization that I want. In fact i'm curious why many open source projects don't make more opportunities for users to pay for certain features/optimizations, especially when many of them could be trivial to implement and not really affect the overall direction/roadmap.

So for now if you could provide these things I'd be very grateful:

1. Resolution/size for the image resizing that you provided in previous post, you gave filenames etc so would be great if you can specify resolution that piwigo currently sets for each of those.

2. Tell me of a way, either by editing config file or some php file etc so that I can disable piwigo's attempt to start 'resizing' images for preview purposes on the "Recent Photos" or "Batch Manager" pages in Adminstration control panel.

thanks!

JT · 2014-11-03 03:46:40

So for now until I hear back from Plg, I made some good progress by DISABLING i.php by simply renaming it to i-REISZENOMORE.php

This has allowed me to go to photogallery admin pages 'Recent Photos' or 'Batch Manager' without freezing/crashing the server requiring a reboot. The gallery worked quite well and fast with i.php renamed. I was able to get the resolution for resized images from a couple of successful ones that had already created resized versions for -th, -me, and -sq
-me (medium) 792 x 528 (changing to 792x529 which is closer approximation, GD did 792x528)
-me (vertical/portrait) 396x594
-sq (square) 120x120
-th (thumbnails) 144x96
-th (thumnail portrait/vertical) 96x144

It is confirmed then that the BEST solution is to create thumnails on the CLIENT side, at least all the way from XL, Large, to Medium (-me) instead of having the server create all of them. Especially for when uploading hundreds of high resolution 7.4K x 4.9K images or similarly high res photos it may be a necessity to manually resize to avoid serious server slowdown or even freezes/crashes requiring reboot.

JT wrote:
So for now if you could provide these things I'd be very grateful:

1. Resolution/size for the image resizing that you provided in previous post, you gave filenames etc so would be great if you can specify resolution that piwigo currently sets for each of those.

2. Tell me of a way, either by editing config file or some php file etc so that I can disable piwigo's attempt to start 'resizing' images for preview purposes on the "Recent Photos" or "Batch Manager" pages in Adminstration control panel.

thanks!

Piwigo.org

Announcement

#16 2014-10-30 21:12:12

Re: PIWIGO crashes freezes servers

#17 2014-10-30 23:30:04

Re: PIWIGO crashes freezes servers

#18 2014-10-30 23:56:38

Re: PIWIGO crashes freezes servers

#19 2014-10-31 00:48:35

Re: PIWIGO crashes freezes servers

#20 2014-10-31 09:35:57

Re: PIWIGO crashes freezes servers

#21 2014-10-31 21:59:09

Re: PIWIGO crashes freezes servers

#22 2014-10-31 22:03:55

Re: PIWIGO crashes freezes servers

#23 2014-10-31 22:05:34

Re: PIWIGO crashes freezes servers

#24 2014-10-31 22:07:12

Re: PIWIGO crashes freezes servers

#25 2014-10-31 22:37:23

Re: PIWIGO crashes freezes servers

#26 2014-10-31 22:53:06

Re: PIWIGO crashes freezes servers

JT wrote:

#27 2014-10-31 23:58:41

Re: PIWIGO crashes freezes servers

#28 2014-11-01 00:17:45

Re: PIWIGO crashes freezes servers

#29 2014-11-01 00:43:09

Re: PIWIGO crashes freezes servers

#30 2014-11-03 03:46:40

Re: PIWIGO crashes freezes servers

JT wrote:

Board footer