#46 2021-10-22 14:53:39


Re: PIWIGO crashes freezes servers

Apologies for resurrecting an old thread but I hit this issue and thought I would share my solution. It's probably only relevant to small self-hosted servers but it may help someone.

After success with some small tests with Piwigo on my newly built NAS I tried with some of our real backup directories of photos using symbolic links and Synchronize. It coped very well with the older (smaller) ones but once I started linking in some more recent stuff I hit this issue several times when trying to use the batch manager to tag an album, needing to power the NAS off and on to make it work again.

After poking around with top and vmstat to see what was actually happening on a smaller directory I found that the CPU was almost idle but the memory was full with plenty of swap space used. The problem seemed to be a caused by a large number of ImageMagick 'convert' processes each requesting a decent chunk of memory.

As far as I can see Piwigo puts 20 thumbnails on the batch manager page. Each thumbnail causes the browser to send a request back to the server, many of them in parallel. If the thumbnail does not exist then the server spawns a 'convert' process to generate the thumbnail. In my case the NAS memory could not cope with this demand.

My first step was to modify /etc/apache2/apache2.conf and add a 'MaxRequestWorkers 3' line at the end. This limits the number of Apache workers used to handle incoming requests. The apache workers spawn the 'convert' processes so limiting the workers also limits the number of 'convert' processes that will run at the same time.

This made my NAS stable again which is a good thing. If you have this issue and are running your own Apache then it may work for you. It is rather a blunt instrument because there may be other situations when having such a small number of Apache workers will be a problem but I'm only running Piwigo on my Apache so it's fine. It does mean that thumbnails tend to get generated and then display in batches of three... but that is far preferable to killing the NAS.

Piwigo/ImageMagick still struggles with 20 megapixel images from my DSLR and is still not making good use of the CPU. Now I see a high percentage of iowait which iostat told me was mostly due to I/O on /tmp. The files are not visible using ls in the usual way but can be seen by looking at /proc/<pid>/fd. I've been experimenting with changes to the ImageMagick policy.xml to alter the memory parameters. I think I have improved things but generating thumbnails for DSLR portrait oriented JPEGs is still massively slow. I'm not sure why portrait images seem to be a problem when landscape images of the same size run tolarably quickly. In theory I have allowed enough memory to cache the pixels but still it is going to /tmp. More work needed...



Board footer

Powered by FluxBB

github twitter newsletter Donate © 2002-2024 · Contact