Piwigo.org

Marco61 · 2022-01-03 10:18:59

Hello/Hi/Greetings,

I am getting an error 500 when I try to update the checksums in the bulk admin screen

the first set of 20K was processed fine , the it stoped at the last 10K

Piwigo 12.2.0 Upgradecontrole
Besturingssysteem: WINNT
PHP: 7.2.12 (Toon info) [2022-01-03 12:15:36]
MySQL: 8.0.11 [2022-01-03 10:15:35]
Afbeeldingen-bibliotheek: GD bundled (2.1.0 compatible)
Cache grootte 0.00 MB direct Ververs

Geactiveerde plug-in lijst6
Admin Tools
Language Switch
LocalFiles Editor
Rotate Image
Take A Tour of Your Piwigo
VideoJS

erAck · 2022-01-03 12:01:14

HTTP status code 500 is a last resort collective Internal Server Error code. It can be anything. Check your server logs.

shooftie · 2023-11-17 10:55:59

Apologies if this is hijacking the thread, I am happy to join another or start a new one. I have been through a number of them and none seem to offer a fix:

- increased memory_limit to 2048M
- increased FastCGI to 3600s
- added request_terminate_timeout = 300 to php/www2.conf
- unchecked "combine CSS & JS" and cleared cached templates

Is it possible to "Compute missing checksums" for only a small subset of photos? I cannot see any option to only "calculate for selected photos".

Computing missing checksums in the batch manager seems to choke (HTTP 502) if I try to do this for too many items (> ~1000). http://192.168.1.140/ws.php?format=json&method=pwg.images.setMd5sum.

Nginx errors include mostly errors emanating from FastCGI during GET /admin.php?page=batch_manager&filter=prefilter-no_sync_md5sum

- Undefined array key "search" in /.../file.batch_manager_global.tpl.php
- Trying to access array offset on value of type null in /.../file.batch_manager_global.tpl.php

I am also getting a lot of 504s during syncing. I have had more success by ignoring all directories and then progressively un-ignoring them.

I am running Piwigo inside Docker, if it means anything.

Environment

Code:

Piwigo 13.8.0
Installed on 4 November 2023
Operating system: Linux
PHP: 8.1.22 [2023-11-16 23:04:05]
MySQL: 11.1.2-MariaDB-1:11.1.2+maria~ubu2204 [2023-11-16 23:04:05]
Graphics Library: ImageMagick 7.1.0-62
Cache size 25.91 MB

PHP

Code:

Admin Tools
Check Uploads
Community
Language Switch
LocalFiles Editor
Read Metadata

It seems that I have some knowledge around this but clearly not enough. Any guidance would be of great help to me and I am sure to others, too.

erAck · 2023-11-17 12:44:15

If you already apologize for hijacking a thread then why do you even do it, instead of creating a new topic?

Anyway.. HTML code 504 Gateway Timeout and 502 Bad Gateway are server errors, you have an internal network problem, nothing Piwigo could do about.

shooftie · 2023-11-18 02:46:48

Apologies if this is hijacking the thread

Now I know. Thanks for your help.

shooftie · 2023-11-20 22:06:21

@Marco61, I know that this is an old thread but it has been indexed by Google, which is how I found it. Maybe this next bit will help another lonely traveller, even if you have arrived!

The failure was happening on /ws.php?format=json&method=pwg.images.setMd5sum.

I increased all of the limits I could find across nginx.conf, php-fpm.ini, php-fpm.ini:

- fastcgi_read_timeout
- max_execution_time
- memory_limit
- request_terminate_timeout
- pm.max_children
- pm.start_servers
- pm.min_spare_servers
- pm.max_spare_servers
- pm.max_requests

Eventually, instead of just timeouts, I got a worthwhile log entry (I assume from php-fpm...)

Code:

WARNING: [pool www] child 298, script '/app/www/public/ws.php' (request: "POST /ws.php?format=json&method=pwg.images.setMd5sum") execution timed out (3834.825314 sec), terminating

So I switched from using the frontend to interacting with the API directly to minimise the noise and the same thing was happening. However, replicating the payload that was being sent from the UI required me to pass a block_size in the payload

Code:

> POST /ws.php?format=json&method=pwg.images.setMd5sum HTTP/1.1
> Host: 192.168.XXX.XXX:80
> Cookie: pwg_id=7ogtsa[...]jrgjhuf
> Content-Type: multipart/form-data; boundary=X-INSOMNIA-BOUNDARY
> User-Agent: insomnia/8.3.0
> Accept: */*
> Content-Length: 215

| --X-INSOMNIA-BOUNDARY
| Content-Disposition: form-data; name="pwg_token"
| 4072a57[...]8f7cc8
| --X-INSOMNIA-BOUNDARY
| Content-Disposition: form-data; name="block_size"
| 1000
| --X-INSOMNIA-BOUNDARY--

Furthermore, it turns out that by reducing the block_size property in the payload from 1000 to 100, everything started working again, I mean, I am having to do it in batches of 100 but at least it is not timing out.

For anyone still reading...

block_size is being calculated on the front end:

Code:

<span id="md5sum_to_add" data-origin="62089">62089</span>

jQuery('#sync_md5sum').click(function(e) {
  ...
  var addBlockSize = Math.min(
    Number((jQuery('#md5sum_to_add').data('origin') / 2).toFixed()),
    1000
  );
  add_md5sum_block(addBlockSize);
  return false;
});

...which is where the mysterious 1000 comes from!

Once the API request lands:

- setMd5sum method is called from ws.php
- get_photos_no_md5sum retrieves the images without checksums from the DB
- an array of length 'block_size' is then sliced off the array of retrieved images
- add_md5sum then generates the checksum and updates the DB by means of mass_updates

As far as I can see, there is no logging along the way and I haven't written a line of PHP in over a decade so I am probably at the end of my journey with this one.

If I manage to find out anything more then I will report back here.

shooftie · 2023-11-24 10:50:26

So I have now managed to generate checksums for all of the images in my DB (approx 60,000). I did so manually i.e. calling the API directly and not through the UI. I used Insomnia to make the calls, it has a handy feature whereby you can set a "re-call interval". I just told it to make the same call every 30 mins and then went to bed!

As I have already deduced this is a combination of factors:

- the timeouts in the server config needed increasing (to frankly unacceptable values)
- the resources of the server needed increasing

I do however, think that it is a bit disingenuous to dismiss this as

...nothing Piwigo could do about.

as this problem was compounded by the fact that the block size is set by reading the value from the DOM, rather than by allowing this value to be configured.

As soon as I get a chance I will submit a PR.

Piwigo.org

Announcement

#1 2022-01-03 10:18:59

Checksum update Error 500

#2 2022-01-03 12:01:14

Re: Checksum update Error 500

#3 2023-11-17 10:55:59

Re: Checksum update Error 500

Code:

Code:

#4 2023-11-17 12:44:15

Re: Checksum update Error 500

#5 2023-11-18 02:46:48

Re: Checksum update Error 500

#6 2023-11-20 22:06:21

Re: Checksum update Error 500

Code:

Code:

Code:

#7 2023-11-24 10:50:26

Re: Checksum update Error 500

Board footer