Thanks for the update. I’ve run into some of the same issues a while ago. I asked an AI of my choice back then, tried the suggested changes – and they worked really well for me.

So here are a few things I ended up doing:

1. MySQL limits: max_allowed_packet and related

I had recurring errors like:

Got a packet bigger than 'max_allowed_packet' bytes
MySQL server has gone away

This happened especially during large thumbnail/poster batches. I increased a few limits in my.cnf like this:

[mysqld]
max_allowed_packet = 512M
group_concat_max_len = 8096
innodb_buffer_pool_size = 1024M
tmp_table_size = 256M
max_heap_table_size = 256M
wait_timeout = 28800
interactive_timeout = 28800

No more packet-size crashes since then.

Especially when running full album scans or batch jobs across a large gallery, it's really important to raise these limits – otherwise things can silently fail due to memory or buffer restrictions.

Just to clarify: I already ran into these problems before AlbumPilot even existed, using regular Piwigo core features like metadata refresh, global sync, or batch thumbnail generation. So this isn't plugin-specific – it's a general issue with memory limits in large galleries.

2. category_id overflow: column type adjustment

Just as a side note for others with large installations: I had more than 16,000 albums at some point and started getting errors like:

Out of range value for column 'category_id'

Turns out that piwigo_history.category_id is defined as MEDIUMINT UNSIGNED by default .I changed it to INT UNSIGNED using "ALTER TABLE piwigo_history MODIFY category_id INT UNSIGNED"

That solved the issue completely. I’ve done similar changes in other places too, where Piwigo’s defaults were too tight for my use case.

Finally: 3. AlbumPilot v0.3.6 released

I’ve just uploaded version 0.3.6 of AlbumPilot (https://github.com/HendrikSchoettle/Alb … tag/v0.3.6). It includes a major internal fix in the poster generation logic, which previously caused a lot of processing errors for me. This version seems to have resolved all of them.

It took a good amount of time to track that down and fix it, so further feature updates will probably pause for a few days. But I hope the improvements are noticeable.

That should be it for new versions for the next few days... Hopefully this release holds up well and gives you a chance to properly test things.

The code is now running and working, but of course there’s still room to clean it up and make it leaner. I’ll hopefully be able to do that – just not today... For now I’m happy that it’s stable and usable (hopefully).

If you get a chance to try it out and it works for you, I’d really appreciate a short confirmation. If things are stable, I’ll upload it to the official plugin directory soon.

Thanks for testing and giving feedback so far!

Last edited by rw22mhhs (2025-06-18 00:56:05)

I know I said I’d hold off on new releases for a bit, but I’ve slipped in two very minor updates, v0.3.7 and now v0.3.8, solely to address a few edge-case bugs. There aren’t any significant new features right now, but I preferred to fix those issues promptly rather than let you run into unnecessary issues. Please give v0.3.8 a try. I’ll continue to roll out small fixes as they come up. Thanks for your understanding!

Okay, that sounds odd.

I had hoped that this issue had been resolved from version 0.3.7 onwards. In version 0.3.6, I had the issue that image sizes were handled incorrectly if they matched one of the predefined thumbnail sizes. If you can provide me with the image sizes and the thumbnail sizes of the processed images, that may help, but that's just my first guess. However, this issue should no longer occur with version 0.3.8. Let me think about some debugging code to get more information on open issues like that one.

I will come back with debugging code shortly.

Can this be narrowed down to specific images that are processed repeatedly? Or is it just a random set of images?

I still encounter the same issue initially, but in my case, it was limited to a set of images that I already knew were broken. But this affected rather a handful of images instead of several dozens or thousands. But I will cross-check once more to ensure that these are indeed broken image files.

By the way, there is another issue when updating metadata from large video files, which causes the plug-in to hang. I’ll be working on that as well.

I presume this issue doesn't appear if you run the official batch manager, right?

Ok, I seem to have multiple issues.

Looking at some of the images based on the filenames appearing some of these go back to an issue I had in 2015 with corrupted images. I had a NAS issue that messed up a bunch of files and I lost the original (full size) images and had to recover using Piwigio's resized versions. As a result, I have a bunch of images where the resolution in the database (height/width) don't match the actual file (which is much smaller). This seems to be impacting the generation of the medium size image.

For example, I had a photo that originally looked like this:

IMG_7429.jpg
5616×3744 pixels, 9.17 MB
JPG file type

Which showed an (!) icon when viewing the medium size in my gallery. I re-uploaded the file, which then properly updated the database:

IMG_7429.jpg
640×427 pixels, 0.09 MB
JPG file type

And then the resizes were generated correctly. I probably have a LOT of images still in this state and I'd rather not have to manually re-upload each one to get the database correct. Is there any way you know of I'd be able to correct this in some sort of automatic fashion?

The second problem might be related to special characters in image filenames. The other images I see being regenerated over and over (that don't fall into the corrupted category I described above) seem to have a space, or parentheis (or both) in the original filename, like:

Tampa Bay Lightning 2014-2015 Highlights .jpg
P6050005 (2).JPG
mu-vs-uc-mumb-tv (8).jpg
2020-10-24 Rays win game 4 in crazy fashion (from first base).jpg
xmas-1987 (4).jpg

Some of those (like the Lightning and Rays ones above) are actually thumbnails for videos (in the pwg_representative subfolder) so I'm not sure why Step 3 is even looking at those.

Hope that provides some info ...

Just one more thing - does the same happen when you create the thumbnails with the Piwigo core function? Maybe you can try that one as well and let me know. Thanks!

Doing some more testing, it does seem like having spaces in the filename of images is causing the plugin to constantly regenerate the thumbnails (even if they exist already). So I'm guessing there's something in the code that is not "matching" the name in the database table and the existing file on disk and thinks the thumbnail needs to be generated.

Possible feature request (not sure if it makes sense). When I sync, I sometimes set the "Who can see these photos?" option (I have some non-public albums). Now, after syncing with AlbumPilot I could go into the caddie and then manually mark the required photos with the proper permissions, but would it make sense to allow this setting to be exposed in the AlbumPilot UI?

Finally, I tried to contact you (via "send e-mail link" on this forum) about re-using some of your album selector code in my own plugin. Did you get that? Just want to make sure I am doing attribution correctly before I release a new version.

Thanks Hendrik!

rw22mhhs wrote:

About the filename spacing, does the repeated thumbnail regeneration also happen when using Piwigo's core functions, for example the batch manager? As said, I had similar issues with spaces in filenames and even tried to loosen the sync_chars_regex a while ago, but that led to other issues, so I reverted it again. And regarding the files reported as missing during the metadata scan - does this also appear if you run step 1 (scan for new files) first? It would be great to know if this is limited to AlbumPilot or a general thing in Piwigo.

With regards to filenames with spaces: upon further testing, if I use Piwigo core to generate the resizes, it works, and then AlbumPilot stops trying to regenerate. So it does seem to be something with AlbumPilot.

For the metadata scan: in my test gallery, yes, even though I am running Step 1, the same few images keep appearing in (now) Step 2 (formerly Step 5). Although running Step 1 shouldn't matter since these aren't new files, they've always been in the gallery from before I started testing AlbumPilot:

2. Import metadata from image files into database (slow!)
Searching for images to update metadata...
Total images to process: 155
File missing for ID 434 (./galleries/exp_PG_test/test2.jpg) – file not found
File missing for ID 1397 (./galleries/places/conn/IMG_8501-20250616172150.JPG) – file not found
File missing for ID 361 (./galleries/places/testing2/v221_empty2-test/batch-test-02-20170612225629-20240424200424.png) – file not found
File missing for ID 1316 (./upload/2024/02/26/20240226181040-1e65cd65.heic) – file not found
Step completed: 2. Import metadata from image files into database (slow!) for 155 images.

If I use batch manager (or the individual admin pages) to sync the metadata for those image, it seems to work just fine. The images definitely exist, but like you said this is clearly some sort of weird edge case and maybe not worth pursuing (unless we can get other people to test and see if they encounter a similar problem so there's more detail to work from).

I should probably be logging issues in Github for you for this kind of stuff instead of long forum posts (and maybe once you "officially" release I can do that ;-)), but two other things:

- label consistency suggestion: for Step 3, should the "Overwrite existing poster (if available)" instead be "Overwrite existing poster (if present)" to be consistent with the wording on Step 4?

- speaking of "Overwrite existing thumbnails (if present)" on Step 4: is it expected that turning on this checkbox should effectively regenerate ALL resizes (for the selected albums)? Because on my test site, with that checkbox on, AlbumPilot doesn't regenerate any thumbnails (now that I have everything actually generated). Should it be instead re-creating all thumbnails for my 155 images?

I always wanted to stay close to Piwigo's built-in logic. So I am generally avoiding deep workarounds for things that should be handled at the system level.

Totally understand and agree!

Let me know if you think the plugin is in good shape now to be added to the official Piwigo store. I would probably do a little cleanup first. Right now the codebase is still a bit messy, but functional.

I think you're probably good for an initial "official" release. One quick code cleanup thing I noticed. In include/images.php, there are two calls to abortOnRootNoSubs in quick succession (starting on line 56). Is that a copy/paste typo? Seems like only one call should be necessary?

    // If root album selected but "search in subalbums" is OFF, abort without scanning
    abortOnRootNoSubs($albumId, $includeSubalbums, $log);

    include_once(PHPWG_ROOT_PATH . 'include/derivative.inc.php');
    include_once(PHPWG_ROOT_PATH . 'include/derivative_params.inc.php');

    // If root album selected but "search in subalbums" is OFF, abort without scanning
    abortOnRootNoSubs($albumId, $includeSubalbums, $log);

Not sure if this is expected behavior: I ran a scan for Step 4 to regenerate thumbnails but turned on simulation mode expecting it to just list the number of thumbnails to be generated but not actually do anything. But it actually went through each thumbnail so it took just as long as a regular run would:

4. Generate thumbnails
Scanning for missing thumbnails...
Total thumbnails to generate: 367
thumbnails 47 of 367 (12%) – Image ID 64213 (Simulation) – Type: thumb
File: 2022-09-19 timelapse drive home.jpg

Notice it does show "simulation" next to the image id, maybe it should be skipping the loop through all the individual files to speed up simulation mode?

Last edited by windracer (2025-06-26 04:10:35)