Announcement

#1 2017-12-24 17:50:51

afmoene
Member
2017-12-24
4

number of categories above 65536: why and solution?

Dear all,

While using synchonization to add photos to my album I ran into a similar error message as given in http://piwigo.org/forum/viewtopic.php?id=17284. So my addition of photos failed. In the aforementioned post the reason for this error was given to be the fact that the type used for the category index is smallint(5) unsigned (which can old numbers up to 65535).

Hence I used the proposed solution to upgrade all uses of cat_id to mediumint unsigned. I realised that this fix was given for Piwigo 2.1, not my current version 2.9.2, but I tend to be brave, or stupid (and have a backup of my database ;). The fix solved my problem of not being able to add new photos to my site. But then the problems occurred elsewhere: the new albums do not show up in the gallery, and setting the permissions to a new value does not persist.

My questions are the following:
* In which other tables should I change the number type of cat_id ?
* Why does the category counter increase to such large values? I have a about 32000 photos and a few hundred albums (one folder per upload per camera). It seems as if the cat_id is increased with each photo, or even worse.

Best regards,
Arnold

PS: I really enjoy piwigo and it is at the heart of the photo management of my family!

Piwigo version: 2.9.2
PHP version: 5.6.30-0+deb8u1
MySQL version: 5.5.57-0+deb8u1

Last edited by afmoene (2017-12-24 17:51:51)

Offline

 

#2 2017-12-24 23:42:39

afmoene
Member
2017-12-24
4

Re: number of categories above 65536: why and solution?

I worked on this a little bit more myself, browsing through the database using phpmyadmin. I've looked for all fields in the database that related to categories and where of type smallint(5) unsigned. I found:

piwigo_categories: id, id_uppercat
piwigo_category_filters: category_id
piwigo_group_access: cat_id
piwigo_history: category_id
piwigo_image_category: category_id
piwigo_old_permalinks: cat_id
piwigo_phy2virt_categories: phy_category_id, virt_category_id
piwigo_user_access: cat_id
piwigo_user_cache_categories: cat_id

Each of them I've upgraded to mediumint(8) unsigned. This seemed to do the trick.

However, I still don't understand why the category count is so high, given the fact that I have only 500+ albums in my gallery.

Thanks in advance, Arnold

Offline

 

#3 2017-12-27 11:57:53

afmoene
Member
2017-12-24
4

Re: number of categories above 65536: why and solution?

Apparently the above 'solution' was not really a solution. Now the number of photos in some albums keeps on growing on each synchronization (I do the synchronization automatically every night). So most of my gallery keeps on working as it should, but the albums that were added after my changes to the variable type given above have problems.

Until I understand why/how the number of categories grows so fast (apparently as fast as the number of photos or even more) I will simply start again with a clean gallery and synchronize with my folders and photos on disk to rebuild my gallery (now I'm glad that all my meta-information, including tags, is kept in the photo files).

To be continued.
Arnold

Offline

 

#4 2017-12-28 15:29:54

plg
Piwigo Team
Nantes, France, Europe
2002-04-05
13101

Re: number of categories above 65536: why and solution?

Hi Arnold,

Needing more than 65k albums is a very uncommon need. I'm not saying it's not relevant, just that it is very uncommon. For example, on Piwigo.com (which is where we can make some stats), the biggest Piwigo, on albums, is "only" 12k albums. 5 times less than the technical limit.

afmoene wrote:

* Why does the category counter increase to such large values? I have a about 32000 photos and a few hundred albums (one folder per upload per camera). It seems as if the cat_id is increased with each photo, or even worse.

This is absolutely not normal. This is the problem to fix.

How many real directories do you have in your "galleries" directory? you can perform a:

Code:

find galleries -type d | wc -l

... if you have a command line access.


Latest blog post (November 28th 2017) Server failures in october and november 2017

Offline

 

#5 2017-12-29 16:15:43

afmoene
Member
2017-12-24
4

Re: number of categories above 65536: why and solution?

Thanks for your reply.

The output of the 'find' command is 481. So the number of folders is within 'normal' limits.

What may be the cause for the excessive number of albums/categories is the way I supply the photos to Piwigo.
* I have the photos on a windows share (samba) so that family members can dump their photos there using their windows PC.
* From within the galleries dictory of Piwigo I make symbolic links to the folders and files on that samba-share. The script is run every night and is rather brute force (one of the things it needs to do is to remove forbidden characters from the folder names and file names).

What may happen is that under certain circumstances the script *recreates* folders and links. If Piwigo looks at the time of creation or modification of folders or links, this might lead to the recreation of albums? I don't know enough (if anything) about the internals of Piwigo to judge this. If it would help: I have quite a number of database backups in which I could trace how the ID's of certain albums have changed (and if it is linked to addition of new photo's or to other events).

So to conclude: for now I assume that this issue is specific to the way I supply my photos to Piwigo, and not to any issue in Piwigo.

With kind regards, Arnold

Offline

 

#6 2017-12-31 07:51:35

birdlives
Member
2017-08-11
2

Re: number of categories above 65536: why and solution?

Sounds like you had a circular symbolic link. Like if you had a symbolic link point to folder "dir" inside of that same folder. You need to tell find to follow links. Try find -L -type d. Don't pipe it to wc because if you do have an endless loop, it's going to keep counting the number of lines until it ends. Which could be never. Just see if you see directories repeating themselves within themselves.

During physical update, piwigo compares the full paths or categories. It doesn't take into consideration modification dates of folders or files even, when adding or deleting.

You should look over your script. Sometimes an oddly named folder or file can throw it off.

But there are other reasons it could happen. I've found certain unicode characters seem to slip past the valid filename check that piwigo does and really messes with the INSERT statement that piwigo does during sync.

Or configuring piwigo to accept spaces during sync, and then syncing files with spaces in their names, also messes things up.

Ultimately though, you should rebuild your database. Or look through your backups using something like mysql-workbench and see if you can identify at which update it went wrong. You would probably want to look at the table piwigo_categories and see at which one of your backups did it go wrong.

Last edited by birdlives (2017-12-31 07:57:42)

Offline

 

Board footer

Powered by FluxBB

github twitter facebook google+ newsletter Donate Piwigo.org © 2002-2018 · Contact