Announcement

#1 2022-12-19 13:07:44

plg
Piwigo Team
Nantes, France, Europe
2002-04-05
13634

bug in update to 13.3.0 between 8th and 15 december 2022

Several users have reported the same problem: the "upload" directory has been deleted after an upgrade to Piwigo 13.3.0. We now have quite a clear view about what happened. Here is the timeline:

[November 28th 2022] version 13.3.0 is released, with the archive to upgrade from 13.x to 13.3.0

[December 8th 2022] erAck, very active contributor, tells me this upgrade archive is not complete, see [Github] Piwigo issue #1818 I release a new archive by using the described procedure.

[December 11th 2022] YannB (on the French forum) tells us some photos have been deleted from his "upload" directory. At this time I think it's a one-time issue, not linked to the upgrade itself. I don't investigate any further.

[December 15th 2022] other users boge90 and FaBay89 report the same problem on [Github] Piwigo issue #1827

[December 15th 2022, still] I don't believe in coincidence. This issue becomes my top priority. I work all the afternoon with ddtddt to investigate and find solutions. We quickly find the problem (in the upgrade archive, we find an "obsolete.list" file with a single line "upload" in it, which leads to the automatic deletion of directory "upload" during upgrade). I can't reproduce the creation of this obsolete.list file and I know the upgrade archive has not been hacked. I (plg) have created this file (more precisely the script that generate the upgrade archive, but it's me who wrote/run this script) but I can't understand why yet.

We immediately update the archive to remove this obsolete.list file. All Piwigo installations who automatically updated a 13.x version to 13.3.0 between December 8th and 15th are concerned. Drops of sweat bead on my forehead.

We change the upgrade archive generator so that this problem can't happen ever again. We also think about the most efficient way to warn concerned users. Happily the vast majority of upgrades to 13.3.0 occured before December 8th.

I enhanced [extension by plg] Check Uploads so that it can check every photo listed in the database exists on the file sytem.

[December 16th 2022] I implement an automatic check on a sample of photos to make sure they're available on the filesystem, see [Github] Piwigo commit 97b665d8 if you like details. If any photo is missing you get a huge warning in the administration panel, asking to find details with the Check Uploads plugin.

Because we care a lot about privacy, we don't know who did the upgrade and when. It would have been very nice to have a list of email address and be able to contact them directly. But we can't do that, because of privacy.

Piwigo 13.4.0 is released. Of course, I tripled check the upgrade archive.

[December 19th 2022] I think I have an explanation for the arrival of the "upload" line in the obsolete.list file of the upgrade archive. When I create these upgrade archives, I create a /tmp/13.3.0 directory on my MacOS (I make it explicit because it's important). Then I extract files from piwigo-13.2.0.zip and piwigo-13.3.0.zip and run the create_patch.pl script to create the upgrade archive. The script compares directory 1320 and 1330: if a file is new or modified, it's added in the upgrade archive, if a file is not in 1330 it is added to the obsolete.list file.

Why would the "upload" be gone from the "1330" directory? It seems that MacOS, once a day, automatically removes all empty directories older than X days. My guess is that I may have used the same unzipped 1330 directory from November 28th 2022 to regenerate the patch on December 8th 2022. Due to automatic removal by MacOS, the "upload" directory was no longer there. Please note that I don't remember having never regenerate an upgrade archive. The lack of "1330/upload" was considered as an obsolete directory, thus to remove :-/

This whole situation "sucks". I obviously did not want it to happen and I transparently provide here the details. What I deeply hope is that concerned users, and all users in general, follow our recommendations:

upgrade guide wrote:

Always have a backup of your database and files. The best is to have them made automatically on a regular basis. If anything bad happens during the update, you would be able to retore a backup.

You see here it's a combination of "worst case scenario". If the bug had been obvious I would have detected it during my upgrade test even before releasing it (because yes, all the upgrade archives are tested). Maybe I should have created a version 13.4.0 on December 8th instead of updating the archive, but I didn't see the point since there was absolutely no change compared to 13.3.0.

We have improved the checks during upgrade archive creation and we're also going to put all upgrades (and plugins install/upgrade/deletion) in a specific history. This way will be able to provide this information on the maintenance page.

Offline

 

#2 2022-12-19 22:09:17

plg
Piwigo Team
Nantes, France, Europe
2002-04-05
13634

Re: bug in update to 13.3.0 between 8th and 15 december 2022

In version 13.b of [extension by plg] Check Uploads I have implemented a system to restore missing files from the multiple size cache, if any.

Offline

 

#3 2022-12-20 15:26:12

gueba
Member
2021-12-31
20

Re: bug in update to 13.3.0 between 8th and 15 december 2022

Thank's for making this clear. Shit happens and everyone who has no backup should blame himself first.

Offline

 

#4 2022-12-20 18:17:17

plg
Piwigo Team
Nantes, France, Europe
2002-04-05
13634

Re: bug in update to 13.3.0 between 8th and 15 december 2022

gueba wrote:

everyone who has no backup should blame himself first.

This is not the way I want my earlier message to be understood but please, have backups. Here we encountered a bug in a web application, but tomorrow it could be a hack of the server, a fire in the datacenter, a failure on the hardware. Anything. Have backups.

Offline

 

#5 2022-12-21 13:11:19

deemon
Member
2021-05-03
41

Re: bug in update to 13.3.0 between 8th and 15 december 2022

"Some photos are missing from your file system. Details provided by plugin Check Uploads".

OMG... got hit hard. Especially that I didn't notice the problem up until the big red message appeared in the admin page after upgrading to 13.4.0.

8 unexpected files : delete them all
121138 restorable photos from sizes cache. : Restore them all
121192 files are listed in the database, but not available in the filesystem

... the 8 unexpected files I assume are those:
buffer/0b4af4011a956b0b77dcfb0fac6735d6.part is not in the database
buffer/97ca5a8faa747301eb088560a4403a8e.part is not in the database
...
so those 8 are safe to delete?

Anyway, lets say I get my pre-nuke "backup-uploads" folder back from backup admin, I can just
`mv -n /backup-uploads/* /var/www/html/galerii/upload`
or
`rsync -a /backup-uploads/ /var/www/html/galerii/upload`
them back into the current uploads folder and everything will be fine?

Last edited by deemon (2022-12-21 13:45:24)

Offline

 

#6 2022-12-21 15:00:00

erAck
Only trying to help
2015-09-06
1573

Re: bug in update to 13.3.0 between 8th and 15 december 2022

Probably yes. The *.part files are incomplete uploads.


Running Piwigo at https://erack.net/gallery/

Offline

 

#7 2022-12-21 18:35:19

plg
Piwigo Team
Nantes, France, Europe
2002-04-05
13634

Re: bug in update to 13.3.0 between 8th and 15 december 2022

deemon wrote:

OMG... got hit hard. Especially that I didn't notice the problem up until the big red message appeared in the admin page after upgrading to 13.4.0.

This bug is very vicious because you may not realize it before a long time, if your multiple sizes are already in the cache. That's why I didn't see it with my own test of update :-/

We assumed that users having updated to 13.3.0 would also update to 13.4.0. This is why we implemented the big warning in 13.4.0 and released it as soon as possible.

I'm happy you did see the warning and understood what you had to do then.

deemon wrote:

Anyway, lets say I get my pre-nuke "backup-uploads" folder back from backup admin, I can just `rsync -a /backup-uploads/ /var/www/html/galerii/upload`
them back into the current uploads folder and everything will be fine?

rsync is perfectly fine. I don't exactly know what's inside your /backup-uploads folder, but I assume you know how to use rsync ;-)

Offline

 

#8 2022-12-21 18:39:13

plg
Piwigo Team
Nantes, France, Europe
2002-04-05
13634

Re: bug in update to 13.3.0 between 8th and 15 december 2022

For your information, I made a change in Piwigo for version 13.5.0 : file obsolete.list will no longer be taken into account during minor updates. See [Github] Piwigo commit 00d5c877 for details

Offline

 

#9 2022-12-21 23:37:46

deemon
Member
2021-05-03
41

Re: bug in update to 13.3.0 between 8th and 15 december 2022

Got my files back from actual backup *PHEW*. And the big red warning disappeared. However. When I use the "check uploads" plugin, I have some extra files now. Should/could I just delete them?

27 unexpected files delete them all
2017/11/16/20171116141552-6a570fb0.jpg is not in the database
2019/01/23/20190123225236-e4b3e4a0.jpg is not in the database
2018/06/28/20180628155610-b17db9f9.jpg is not in the database
2018/04/10/20180410173124-9d115f2b.jpg is not in the database
2018/04/10/20180410173102-3fce06fa.jpg is not in the database
...

and several .part files in buffer (and couple of actual non-.part pictures in buffer, from year 2018 and 2017...), which I assume I clearly can delete.

EDIT: Ah yes. did some exploration and seems all of those files have some form of defect or error in them. Some of them Chrome browser was able to open as picture, while every other program in my computer (Firefox, Gwenview, Gimp, Okular) simply failed either with error or just showing blank black picture. So I assume they got uploaded, but since Piwigo failed to process the file (because of some defect in the file) it got just left there and never added to the album and database.

Last edited by deemon (2022-12-22 00:13:22)

Offline

 

#10 2022-12-22 10:50:34

plg
Piwigo Team
Nantes, France, Europe
2002-04-05
13634

Re: bug in update to 13.3.0 between 8th and 15 december 2022

deemon wrote:

but since Piwigo failed to process the file (because of some defect in the file) it got just left there and never added to the album and database.

OR... Piwigo has added them and they appeared buggy at display and you did delete them. They dissapeared from your "upload" directory but not from your backup.

Offline

 

#11 2022-12-24 12:12:48

rjpp
Member
2021-07-13
21

Re: bug in update to 13.3.0 between 8th and 15 december 2022

Just a "thinking out loud" moment but in relation to your "don't know who installed it and when", what about amending the Environment data to say as much?

Currently mine says
Piwigo 13.3.0 Check for upgrade
Installed on 10 July 2021, 1 year 5 months 1 week 6 days ago
(and obviously done before Dec 8th)

But that's obviously when the original installation was performed, not the update (or any update for that matter!) Not sure what doing so would prove or help with, hence just thinking out loud.

On the subject of updates pre-Dec 8th, I don't actually have anything in Upload at all (apart from a htm file that's dated the original install date of 10 July 2021) - I add photos via FTP and use the Dashboard local sync. Does that help narrow anything down?

Offline

 

#12 2022-12-24 13:23:50

plg
Piwigo Team
Nantes, France, Europe
2002-04-05
13634

Re: bug in update to 13.3.0 between 8th and 15 december 2022

rjpp wrote:

Just a "thinking out loud" moment but in relation to your "don't know who installed it and when", what about amending the Environment data to say as much?

Currently mine says
Piwigo 13.3.0 Check for upgrade
Installed on 10 July 2021, 1 year 5 months 1 week 6 days ago
(and obviously done before Dec 8th)

But that's obviously when the original installation was performed, not the update (or any update for that matter!)

During the last days, I've been working on registering several "system" activities (in the "piwigo_activity" table). Including minor updates, of course. For now, it's just registered. Later it will be displayed somewhere.

database wrote:

MariaDB [tmp_pwgmig]> select activity_id,object,object_id,action,performed_by,occured_on,details from piwigo_activity where object='system'\G
*************************** 1. row ***************************
activity_id: 9
      object: system
   object_id: 1
      action: update
performed_by: 1
  occured_on: 2022-12-22 12:01:13
     details: a:3:{s:12:"from_version";s:6:"13.2.0";s:10:"to_version";s:6:"13.4.0";s:6:"script";s:13:"admin/updates";}

We're going to have : core updates, plugin install/activate/update/deactivate/delete/restore, theme install/activate/set_default/deactivate/delete, maintenance actions, configuration settings saved.

I think this information will be useful to what has been done (and by who, if you're working in team) and when. Just like actions on content (photo/albums/users/groups/tags...)

rjpp wrote:

On the subject of updates pre-Dec 8th, I don't actually have anything in Upload at all (apart from a htm file that's dated the original install date of 10 July 2021) - I add photos via FTP and use the Dashboard local sync. Does that help narrow anything down?

If you use "sync" to add photos, the bug does not impact you at all. That's also why we didn't want to scare everybody for nothing.

Offline

 

#13 2022-12-24 20:09:42

plg
Piwigo Team
Nantes, France, Europe
2002-04-05
13634

Re: bug in update to 13.3.0 between 8th and 15 december 2022

plg wrote:

During the last days, I've been working on registering several "system" activities (in the "piwigo_activity" table). Including minor updates, of course. For now, it's just registered. Later it will be displayed somewhere.

See [Github] Piwigo issue #1841 and this commit [Github] Piwigo commit bb692692

Offline

 

Board footer

Powered by FluxBB

github twitter newsletter Donate Piwigo.org © 2002-2023 · Contact