🌍
English
Announcement
- [2024-04-17] Piwigo 14.4.0
- [2024-04-01] Piwigo in Hobbit runes
- [2024-03-01] Piwigo 14.3.0
- [2024-01-30] Piwigo 14.2.0
- [2023-12-29] Piwigo 14.1.0
#1 2021-02-27 11:11:39
- PBENZ404
- Member
- 2021-02-01
- 15
Can't synchronise directory listing
Hi,
I have many albums appearing in synchronise directory listing that do not exist. The gallery folder structure of actual data looks something like this :
Samsung_pics
├── Samsung_pics
│ ├── 01.02.2021
│ ├── 02.02.2021
│ ├── 03.02.2021
│ ├── 04.02.2021
│ ├── 05.02.2021
│ ├── 06.02.2021
But in the synchronise tab the directory structure is like the following
Samsung_pics
├── Samsung_pics
│ ├── 01.02.2021
│ ├── 02.02.2021
│ ├── 03.02.2021
│ ├── 04.02.2021
│ ├── 05.02.2021
│ ├── 06.02.2021
01.02.2021
02.02.2021
03.02.2021
04.02.2021
05.02.2021
06.02.2021
01.02.2021
02.02.2021
03.02.2021
04.02.2021
05.02.2021
06.02.2021
i.e. sub-directories appear at the root folder location and their are repeats of the sub-directories, there are of course no photos in the storage directory at these phantom locations. I need to delete them.
I try to synchronise "directories only" but this fails with a 504 time out via nginx (even with an extended timeouts).
I believe the database is corrupt. How do i resynchronise the database from scratch. I'm assuming the sql is stored in /var/lib/mysql/piwigo should I delete. I see no option to delete these phantom directories.
I would appreciate any help you have to offer.
Best regards,
P404
Piwigo 11.3.0 Check for upgrade
Operating system: Linux
PHP: 7.3.19-1~deb10u1 (Show info) [2021-02-27 09:50:45]
MySQL: 5.5.5-10.3.27-MariaDB-0+deb10u1 [2021-02-27 09:50:45]
Graphics Library: ImageMagick 6.9.10-23
Offline
#2 2021-02-27 20:28:43
- windracer
- Member
- St. Pete, FL
- 2014-12-28
- 457
Re: Can't synchronise directory listing
Did you try "Repair and optimize database" or "Reinitialize integrity check" under Maintenance in the Admin area?
Offline
#3 2021-02-27 21:51:07
- PBENZ404
- Member
- 2021-02-01
- 15
Re: Can't synchronise directory listing
Thanks for the suggestion, I did try that. Optimisations completes with the message:
"All optimizations have been successfully completed."
and the integrity check completes (or at least doesn't present an error message). However, the list shown in synchronisation is not correctly updated.
Offline
#4 2021-02-27 23:11:04
- executive
- Member
- 2017-08-16
- 1214
Re: Can't synchronise directory listing
you have two directories with the exact same name. Try renaming one
Offline
#5 2021-02-28 08:05:46
- TOnin
- Member
- 1970-01-01
- 106
Re: Can't synchronise directory listing
Hi,
Does your directory tree looks the same in Tools > Synchronize > reduce to single existing albums and in Albums > Move?
Maybe you can ask your database the list of known directories named '06.02.2021':
Code:
SELECT * FROM `pwg_categories` WHERE `name` = '06.02.2021' ORDER BY `id` DESC ;
Offline
#6 2021-02-28 13:54:07
- PBENZ404
- Member
- 2021-02-01
- 15
Re: Can't synchronise directory listing
I figured out how to query the database, as you can see I have a bunch of NULL id for the phantom directories, can I delete these or will I screw up my database? I've tried to explain below with the table and some comments to the right of the database output from the MariaDB query.
Code:
+----------------------------------------------------+-------------+------+ | name | id_uppercat | id | +----------------------------------------------------+-------------+------+ | 1945 | NULL | 546 | (Truncated) | 2019 | NULL | 722 | | 01 | 722 | 723 | | 02 | 722 | 724 | | 03 | 722 | 725 | | 04 | 722 | 726 | | 05 | 722 | 727 | | 06 | 722 | 728 | | 07 | 722 | 729 | | 08 | 722 | 730 | | 09 | 722 | 731 | | 10 | 722 | 732 | | 11 | 722 | 733 | | 12 | 722 | 734 | (Truncated, things behaving sensibly before here) | iPhone Name | NULL | 949 | | iPhone | 949 | 950 | | Recents | 950 | 951 | | Test | 950 | 952 | | Videos | 950 | 953 | | Monday night | NULL | 954 | | pic downselect | NULL | 955 | | Samsung Pics | NULL | 956 | <----- Correct route directory | PNGs | NULL | 963 | | Locations | NULL | 966 | | OSM46.54008934716014 | 966 | 967 | | 01.02.2021 | NULL | 968 | | 02.02.2021 | NULL | 969 | | 03.02.2021 | NULL | 970 | | 04.02.2021 | NULL | 971 | | 05.02.2021 | NULL | 972 | | 06.02.2021 | NULL | 973 | | 07.02.2021 | NULL | 974 | | 08.02.2021 | NULL | 975 | | 09.02.2021 | NULL | 976 | | 10.02.2021 | NULL | 977 | | 23.01.2021 | NULL | 978 | | 24.01.2021 | NULL | 979 |. <----- These Nulls are incorrect | 25.01.2021 13.49 | NULL | 980 | | 26.01.2021 | NULL | 981 | | 27.01.2021 | NULL | 982 | | 28.01.2021 | NULL | 983 | | 29.01.2021 | NULL | 984 | | 30 31.01.2021 | NULL | 985 | | Camera | NULL | 986 | | Videocaptures | NULL | 987 | | Whatsapp 12345 12345 | NULL | 988 | | 01.02.2021 | NULL | 989 | | 02.02.2021 | NULL | 990 | | 03.02.2021 | NULL | 991 | | 04.02.2021 | NULL | 992 | | 05.02.2021 | NULL | 993 | | 06.02.2021 | NULL | 994 | | 07.02.2021 | NULL | 995 | | 08.02.2021 | NULL | 996 | | 09.02.2021 | NULL | 997 | | 10.02.2021 | NULL | 998 | | 23.01.2021 | NULL | 999 | | 24.01.2021 | NULL | 1000 | | 25.01.2021 13.49 | NULL | 1001 | | 26.01.2021 | NULL | 1002 | | 27.01.2021 | NULL | 1003 | | 28.01.2021 | NULL | 1004 | | 29.01.2021 | NULL | 1005 | | 30 31.01.2021 | NULL | 1006 | | Camera | NULL | 1007 | | Videocaptures | NULL | 1008 | | Whatsapp 12345 12345 | NULL | 1009 | | 01.02.2021 | NULL | 1010 | | 02.02.2021 | NULL | 1011 | | 03.02.2021 | NULL | 1012 | | 04.02.2021 | NULL | 1013 | | 05.02.2021 | NULL | 1014 | | 06.02.2021 | NULL | 1015 | | 07.02.2021 | NULL | 1016 | | 08.02.2021 | NULL | 1017 | | 09.02.2021 | NULL | 1018 | | 10.02.2021 | NULL | 1019 | | 23.01.2021 | NULL | 1020 | | 24.01.2021 | NULL | 1021 | | 25.01.2021 13.49 | NULL | 1022 | | 26.01.2021 | NULL | 1023 | | 27.01.2021 | NULL | 1024 | | 28.01.2021 | NULL | 1025 | | 29.01.2021 | NULL | 1026 | | 30 31.01.2021 | NULL | 1027 | | Camera | NULL | 1028 | | Videocaptures | NULL | 1029 | | Whatsapp 12345 12345 | NULL | 1030 | | 01.02.2021 | NULL | 1031 | | 02.02.2021 | NULL | 1032 | | 03.02.2021 | NULL | 1033 | | 04.02.2021 | NULL | 1034 | | 05.02.2021 | NULL | 1035 | | 06.02.2021 | NULL | 1036 | | 07.02.2021 | NULL | 1037 | | 08.02.2021 | NULL | 1038 | | 09.02.2021 | NULL | 1039 | | 10.02.2021 | NULL | 1040 | | 23.01.2021 | NULL | 1041 | | 24.01.2021 | NULL | 1042 | | 25.01.2021 13.49 | NULL | 1043 | | 26.01.2021 | NULL | 1044 | | 27.01.2021 | NULL | 1045 | | 28.01.2021 | NULL | 1046 | | 29.01.2021 | NULL | 1047 | | 30 31.01.2021 | NULL | 1048 | | Camera | NULL | 1049 | | Videocaptures | NULL | 1050 | | Whatsapp 12345 12345 | NULL | 1051 | | 01.02.2021 | NULL | 1052 | | 02.02.2021 | NULL | 1053 | | 03.02.2021 | NULL | 1054 | | 04.02.2021 | NULL | 1055 | | 05.02.2021 | NULL | 1056 | | 06.02.2021 | NULL | 1057 | | 07.02.2021 | NULL | 1058 | | 08.02.2021 | NULL | 1059 | | 09.02.2021 | NULL | 1060 | | 10.02.2021 | NULL | 1061 | | 23.01.2021 | NULL | 1062 | | 24.01.2021 | NULL | 1063 | | 25.01.2021 13.49 | NULL | 1064 | | 26.01.2021 | NULL | 1065 | | 27.01.2021 | NULL | 1066 | | 28.01.2021 | NULL | 1067 | | 29.01.2021 | NULL | 1068 | | 30 31.01.2021 | NULL | 1069 | | Camera | NULL | 1070 | | Videocaptures | NULL | 1071 | | Whatsapp 12345 12345 | NULL | 1072 | | 01.02.2021 | NULL | 1073 | | 02.02.2021 | NULL | 1074 | | 03.02.2021 | NULL | 1075 | | 04.02.2021 | NULL | 1076 | | 05.02.2021 | NULL | 1077 | | 06.02.2021 | NULL | 1078 | | 07.02.2021 | NULL | 1079 | | 08.02.2021 | NULL | 1080 | | 09.02.2021 | NULL | 1081 | | 10.02.2021 | NULL | 1082 | | 23.01.2021 | NULL | 1083 | | 24.01.2021 | NULL | 1084 | | 25.01.2021 13.49 | NULL | 1085 | | 26.01.2021 | NULL | 1086 | | 27.01.2021 | NULL | 1087 | | 28.01.2021 | NULL | 1088 | | 29.01.2021 | NULL | 1089 | | 30 31.01.2021 | NULL | 1090 | | Camera | NULL | 1091 | | Videocaptures | NULL | 1092 | | Whatsapp 12345 12345 | NULL | 1093 | | 01.02.2021 | NULL | 1094 | | 02.02.2021 | NULL | 1095 | | 03.02.2021 | NULL | 1096 | | 04.02.2021 | NULL | 1097 | | 05.02.2021 | NULL | 1098 | | 06.02.2021 | NULL | 1099 | | 07.02.2021 | NULL | 1100 | | 08.02.2021 | NULL | 1101 | | 09.02.2021 | NULL | 1102 | | 10.02.2021 | NULL | 1103 | | 23.01.2021 | NULL | 1104 | | 24.01.2021 | NULL | 1105 | | 25.01.2021 13.49 | NULL | 1106 | | 26.01.2021 | NULL | 1107 | | 27.01.2021 | NULL | 1108 | | 28.01.2021 | NULL | 1109 | | 29.01.2021 | NULL | 1110 | | 30 31.01.2021 | NULL | 1111 | | Camera | NULL | 1112 | | Videocaptures | NULL | 1113 | | Whatsapp 12345 12345 | NULL | 1114 | | 01.02.2021 | NULL | 1115 | | 02.02.2021 | NULL | 1116 | | 03.02.2021 | NULL | 1117 | | 04.02.2021 | NULL | 1118 | | 05.02.2021 | NULL | 1119 | | 06.02.2021 | NULL | 1120 | | 07.02.2021 | NULL | 1121 | | 08.02.2021 | NULL | 1122 | | 09.02.2021 | NULL | 1123 | | 10.02.2021 | NULL | 1124 | | 23.01.2021 | NULL | 1125 | | 24.01.2021 | NULL | 1126 | | 25.01.2021 13.49 | NULL | 1127 | | 26.01.2021 | NULL | 1128 | | 27.01.2021 | NULL | 1129 | | 28.01.2021 | NULL | 1130 | | 29.01.2021 | NULL | 1131 | | 30 31.01.2021 | NULL | 1132 | | Camera | NULL | 1133 | | Videocaptures | NULL | 1134 | | Whatsapp 12345 12345 | NULL | 1135 | | 01.02.2021 | NULL | 1136 | | 02.02.2021 | NULL | 1137 | | 03.02.2021 | NULL | 1138 | | 04.02.2021 | NULL | 1139 | | 05.02.2021 | NULL | 1140 | | 06.02.2021 | NULL | 1141 | | 07.02.2021 | NULL | 1142 | | 08.02.2021 | NULL | 1143 | | 09.02.2021 | NULL | 1144 | | 10.02.2021 | NULL | 1145 | | 23.01.2021 | NULL | 1146 | | 24.01.2021 | NULL | 1147 | | 25.01.2021 13.49 | NULL | 1148 | | 26.01.2021 | NULL | 1149 | | 27.01.2021 | NULL | 1150 | | 28.01.2021 | NULL | 1151 | | 29.01.2021 | NULL | 1152 | | 30 31.01.2021 | NULL | 1153 | | Camera | NULL | 1154 | | Videocaptures | NULL | 1155 | | Whatsapp 12345 12345 | NULL | 1156 | | 01.02.2021 | NULL | 1157 | | 02.02.2021 | NULL | 1158 | | 03.02.2021 | NULL | 1159 | | 04.02.2021 | NULL | 1160 | | 05.02.2021 | NULL | 1161 | | 06.02.2021 | NULL | 1162 | | 07.02.2021 | NULL | 1163 | | 08.02.2021 | NULL | 1164 | | 09.02.2021 | NULL | 1165 | | 10.02.2021 | NULL | 1166 | | 23.01.2021 | NULL | 1167 | | 24.01.2021 | NULL | 1168 | | 25.01.2021 13.49 | NULL | 1169 | | 26.01.2021 | NULL | 1170 | | 27.01.2021 | NULL | 1171 | | 28.01.2021 | NULL | 1172 | | 29.01.2021 | NULL | 1173 | | 30 31.01.2021 | NULL | 1174 | | Camera | NULL | 1175 | | Videocaptures | NULL | 1176 | | Whatsapp 12345 12345 | NULL | 1177 | | Samsung old | 956 | 1249 | | Camera | 1249 | 1250 | | Collage | 1249 | 1251 | | Screenshots | 1249 | 1252 | | Samsung pics | 956 | 1253 | <----- Note lower case 'p'. Correct sub-d of 956 | 01.02.2021 | 1253 | 1254 | <----- Correctly assigned to sub-d of 1253 | 02.02.2021 | 1253 | 1255 | | 03.02.2021 | 1253 | 1256 | | 04.02.2021 | 1253 | 1257 | | 05.02.2021 | 1253 | 1258 | | 06.02.2021 | 1253 | 1259 | | 07.02.2021 | 1253 | 1260 | | 08.02.2021 | 1253 | 1261 | | 09.02.2021 | 1253 | 1262 | | 10.02.2021 | 1253 | 1263 | | 23.01.2021 | 1253 | 1264 | | 24.01.2021 | 1253 | 1265 | | 25.01.2021 13.49 | 1253 | 1266 | | 26.01.2021 | 1253 | 1267 | | 27.01.2021 | 1253 | 1268 | | 28.01.2021 | 1253 | 1269 | | 29.01.2021 | 1253 | 1270 | | 30 31.01.2021 | 1253 | 1271 | | Camera | 1253 | 1272 | | Videocaptures | 1253 | 1273 | | Whatsapp 12345 12345 | 1253 | 1274 | +----------------------------------------------------+-------------+------+
Last edited by PBENZ404 (2021-02-28 14:06:53)
Offline
#7 2021-02-28 14:06:07
- PBENZ404
- Member
- 2021-02-01
- 15
Re: Can't synchronise directory listing
Dear Executive,
That could have been what started the mess. For the nested directories, one has an upper case p the other a lower case p. I can see if renaming helps.
Best regards, PBENZ404
Offline
#8 2021-02-28 16:26:22
- erAck
- Only trying to help
- 2015-09-06
- 2027
Re: Can't synchronise directory listing
id_uppercat=NULL means the album has no parent album, thus is a top level album. Make sure you only delete those that you really want to be gone. However, there may be images associated with them, where in table piwigo_images the storage_category_id field is the same as the piwigo_categories' id field, so you'd have to delete those image entries first. (and if you also assigned such image to a virtual album then removing it from table piwigo_image_category as well, by image_id). There may be other quirks lurking, I just took a quick glance.
I think the underlying cause are the two equal (ignoring case) subdirectory names, apparently that confuses Piwigo. I'd call that a bug, but it might be hard to avoid because the subdirectories are treated as independent albums. You will have to rename directories to be unique to make things work.
If you didn't spend too much time in editing titles or such then probably easiest would be to rename the directories to unique names and restart the database from scratch and sync again.
Running Piwigo at https://erack.net/gallery/
Offline
#9 2021-02-28 19:42:12
- TOnin
- Member
- 1970-01-01
- 106
Re: Can't synchronise directory listing
I succeeded in reproducing the bug.
The problem is the forbidden char space ' ' in directory name, the case is not a problem. Subdirectories can bear the same name as long as they are not in the same upperdirectory (Nothing new here). After synchronization (simulation or real) a RED warning message is shown, if something is wrong.
1 - Remove spaces from your directories and files names.
After changing the names, the phantom subdirectories will be removed after synchronizations. But you will have to perform as many sync as the max number of duplicates of a first level phantom directory name.
2 - Synchronize several times.
Let us know if this solves you problem.
Offline
#10 2021-02-28 21:59:19
- erAck
- Only trying to help
- 2015-09-06
- 2027
Re: Can't synchronise directory listing
Is that bug with, or without a space being present in $conf['sync_chars_regex'] ?
Running Piwigo at https://erack.net/gallery/
Offline
#11 2021-02-28 22:23:50
- TOnin
- Member
- 1970-01-01
- 106
Re: Can't synchronise directory listing
Here is my regexp on piwigo 11.3.0
Code:
// permitted characters for files/directories during synchronization $conf['sync_chars_regex'] = '/^[a-zA-Z0-9-_.]+$/';
And here is the error message after synchronization simulation:
[./galleries/directory_02/Pommes Poires] PWG-UPDATE-1 (wrong filename)
This directory (with space in its spelling) is rejected (not created as an album), then its subdirectories are like becoming phantom first level albums.
Offline
#12 2021-02-28 22:50:23
- PBENZ404
- Member
- 2021-02-01
- 15
Re: Can't synchronise directory listing
Thanks for the help TOnin, erAck,
I'll try what's been suggested, and report back how I get on.
Offline
#13 2021-03-01 00:12:49
- erAck
- Only trying to help
- 2015-09-06
- 2027
Re: Can't synchronise directory listing
TOnin wrote:
[./galleries/directory_02/Pommes Poires] PWG-UPDATE-1 (wrong filename)
This directory (with space in its spelling) is rejected (not created as an album), then its subdirectories are like becoming phantom first level albums.
That perfectly makes sense. Would you mind opening an issue for it at https://github.com/Piwigo/Piwigo/issues ? Thanks.
Running Piwigo at https://erack.net/gallery/
Offline
#14 2021-03-01 12:42:22
- TOnin
- Member
- 1970-01-01
- 106
Re: Can't synchronise directory listing
Sure, done :-)
[Github] Piwigo issue #1351
Offline
#15 2021-03-03 17:35:00
- PBENZ404
- Member
- 2021-02-01
- 15
Re: Can't synchronise directory listing
For whatever reason, I couldn't get the sync. process to remove the phantom directories on its own. Sync did however recognise changes when I renamed the nested 'Samsung' directories. Perhaps there is a permissions problem or another problem lurking, but for now I'm back up and running.
In the end I deleted the phantom categories in MariaDB manually, and have not noticed any immediate problems. I did this as below by searching for NULL id_uppercat directories and folders names with a certain string '%.%' that I knew was in the unwanted folders.
Trivial for SQL wizards to do, but took me a bit of time to figure out. I hope this helps out anybody who gets stuck, though I wouldn't advise messing with the database unless you have no choice:
Code:
SELECT id FROM piwigo_categories WHERE id_uppercat IS NULL AND name LIKE ‘%.%’; DELETE FROM piwigo_categories WHERE id BETWEEN X AND Y;
Offline