Announcement

#1 2015-10-29 02:35:19

JT
Member
2014-11-03
16

Quick Sync times out and gives Internal Server error 500

Hello/Hi/Greetings,

I have a couple of issues with "Quick Local Synchronization".

First, it is not very 'quick' :D

Anyway my serious complaint is that it should not delete galleries from database etc without 'asking' first.
Here is an example of a problem that you can "easily" avoid:

1. I have an NFS share mounted on the server hosting Piwigo that serves ONLY the FULL size images, the /i/ folder with caches is not on the nfs.
2. Sometimes the NFS mount fails and the galleries are then no longer 'visible to piwigo'. The gallery will still work fine, you will only "notice" anything wrong if you try and download the "original-size" image.
3. If you go to "Quick Local Synchronization" to do a sync after you added some folders on the NFS shared volume, without knowing that the piwigo server no longer has that volume mounted, then all the sudden you will get message that all those galleries "have been Deleted". Pretty scary message!

So I would propose that after clicking Quick Local Synchronization, that we are simply told what would happen and ask us to CONFIRM the deletion, or do nothing. In this way I could have noticed that something was wrong and I could have FIXED things and re-mounted the NFS share.

Also, I would vote for some way to do INCREMENTAL/DIFFS or some method to not have to RESCAN all existing galleries/ablums when doing a "QUICK LOCAL Synchronization", because especially when albums are mounted on NFS this becomes very slow and even TIMES OUT.

I am now unable to complete any Quick Local Synchronization, it keeps on timing out. Any ideas for quick/workaround solution?

thanks,
JT

Piwigo version: 2.7.x
PHP version: PHP: 5.4.36-0+deb7u3
MySQL version: 5.5.41-0+wheezy1
Graphics Library: External ImageMagick 6.7.7-10

Offline

 

#2 2015-10-29 02:48:42

JT
Member
2014-11-03
16

Re: Quick Sync times out and gives Internal Server error 500

Oh wow, maybe is worse than i thought. Seems maybe the "Quick Local Synchronization" may have deleted all the /i/ caches etc for all the galleries and also purged databases. Sucks because this means I will have to go back and rename all the albums/galleries all over again once I'm able to successfully complete a "quick local sync" again, if i'm able to do it without server timing out all the time.

Also seems i lose all my album descriptions etc, this is sad.. :(

Offline

 

#3 2015-10-29 02:51:56

JT
Member
2014-11-03
16

Re: Quick Sync times out and gives Internal Server error 500

Yep, looks like there should be a HUGE RED disclaimer, "Backup your Database" before you ever dare click on the "Quick Local Synchronization" button. :( too depressing...

Offline

 

#4 2015-10-29 03:04:24

JT
Member
2014-11-03
16

Re: Quick Sync times out and gives Internal Server error 500

Ok I do have a mysql backup/dump but from 9 months ago, but hopefully it has some of the original album 'names' and album info/descriptions. Whats the best way for me to recover such data and reimport it into piwigo?

And big WARNING to others, be VERY careful and do FULL DATABASE backup before clicking on the "Quick Local Synchronization" because if by any chance piwigo can no longer see all your galleries in the usual place, it seems to DELETE all the info about them from its database soon as you click that button.

So if you temporarily moved them and forgot to change confi/location, or forgot a symlink or a drive isn't properly mounted, or have any issue with nfs samba etc, you can end up pulling your hair out trying to fix the mess after.

Offline

 

#5 2015-10-29 11:38:31

tazattitude
Member
2015-07-13
49

Re: Quick Sync times out and gives Internal Server error 500

I have only been using Piwigo for a couple months now, but I was thinking that when you use the "Synchronize", this only synchronizes the folder/file system that you FTP'd up to the server to the database.

Where did the descriptions come from that you lost? Did you add these or were they from an import like G2, G3? Or, did you add the descriptions afterwards.
If you were importing, and you change your file system in galleries directory, of course your descriptions will not be in sync.

Also if you delete or move any folders in the ./galleries directory,  it will also remove or move the images/folders when you synchronize - it won't move the data!
It will show a message as 'deleted from the database'
It does not automatically delete them. They have to be removed via FTP for it to show up as 'deleted' in the sync.

By default, "perform a simulation only (nothing will be changed in the database)" is checked.
This does nothing but show you any errors or changes that will be made to the database (if you uncheck the above).

I always know which images/folders will be deleted already as they will be the same as the ones I deleted in the galleries directory.

Sounds like you you may be treating these as 'virtual folders' which you can move around and the descriptions remain intact

As far as timing out, I also had the same problem. I added this to my htaccess file
php_value max_execution_time xxxx (<--just increase this to what you already have)
I have 23000 images and about 1100 albums and sometimes it would go through, sometimes it would not. Haven' had a problem since the change
If this does not work, simply do have of the albums at a time.

Also, where did you see "QUICK LOCAL Synchronization"?
I looked under images>>>add
Tools >>> Synchronize
and the Synchronize 'help' Didn't see it.
of course the more images you have, the slower the Syncronzing process will be.

Yes, always backup your database. I have mine setup using cron to backup everyday.

Offline

 

#6 2015-10-30 04:03:25

JT
Member
2014-11-03
16

Re: Quick Sync times out and gives Internal Server error 500

You can't miss the "Quick Local Synchronization", at least in 2.7.1 it's pretty big visible soon as you click "Administration" link on the left side after login.

And I did not lose any photos, but what was lost should be considered due to a BUG and it SHOULD be FIXED in my opinion.

Basically I did not import from any gallery. The way my gallery works is I ftp or put galleries/albums in folder and then I run the "Quick Local Synchronization".

The problem is that if for any reason the albums inside the "galleries" folder are not there "temporarily" maybe because a drive did not mount on boot, or nfs share or samba share wasn't connected, for any reason at all, bad symlink or you temporarily moved the folders to clean up or whatever, well if you CLICK on the "Quick Local Synchronization" and the album folders are no longer visible inside the galleries directory, then PIWIGO will  IMMEDIATELY DELETE ALL YOUR ALBUMS info from its Database!

What will you lose? You will lose ALL THE CACHE/Thumbnail Images of all sizes, good luck recreating thousands or hundred thousand thumbnails and small images all over again for all your photos. It can bring your server to its knees or crash it if it tries to do too many of those conversions at once.

And not only that, but once you click that button and the album folders were no longer there, it ALSO DELETES the DESCRIPTION and TITLE of the Albums. So if you had typed in nice descriptions or a custom title for your albums it all gets deleted, gone, without asking you.

The other issue is that even if you're so LUCKY to NEVER experience an issue with NFS/samba/network drive or anything that might make your album directories temporarily unavailable when you click "Quick Local Sync", its still has one BIG problem left, that is it takes VERY LONG to index each time because it seems to try and re-scan ALL the albums and photos or something because it take forever. And as you add more and more album-folders it gets slower and slower until the browser will just TIME-OUT and give you a Internal 500 server error.

Hope that explains the issue.

I fear though that the FTP/Local Albums ie Quick Local Sync feature is not really being supported anymore. Seems no one cares about it?

Offline

 

#7 2015-10-30 11:31:33

tazattitude
Member
2015-07-13
49

Re: Quick Sync times out and gives Internal Server error 500

I found the "Quick Local Synchronization" title. It's a button on the right side. I have always gone straight to the Synchronize page.

I'm surprised though that when you synchronize with the "perform a simulation only" field checked that is still deletes info.
I know I have moved folders and re-synced and it moved it to the new location.

Like I mentioned to, why not sync half the folders, then do the other half. Or even a quarter at a time
I know at 23000 it does run slow, but the job gets done.

Sounds like you need to upload FTP, then visualize your albums. THEN when you add descriptions and titles, and move the albums, all info will go with it.

Sounds like you have a ton a photos (100,000+?), VERY surprised you haven't been backing up daily. Pretty risky. Contact your host if you have one. Maybe they have a backup of the database.

I really haven't had a problem like that.
Sorry I was no help :(

Offline

 

#8 2015-10-30 15:20:58

flop25
Piwigo Team
2006-07-06
6839

Re: Quick Sync times out and gives Internal Server error 500

there is no help to bring there unfortunatly ; you're going too sentimental there and not rational
tazattitude almost said everything

I used intensively the FTp way and still use it; it was on a free host, I had like 20k pics and was refreshing at each 500 error since I was exceeding the time limit. Never lost anything. I once had a bug duplicating my data but that was fix

"Quick Local Synchronization" does sync everything which is obviously long on 10k+ pictures because the technology, ressource etc

If you don't perform a simulation or basically you don't check before doing something which delete information permanently, we can't do much about it; I sync and backup from usb and different computers each day on my data at work , and I always perform a simulation then carefully check for any unwanted deletion
And if you don't backup...

You're just experiencing something everyone fears and hates -I deeply sympathise- but if we keep the things rational, this kind of accident can be avoided by simple measures Piwigo has (simulation, database export)


To get a better help : Politeness like Hello-A link-Your past actions precisely described
Check my extensions : more than 30 available
who I am and what I do : http://fr.gravatar.com/flop25
My gallery : an illustration of how to integrate Piwigo in your website

Offline

 

#9 2015-10-30 17:26:20

JT
Member
2014-11-03
16

Re: Quick Sync times out and gives Internal Server error 500

Flop, considering you are part of "Piwigo Team" don't you think its unprofessional to insult Piwigo users calling them sentimental and irrational? Why go name-calling on me?

As I've explained, I simply clicked on the "Quick local synchronization" button at the lower right part of the page at http://mywebsite/photogallery/admin.php

And "before" I got any kind of "options" to do anything it seemed to me that the damage was already done. As soon as you hit that button from the "admin.php" page on which there are no options to choose, piwigo starts 'thinking' and taking quite a long time if there are a lot of albums. But unfortunately for me my NFS mount had dropped and so instead of a ton of folders/albums in my "/galleries" folder, Piwigo only saw "three" album/folders. So it recognized those existing 3 folders and then told me that it "Deleted..." the rest of the albums that were no longer there due to the NFS drive mounting problem.

So what I was asking for, was for Piwigo "Not" to immediately delete any information or cache from its database if it detects certain albums have been removed, but to show us information first such as:
"Album so-and-so would be deleted, no longer detected" ...
"Album so-and-so would be deleted, no longer detected" ...
And ask us whether to allow it to proceed with the Deletion or not.

Currently, after pressing the "Quick local synchronization button", you will get message of:
Deleted .....
Deleted ....
Deleted ....
etc, for all the albums that it no longer detected in the "galleries" folder.

Currently, now that I've fixed my NFS share, I keep getting Error 500 Internal Server error because I assume there are too many albums it needs to resync and although I'm not a PHP coder, I would think it should be pretty trivial for an experienced PHP coder to program Piwigo to:
1. Prompt user for confirmation if to delete any Cache images or info from database for any album-folder it no longer detects, instead of immediately deleting upon execution of "quick local sync.."
2. Modify Quick Local Synchronization feature to give the option to show the detected albums and let the user select which ones to Sync, or simply have piwigo "split" them and iterate through a couple times instead of trying to do absolutely all of them at once and end up timing out. Maybe there's better solution, anything that can help avoid Error 500 due to server timeout.

thanks,
jt

Offline

 

#10 2015-10-30 18:10:47

flop25
Piwigo Team
2006-07-06
6839

Re: Quick Sync times out and gives Internal Server error 500

Hey I'm not a professional We don't sell anything and we mostly are not native English speakers
I notify plg who added that link But what I said is still valid I think


To get a better help : Politeness like Hello-A link-Your past actions precisely described
Check my extensions : more than 30 available
who I am and what I do : http://fr.gravatar.com/flop25
My gallery : an illustration of how to integrate Piwigo in your website

Offline

 

#11 2015-10-30 19:21:46

plg
Piwigo Team
Nantes, France, Europe
2002-04-05
13174

Re: Quick Sync times out and gives Internal Server error 500

Hi :-)

JT wrote:

Flop, considering you are part of "Piwigo Team" don't you think its unprofessional to insult Piwigo users calling them sentimental and irrational? Why go name-calling on me?

Maybe flop did not find the most appropriate words but I don't read any "insult" here :-)

So let's resume the "problem" : the "Quick local sync" (QLS) does not perform a simulation, but a real synchronization. Combined with your "unreliable" (no insult, just facts) filesystem, Piwigo considers all files have gone and sync the database (ie deletes the photos/albums from the database).

In my opinion, the solutions can be

* activate simulation on QLS (which should not be the default behavior)
* hide the QLS button


About the "incremental" synchronization, this is exactly why we let you synchronize only one album.

1) sync only folders (should be pretty fast)
2) sync files+folders on the first folder
3) sync files+folders on the second folder
...


Latest blog post (May 3rd 2018) New subscription form

Offline

 

#12 2015-10-31 07:45:18

JT
Member
2014-11-03
16

Re: Quick Sync times out and gives Internal Server error 500

flop25 wrote:

Hey I'm not a professional We don't sell anything and we mostly are not native English speakers
I notify plg who added that link But what I said is still valid I think

Hi, I thought that you sell Piwigo "hosting" on piwigo.com don't you?

But there's nothing wrong with being professional even if you're not selling anything, especially if you might want to sell something in the future.

flop25 wrote:

Maybe flop did not find the most appropriate words but I don't read any "insult" here :-)

Plg, perhaps its a "culture" difference if in France its not an insult to call someone sentimental and irrational.  Could be everyone's got real tough skin there, I mean last time I was on Champs-Élysées the pedestrians were "fair-game" for the motorists, many times they had to run/jump to the sidewalk for their life! :D not easy crossing the street there, luckily I was fine since I was a fast walker :)

Anyway thanks for the tip on the QLS (quick local synchronization) that it should be avoided, I just now started looking around for another method and found the "Synchronize" link on the menu under "Tools".

So just to confirm, if I had used the "Synchronize" button from under the Tools menu instead of the "Quick local synchronization" button (QLS), that would have properly done just a simulation?

If that's the case I'll make sure to just use that from now on instead of the QLS. And then inside the  "Reduce to single existing album" is where I can select the album or albums I want to synchronize?

But anyway yea I can see how a language difference can cause a little tension here and there. Also, "Reduce to single existing album" could probably be worded better, like "perform on single album" or "synchronize single album" unless I'm misunderstanding its scope and the word reduce is necessary.

thanks!
JT

Offline

 

#13 2015-10-31 10:32:56

plg
Piwigo Team
Nantes, France, Europe
2002-04-05
13174

Re: Quick Sync times out and gives Internal Server error 500

Hi JT,

JT wrote:

Hi, I thought that you sell Piwigo "hosting" on piwigo.com don't you?

Pigolabs sells Piwigo dedicated hosting on Piwigo.com. Piwigo.org is not Piwigo.com. flop25 is not in the Piwigo.com team, this is certainly why he says he has "nothing to sell" :-) flop25 is in Piwigo team, ie the team that build Piwigo, the free software.

Compared to flop25, I am part (I'm the founder) of both Piwigo.org and Piwigo.com, but here I don't try to sell you anything. I'm just helping "for free" :-)

I agree with you, we can be professional even if we have nothing to sell. Everybody has a unique personnality and as long as we all follow forum rules (such as not being offensive) then it's all fine. For me, when flop25 says you're "sentimental" it simply means that you have encountered a (very annoying) problem and you're saying this is due to Piwigo (sort of). From my point of view (and maybe from flop25 also) the problem comes from your unreliable filesystem, not from Piwigo. I think you understand the gap of "understanding" for your problem :-)

By the way, if you need/want some "pro" support, you can go to http://pigolabs.com (oups, here I try to sell something ;-)

JT wrote:

Plg, perhaps its a "culture" difference if in France its not an insult to call someone sentimental and irrational. Could be everyone's got real tough skin there, I mean last time I was on Champs-Élysées the pedestrians were "fair-game" for the motorists, many times they had to run/jump to the sidewalk for their life! :D not easy crossing the street there, luckily I was fine since I was a fast walker :)

Please, don't take "Champs-Elysees" drivers as representative to all french people. If that's the feeling you have from France and french people, then I'm much more like US people, but you can't expect everybody to be like that :-) But honestly I feel much more like being french than american, even if I'm a polite, professionnal and positive guy :-D (just joking, this is also the case of many French people and sometimes not true at all from US people)

JT wrote:

So just to confirm, if I had used the "Synchronize" button from under the Tools menu instead of the "Quick local synchronization" button (QLS), that would have properly done just a simulation?

Yes, page [Administration > Tools > Synchronize] has "simulate" option checked by default.

JT wrote:

If that's the case I'll make sure to just use that from now on instead of the QLS. And then inside the  "Reduce to single existing album" is where I can select the album or albums I want to synchronize?

Yes.

JT wrote:

But anyway yea I can see how a language difference can cause a little tension here and there. Also, "Reduce to single existing album" could probably be worded better, like "perform on single album" or "synchronize single album" unless I'm misunderstanding its scope and the word reduce is necessary.

The idea of this field is to reduce the scope of the synchronization. Maybe a "filter on album" would be better or "synchronize only one album".


Latest blog post (May 3rd 2018) New subscription form

Offline

 

#14 2015-11-10 03:56:15

JT
Member
2014-11-03
16

Re: Quick Sync times out and gives Internal Server error 500

Hi,
Thanks for the confirmations, and yea I would personally recommend the wording of "synchronize only one album" as best for general English users since the word "reduce" can be a little confusing for some.

Another question, is there now a BATCH uploader that supports htaccess (basic http auth)? (Flash doesn't)

And is there any Mobile App or an API that we can use to make our own simple uploading app?
thanks,
jt

Offline

 

Board footer

Powered by FluxBB

github twitter facebook google+ newsletter Donate Piwigo.org © 2002-2018 · Contact