Announcement

#1 2019-01-16 04:16:39

bsdll
Member
2019-01-15
2

Synchronization

Hello/Hi/Greetings,

I have been migrating photos from a custom program I wrote into piwigo.

I am currently up to 264851 images in 31 albums, about a third of the way through.

I have been copying one folder at a time, with one folder = one album, then using synchronization to load the data. I don't load the metadata.

I have finally reached the point where the server runs out of memory and can't increase it any further, so now get the fatal error:

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes) in /volume1/web/piwigo/admin/site_update.php on line 786

However, if I syncronise one specific folder at a time, then I am ok.

I can do this manually for now, but in the future I will be adding photos into albums at random, so will need to be able to synchronise the whole lot at once.

Is there a fix to make the synchronise process not try and cache everything in memory? It seems like a strange process to keep consuming memory until it runs out.

Or is there a way to automate the call to each specific album, so it would just run through the albums one after the other, doing the equivalent of a full synchronise, but not running out of memory?

Piwigo version: 2.9.4
PHP version: 5.6.36
MySQL version: 5.5.5-10.3.7-MariaDB
Piwigo URL: internal

Offline

 

#2 2019-01-17 08:31:51

eliz82
Member
Romania
2016-04-27
246

Re: Synchronization

Or is there a way to automate the call to each specific album, so it would just run through the albums one after the other, doing the equivalent of a full synchronise, but not running out of memory?

Yes, there is a way to do a (shell/powershell/php) script to automate that. Both wget and curl have the ability to save cookies and submit webforms.
This guy have some nice shell code examples how to use this futures with Piwigo https://github.com/pommes-frites/piwigo-scripts
His code is very simple. He make a call to Piwigo Api to save a session cookie file, then using that cookie he submit a form behind Piwigo admin interface. And that is all.
Because you are memory limited you could use something like a delay ( https://www.cyberciti.biz/faq/linux-uni … scripting/ ) between each album sync command to assure you are not starting multiple syncs in the same time.

Last edited by eliz82 (2019-01-17 08:35:30)

Offline

 

Board footer

Powered by FluxBB

github twitter facebook google+ newsletter Donate Piwigo.org © 2002-2019 · Contact