Announcement

  •  » Engine
  •  » Performance issue with high volume

#1 2017-10-28 17:17:07

plg
Piwigo Team
Nantes, France, Europe
2002-04-05
13082

Performance issue with high volume

Preamble : to solve performance issues, we have implemented a "permission cache" in Piwigo.

More and more frequently I encounter a major performance issue with Piwigo when combining these conditions:

* many photos/albums, like 100k photos and 1k albums
* private albums or photos
* photos are being uploaded with web upload or API
* each photo is uploaded in a few seconds (small photos or big bandwidth)
* visitors are active on gallery side

The root of the problem here is that each time a photo is added, this cache is reset. Each time a visitor opens a page, the cache is rebuilt if not available. In the described situation, the cache is rebuilt each time a visitor opens a page. Rebuilding the cache may take 5 to 10 seconds.

Even "locking" the gallery does not really solve the problem, because cache is rebuilt even in this situation (but I can push a change to avoid rebuilding cache if the gallery is locked, I have implemented that quickly on Piwigo.com)

Improving cache time generation sounds very complicated to me. The solution I see for now is to avoid cache reset on each photo added. Unfortunately that's going to lead to incoherences in the database. In addition to we would need to put the added photo in a "pending state" (with images.level=64 for instance) and after X minutes (or any other event) reset cache and remove the "pending" state.

By the way, when I write about cache reset, I describe what function invalidate_user_cache does.

Any better idea?


Latest blog post (March 23th 2017) Piwigo.com Enterprise plans, now official!

Offline

 

#2 2017-10-28 17:30:02

mistic100
Former Piwigo Team
Lyon (FR)
2008-09-27
3267

Re: Performance issue with high volume

I think this is somehow linked to the still not implemented feature of "upload session" : know when a batch of photos has been loaded.

Smart Albums suffers the same problem where every rule is re-evaluated after each upload.

I spend a lot of time on https://alpha.wallhaven.cc/ these days. When you upload files they enter a pending state where you can edit them before publish (they did this on purpose to force users to add tags but the same principle can be done on Piwigo).

So I think your idea is the right one : enforce a pending state for every uploaded file. The user will be able to prepare it's photos (useful when you don't have correct metadata) and click a nice "Publish" button, and there you can reset the cache (among other things). This could be integrated in the batch manager. This will replace the "upload done" page we currently have.

If the upload was aborted (browser closed, etc.) make it clear on the admin homepage that there are pending files.

Offline

 

#3 2017-10-29 09:10:04

rvelices
Piwigo Team
2005-12-29
1954

Re: Performance issue with high volume

Not an easy situation... If there are many logged users with same rights we could eventually cache permissions only once for all the users with the same permission (but it's a breaking change... And won't work in all the cases)

Offline

 

#4 2017-10-30 14:10:00

plg
Piwigo Team
Nantes, France, Europe
2002-04-05
13082

Re: Performance issue with high volume

rvelices wrote:

Not an easy situation... If there are many logged users with same rights we could eventually cache permissions only once for all the users with the same permission (but it's a breaking change... And won't work in all the cases)

I think it would only make the problem less problematic, but the problem would still be here :

* 07h45m12s : new photo added, invalidate_user_cache
* 07h45m13s : user u1 opens a page, cache rebuilt for u1
* 07h45m14s : user u1 opens a page, cache is fine
* 07h45m14s:  user u2 opens  a page, cache rebuilt for u2
* 07h45m15s: new photo added, invalidate_user_cache
* 07h45m16s:  user u1 opens a page, cache rebuilt for u1
* ...

Now imagine you have 100 visitors instead of 2. Every time each user opens a page, the cache is rebuilt :-/


Latest blog post (March 23th 2017) Piwigo.com Enterprise plans, now official!

Offline

 

#5 2017-10-30 14:20:58

plg
Piwigo Team
Nantes, France, Europe
2002-04-05
13082

Re: Performance issue with high volume

mistic100 wrote:

I think this is somehow linked to the still not implemented feature of "upload session" : know when a batch of photos has been loaded.

I have implemented something in Community : [Github] Piwigo-community commit d7188274 API method community.images.uploadCompleted

I could implement an equivalent for Piwigo core. The problem is you're not 100% sure the API method will ever get fired (any problem can occur on browser side, as you say later in your message).

mistic100 wrote:

So I think your idea is the right one : enforce a pending state for every uploaded file. The user will be able to prepare it's photos (useful when you don't have correct metadata) and click a nice "Publish" button, and there you can reset the cache (among other things). This could be integrated in the batch manager. This will replace the "upload done" page we currently have.

If the upload was aborted (browser closed, etc.) make it clear on the admin homepage that there are pending files.

OK, so you're "validating" the basic idea of a "staging" state and you're introducing a "manual" publish action, while I was more thinking about an "automatic" action. For example, you keep in database a $conf['oldest_staging_image'] = <date>. In include/common.inc.php, if there is such a config setting and if its value is older than X minutes, perform the "publish action" automatically. The idea behind auto-publish is to keep the current workflow valid. But the manual-publish is interesting too and it could be a configuration option : publish => auto/manual


Latest blog post (March 23th 2017) Piwigo.com Enterprise plans, now official!

Offline

 

#6 2017-10-30 15:30:45

flop25
Piwigo Team
2006-07-06
6603

Re: Performance issue with high volume

I agree with the idea of a publishing step which has to be done manually by default or could be also done automatically.

Beside solving the technical issue, the idea behind is that people might upload wrongly their picture and a manual publishing step for verification is a additional safety measure quite comforting (the sure has full control, a full overview and time). When you have private and public stuff, or different privacy policies it can be a bit stressful.

The disadvantage is that it sort of disable the great advantage of never going to your admin panel by uploading thought api : so that case should be handle specifically

The workflow I was thinking involved an email sent after Xhours with a link to check the pending pictures and a link to directly publish the picture (but no thumbnail, just x files sent and when) ; in the case of files sent through the api, we could add a method in the api to trigger the email, so people sending from an third party app could directly publish by clicking the link (smartphone, long upload, email, click and shazam)

That email solution could be an additional solution on top of a config variable to automatically publish after Xtime ; what I mean is that I see that's technically one step further, so it could be done afterwards. The config var is necessary but the emails is just an enhancement of the feature.

BUT instead of one $conf['oldest_pending_files'] i would really prefer two, one for pictures from the admin panel and one for the "real" api uploads: because I would set a very short time for api upload and a longer one for web upload. Usually people are preparing much more carefully their uploads when it's done from a third party software (Lightroom for instance)

my 2 cents


To get a better help : Politeness like Hello-A link-Your past actions precisely described
Check my extensions : more than 30 available
who I am and what I do : http://fr.gravatar.com/flop25
My gallery : an illustration of how to integrate Piwigo in your website

Offline

 

#7 2017-10-30 15:39:01

plg
Piwigo Team
Nantes, France, Europe
2002-04-05
13082

Re: Performance issue with high volume

Good point: staging photos is problematic for remote app (Lightroom, mobile apps, digiKam...) because it would require admins to go in the Piwigo admin :-/ In that case, automatic publish sounds mandatory to me.

I would also like to point out that web upload and API upload is quite the same: since Piwigo 2.7 and HTML5 upload, Piwigo web upload form uses API method pwg.images.upload, just like the iOS app.


Latest blog post (March 23th 2017) Piwigo.com Enterprise plans, now official!

Offline

 

#8 2017-10-30 15:50:52

flop25
Piwigo Team
2006-07-06
6603

Re: Performance issue with high volume

plg wrote:

I would also like to point out that web upload and API upload is quite the same: since Piwigo 2.7 and HTML5 upload, Piwigo web upload form uses API method pwg.images.upload, just like the iOS app.

yeah i Know but for the user that an entire different workflow (at least as I understand how our users work)


To get a better help : Politeness like Hello-A link-Your past actions precisely described
Check my extensions : more than 30 available
who I am and what I do : http://fr.gravatar.com/flop25
My gallery : an illustration of how to integrate Piwigo in your website

Offline

 

#9 2017-10-30 16:09:18

plg
Piwigo Team
Nantes, France, Europe
2002-04-05
13082

Re: Performance issue with high volume

of course, I just wanted to point out that "technically speaking" web upload ~= API upload :-) I fully agree with you, on a user perspective, upload through Lightroom is totally different from web upload!


Latest blog post (March 23th 2017) Piwigo.com Enterprise plans, now official!

Offline

 
  •  » Engine
  •  » Performance issue with high volume

Board footer

Powered by FluxBB

github twitter facebook google+ newsletter Donate Piwigo.org © 2002-2017 · Contact