Piwigo.org

You are not logged in. (Register / Login)

Announcement

#1 2017-10-19 09:56:22

Underdog89
Member
Registered: 2017-10-19
Posts: 3

300,000 - 1,000,000 photos...can it be done?

Hello all,

I am looking for a photo hosting solution. I will be looking to host 300,000 photos at around 2mb each to start. This will be growing to around 600,000 by the end of 2018. Ultimately I want the system to be able to comfortably handle 1 million photos with no slow down or crashes.

I have the hardware lined up so that should not be a problem.

My question is has any body hosted this number of photos with Piwigo? If so, how does it perform? Is there any advice when it comes to hosting that number of images.

Thanks in advance :-)

Offline

 

#2 2017-10-19 10:13:20

maximebrmd
Member
Registered: 2017-09-21
Posts: 11

Re: 300,000 - 1,000,000 photos...can it be done?

Hello Underdogs89,

Yes, Piwigo is able to handle 100,000 to 1 millions photos, that one of it's main qualities and you will need a huge server to stock all these photos (a server with 2 or 3 TB approximately)

- Maxime

Offline

 

#3 2017-10-19 10:19:47

Underdog89
Member
Registered: 2017-10-19
Posts: 3

Re: 300,000 - 1,000,000 photos...can it be done?

maximebrmd wrote:

Hello Underdogs89,

Yes, Piwigo is able to handle 100,000 to 1 millions photos, that one of it's main qualities and you will need a huge server to stock all these photos (a server with 2 or 3 TB approximately)

- Maxime

Nice I was hoping it would.

You are correct with regard to the server. Been told the following should be a good starting point:

- Dell R420 1U Server
- 2 x Xeon E5-2430 CPU's = 24 Cores in total
- 32GB RAM (can increase to 96GB)
- 2 x 3TB SATA drives

Offline

 

#4 2017-10-19 10:45:35

plg
Piwigo Team
Location: Nantes, France, Europe
Registered: 2002-04-05
Posts: 13080
Website

Re: 300,000 - 1,000,000 photos...can it be done?

Hi,

I confirm what maximebrmd said earlier and I would also like to add a "caution". We have identified a performance issue when you combine these factors:

* you have many photos, like 100k+
* you have some "permissions" (ie private albums)
* you are currently adding new photos with any method except FTP+sync (with web upload form for example)
* you have visits while adding photos

If you combine all these, Piwigo will have some difficulties because it will constantly reset its "permissions cache" (when adding a photo) and rebuild this cache (when a visitor opens a page). Rebuilding the case can take several seconds with a huge photo set.

We are working on improving that process. Maybe the simpler solution for now is to "lock gallery" (in [Administration > Tools > Maintenance]) while adding photos.

Remember, here I'm only talking about the minutes during which you add photos. Piwigo will run perfectly fast the rest of the time, even with 1 million photos :-)


Latest blog post (March 23th 2017) Piwigo.com Enterprise plans, now official!

Offline

 

#5 2017-10-19 10:52:07

Underdog89
Member
Registered: 2017-10-19
Posts: 3

Re: 300,000 - 1,000,000 photos...can it be done?

plg wrote:

Hi,

I confirm what maximebrmd said earlier and I would also like to add a "caution". We have identified a performance issue when you combine these factors:

* you have many photos, like 100k+
* you have some "permissions" (ie private albums)
* you are currently adding new photos with any method except FTP+sync (with web upload form for example)
* you have visits while adding photos

If you combine all these, Piwigo will have some difficulties because it will constantly reset its "permissions cache" (when adding a photo) and rebuild this cache (when a visitor opens a page). Rebuilding the case can take several seconds with a huge photo set.

We are working on improving that process. Maybe the simpler solution for now is to "lock gallery" (in [Administration > Tools > Maintenance]) while adding photos.

Remember, here I'm only talking about the minutes during which you add photos. Piwigo will run perfectly fast the rest of the time, even with 1 million photos :-)

That is useful to know. Thank you very much for the information :-)

Offline

 

Board footer

Powered by FluxBB

github twitter facebook google+ newsletter Donate Piwigo.org © 2002-2017 · Contact