Announcement

  •  » Miscellaneous
  •  » 300,000 - 1,000,000 photos...can it be done?

#1 2017-10-19 09:56:22

Underdog89
Member
2017-10-19
6

300,000 - 1,000,000 photos...can it be done?

Hello all,

I am looking for a photo hosting solution. I will be looking to host 300,000 photos at around 2mb each to start. This will be growing to around 600,000 by the end of 2018. Ultimately I want the system to be able to comfortably handle 1 million photos with no slow down or crashes.

I have the hardware lined up so that should not be a problem.

My question is has any body hosted this number of photos with Piwigo? If so, how does it perform? Is there any advice when it comes to hosting that number of images.

Thanks in advance :-)

Offline

 

#2 2017-10-19 10:13:20

maximebrmd
Member
2017-09-21
11

Re: 300,000 - 1,000,000 photos...can it be done?

Hello Underdogs89,

Yes, Piwigo is able to handle 100,000 to 1 millions photos, that one of it's main qualities and you will need a huge server to stock all these photos (a server with 2 or 3 TB approximately)

- Maxime

Offline

 

#3 2017-10-19 10:19:47

Underdog89
Member
2017-10-19
6

Re: 300,000 - 1,000,000 photos...can it be done?

maximebrmd wrote:

Hello Underdogs89,

Yes, Piwigo is able to handle 100,000 to 1 millions photos, that one of it's main qualities and you will need a huge server to stock all these photos (a server with 2 or 3 TB approximately)

- Maxime

Nice I was hoping it would.

You are correct with regard to the server. Been told the following should be a good starting point:

- Dell R420 1U Server
- 2 x Xeon E5-2430 CPU's = 24 Cores in total
- 32GB RAM (can increase to 96GB)
- 2 x 3TB SATA drives

Offline

 

#4 2017-10-19 10:45:35

plg
Piwigo Team
Nantes, France, Europe
2002-04-05
13786

Re: 300,000 - 1,000,000 photos...can it be done?

Hi,

I confirm what maximebrmd said earlier and I would also like to add a "caution". We have identified a performance issue when you combine these factors:

* you have many photos, like 100k+
* you have some "permissions" (ie private albums)
* you are currently adding new photos with any method except FTP+sync (with web upload form for example)
* you have visits while adding photos

If you combine all these, Piwigo will have some difficulties because it will constantly reset its "permissions cache" (when adding a photo) and rebuild this cache (when a visitor opens a page). Rebuilding the case can take several seconds with a huge photo set.

We are working on improving that process. Maybe the simpler solution for now is to "lock gallery" (in [Administration > Tools > Maintenance]) while adding photos.

Remember, here I'm only talking about the minutes during which you add photos. Piwigo will run perfectly fast the rest of the time, even with 1 million photos :-)

Offline

 

#5 2017-10-19 10:52:07

Underdog89
Member
2017-10-19
6

Re: 300,000 - 1,000,000 photos...can it be done?

plg wrote:

Hi,

I confirm what maximebrmd said earlier and I would also like to add a "caution". We have identified a performance issue when you combine these factors:

* you have many photos, like 100k+
* you have some "permissions" (ie private albums)
* you are currently adding new photos with any method except FTP+sync (with web upload form for example)
* you have visits while adding photos

If you combine all these, Piwigo will have some difficulties because it will constantly reset its "permissions cache" (when adding a photo) and rebuild this cache (when a visitor opens a page). Rebuilding the case can take several seconds with a huge photo set.

We are working on improving that process. Maybe the simpler solution for now is to "lock gallery" (in [Administration > Tools > Maintenance]) while adding photos.

Remember, here I'm only talking about the minutes during which you add photos. Piwigo will run perfectly fast the rest of the time, even with 1 million photos :-)

That is useful to know. Thank you very much for the information :-)

Offline

 

#6 2018-04-27 00:25:59

goldennumber
Member
2018-03-05
12

Re: 300,000 - 1,000,000 photos...can it be done?

Underdog89 wrote:

Hello all,

I am looking for a photo hosting solution. I will be looking to host 300,000 photos at around 2mb each to start. This will be growing to around 600,000 by the end of 2018. Ultimately I want the system to be able to comfortably handle 1 million photos with no slow down or crashes.

I have the hardware lined up so that should not be a problem.

My question is has any body hosted this number of photos with Piwigo? If so, how does it perform? Is there any advice when it comes to hosting that number of images.

Thanks in advance :-)

Greetings!  How did your installation go?  Might you have any links to your install?  Might you have any tips?  Thanks so much!

I just posted a somewhat related question here:

http://piwigo.org/forum/viewtopic.php?id=28681

goldennumber wrote:

Greetings!

I am running the latest Piwigo on a test Centos Linux server just fine right now!

I will be building a second Centos Linux server soon to host over 500,000 photos in over 1,000 albums.  I am hoping to get some feedback/tips for optimizing the server, and I will share my progress!

The server will have two 6TB drives in a RAID-1 configuration.  The entire server will be devoted to the one Piwigo install, and I will be the only one adding photos, about once or twice a day or so. 

1. What processor(s) and RAM would you recommend?
2. Would it be good to get an SSD drive for the database and/or code?  How large of an SSD Drive?
3. What are the fastest/best-optimized Piwigo themes?
4. I will be the only one adding photos.  Are there rules to follow so as to optimize performance?
5. Is it worth getting a RAID hardware controller?  What is the fastest RAID configuration?
6. What other tips might you give someone building a server from scratch?

Thank you! :)

Thanks so much for any feedback! :)

Last edited by goldennumber (2018-04-27 00:41:08)

Offline

 

#7 2018-04-27 08:50:17

Underdog89
Member
2017-10-19
6

Re: 300,000 - 1,000,000 photos...can it be done?

Hello there,

In the end I did not opt for a self hosted solution. I came to the conclusion that the maintence costs and time would outway that of third party hosting.

I now use ShootProof to host the company photos(300,000) and have built a custom gallery using their API. But if I was going to go down the self hosted root Piwigo would have been the first choice.

I would be interested in seeing your solution once you have one built though :-)

Good luck!

Offline

 

#8 2018-04-27 09:17:46

flop25
Piwigo Team
2006-07-06
7037

Re: 300,000 - 1,000,000 photos...can it be done?

Underdog89 wrote:

I came to the conclusion that the maintence costs and time would outway that of third party hosting.

I totally understand that
But then you say:

Underdog89 wrote:

I now use ShootProof [...] and have built a custom gallery using their API

so you still have a server and a code to maintain... I don't see how you are saving money here: could you elaborate that would interesting


To get a better help : Politeness like Hello-A link-Your past actions precisely described
Check my extensions : more than 30 available
who I am and what I do : http://fr.gravatar.com/flop25
My gallery : an illustration of how to integrate Piwigo in your website

Offline

 

#9 2018-04-27 09:54:42

Underdog89
Member
2017-10-19
6

Re: 300,000 - 1,000,000 photos...can it be done?

Of course. Purely from a demand of server resources of users viewing thousands of images. We would have had to more than double our server costs to cope with demands that hosting our own gallery would have required.

Whereas the cost of the third party hosting solution was much less than that of the additional server costs AND I have the benefit of not having to maintain the hosting of the galleries and worry about downtime, security etc.

There are other benefits also to the third party. We have photographers across the UK that login to the hosting platform and upload their images that are then pulled through to the main website. I understand this could be done self-hosting. But at this moment in time, the third party makes more sense for cash flow and maintenance wise...one less thing to worry about for me.

:-)

Offline

 

#10 2018-04-27 09:57:00

flop25
Piwigo Team
2006-07-06
7037

Re: 300,000 - 1,000,000 photos...can it be done?

Hmm ok i get it; displaying and storing the picture is indeed costy
Thank you for your honest answers! Have a good day!


To get a better help : Politeness like Hello-A link-Your past actions precisely described
Check my extensions : more than 30 available
who I am and what I do : http://fr.gravatar.com/flop25
My gallery : an illustration of how to integrate Piwigo in your website

Offline

 

#11 2018-04-27 22:48:06

goldennumber
Member
2018-03-05
12

Re: 300,000 - 1,000,000 photos...can it be done?

Underdog89 wrote:

Hello there,

In the end I did not opt for a self hosted solution. I came to the conclusion that the maintence costs and time would outway that of third party hosting.

I now use ShootProof to host the company photos(300,000) and have built a custom gallery using their API. But if I was going to go down the self hosted root Piwigo would have been the first choice.

I would be interested in seeing your solution once you have one built though :-)

Good luck!

Thanks!  Is your solution fast and reliable?

Are you able to present the images exactly as you want?  And can you serve ads alongside the images?

Might ShootProof have any API gallery software one could test, or links to sample sites?

What advantages might hosting with Piwigo on your own server grant you?

Thank you!

Offline

 

#12 2018-04-28 14:09:31

Underdog89
Member
2017-10-19
6

Re: 300,000 - 1,000,000 photos...can it be done?

goldennumber wrote:

Underdog89 wrote:

Hello there,

In the end I did not opt for a self hosted solution. I came to the conclusion that the maintence costs and time would outway that of third party hosting.

I now use ShootProof to host the company photos(300,000) and have built a custom gallery using their API. But if I was going to go down the self hosted root Piwigo would have been the first choice.

I would be interested in seeing your solution once you have one built though :-)

Good luck!

Thanks!  Is your solution fast and reliable?

Are you able to present the images exactly as you want?  And can you serve ads alongside the images?

Might ShootProof have any API gallery software one could test, or links to sample sites?

What advantages might hosting with Piwigo on your own server grant you?

Thank you!

Yes, ShootProof is fast and reliable. However, the free package can only hold a few images(100 I think).

The downside is that there are no prebuilt galleries that I could find. I had to build a bespoke gallery within my sites to pull the images and albums using an API and cron jobs to keep updates.

Piwigo and self-hosting give you the advantage of full control and integration how you see fit. If I had under 50,000 images I would have gone with Piwigo. But as we run from 300,000 to possible 1,000,000 images the server costs would have been large compared to a third party hosting service.

As with regard to ads. I don't run ads on the sites I have built. But you could easily integrate ads within a custom built gallery.

The only reason I didn't go with Piwigo is due to the number of images and the server costs. I would deffo keep them in mind for future projects.

Offline

 

#13 2018-04-28 21:14:00

goldennumber
Member
2018-03-05
12

Re: 300,000 - 1,000,000 photos...can it be done?

Underdog89 wrote:

goldennumber wrote:

Underdog89 wrote:

Hello there,

In the end I did not opt for a self hosted solution. I came to the conclusion that the maintence costs and time would outway that of third party hosting.

I now use ShootProof to host the company photos(300,000) and have built a custom gallery using their API. But if I was going to go down the self hosted root Piwigo would have been the first choice.

I would be interested in seeing your solution once you have one built though :-)

Good luck!

Thanks!  Is your solution fast and reliable?

Are you able to present the images exactly as you want?  And can you serve ads alongside the images?

Might ShootProof have any API gallery software one could test, or links to sample sites?

What advantages might hosting with Piwigo on your own server grant you?

Thank you!

Yes, ShootProof is fast and reliable. However, the free package can only hold a few images(100 I think).

The downside is that there are no prebuilt galleries that I could find. I had to build a bespoke gallery within my sites to pull the images and albums using an API and cron jobs to keep updates.

Piwigo and self-hosting give you the advantage of full control and integration how you see fit. If I had under 50,000 images I would have gone with Piwigo. But as we run from 300,000 to possible 1,000,000 images the server costs would have been large compared to a third party hosting service.

As with regard to ads. I don't run ads on the sites I have built. But you could easily integrate ads within a custom built gallery.

The only reason I didn't go with Piwigo is due to the number of images and the server costs. I would deffo keep them in mind for future projects.

Thanks!

How much was your Piwigo server going to cost, and what were its specs?  How many users & views per day were you building it for?

Would love to see your final site, if you wish to share a link. :)

Would love to talk to you more--perhaps we could hire you for your solution if you had time?  Might you have an email?  Mine is goldennumberratio@gmail.com .  Please feel free to contact me!

Thank you! :)

Offline

 

#14 2020-12-04 14:37:32

photo_friend
Member
Berlin
1970-01-01
202

Re: 300,000 - 1,000,000 photos...can it be done?

plg wrote:

Hi,

....

If you combine all these, Piwigo will have some difficulties because it will constantly reset its "permissions cache" (when adding a photo) and rebuild this cache (when a visitor opens a page). Rebuilding the case can take several seconds with a huge photo set.

We are working on improving that process. Maybe the simpler solution for now is to "lock gallery" (in [Administration > Tools > Maintenance]) while adding photos.

Remember, here I'm only talking about the minutes during which you add photos. Piwigo will run perfectly fast the rest of the time, even with 1 million photos :-)

=> I have found this post because I have similar issue. I run a piwigo installation with 150k photos/1500 albums and I regularly update the images directly via Lightroom Plugin (alloyphoto.com) with image Adjustments, Tags, Ratings, GPS, ... .
Updating a few thousand images can take a few hours - which is fine.
The problem is, that during the upload/update process the website is nearly inaccessible, because the whole website slows down. As soon the upload stops, the side runs smooth and fast again.
If I access the gallery via iOS-App during update, there is no issue.

Any solution here? You said in the post, Piwigo will have some difficulties because it will constantly reset its "permissions cache" (when adding a photo) and rebuild this cache (when a visitor opens a page). Rebuilding the case can take several seconds with a huge photo set.
Thanks for your support!!


Piwigo 2.10.2
Betriebssystem: Linux
PHP: 7.4.13
MySQL: 5.5.5-10.1.47-MariaDB-0ubuntu0.18.04.1
Grafikbibliothek: ImageMagick 6.9.7-4
2TB SSD
16 Core INTEL XEON
48GB RAM

Offline

 

#15 2021-01-04 22:14:05

photo_friend
Member
Berlin
1970-01-01
202

Re: 300,000 - 1,000,000 photos...can it be done?

photo_friend wrote:

plg wrote:

Hi,

....

If you combine all these, Piwigo will have some difficulties because it will constantly reset its "permissions cache" (when adding a photo) and rebuild this cache (when a visitor opens a page). Rebuilding the case can take several seconds with a huge photo set.

We are working on improving that process. Maybe the simpler solution for now is to "lock gallery" (in [Administration > Tools > Maintenance]) while adding photos.

Remember, here I'm only talking about the minutes during which you add photos. Piwigo will run perfectly fast the rest of the time, even with 1 million photos :-)

=> I have found this post because I have similar issue. I run a piwigo installation with 150k photos/1500 albums and I regularly update the images directly via Lightroom Plugin ......

.
.
?Any updates regarding this issue with large number of photos?

Offline

 
  •  » Miscellaneous
  •  » 300,000 - 1,000,000 photos...can it be done?

Board footer

Powered by FluxBB

github twitter newsletter Donate Piwigo.org © 2002-2024 · Contact