🌍
English
Announcement
- [2024-12-20] Piwigo 15.3.0 : same as 15.2.0 but compatible with PHP 7
- [2024-12-19] Piwigo 15.2.0 : better user activation and many fixes
- [2024-11-11] Piwigo 15.1.0 : small fixes and what's new popup
- [2024-10-22] Piwigo 15
- [2024-09-23] Piwigo 15.0.0RC1, almost there
#1 2003-11-08 04:11:44
- zjs2k
- Guest
How Big Is Too Big?
Hi,
First, thank you for a great piece of code and it's just what I am looking for.
After exploring the codes for a while, I started to worry about the performance for a huge gallery. I have a gallery with amost 2000 categories (over 10 GB). I understand that the category list is reading the whole 'categories' table into an array. If there are a couple thousands entries in the array and it's called on every category pages, I think it will really slow down things. Of course, I can put several copies of phpwebgallery to share the load. But I am wondering if there is a better way to do it. I don't know if phpwebgallery can handle that many categories or not. Do you have any speculation on that?
Thanks for any help!
Jin
#2 2003-11-08 09:24:22
- plg
- Piwigo Team
- Nantes, France, Europe
- 2002-04-05
- 13885
Re: How Big Is Too Big?
Well, well, well :-? In reality, the main default of PhpWebGallery 1.3 is the management of many categories... That's stupid, I made my developements with only a dozen of categories and there was no problem. I've recently made a "stress test" with 150 categories (far from 2000) and 6 levels and times were really bad.
You took a look at the code and realized quickly where the problem is : get_plain_structure, create_structure and update_structure ! Even if you show 20 categories in the left menu, ALL the categories are retrieved from the database and organised in the $page['structure'] array each time you open category.php.
For the moment, I have no solution, but I'm working on an improvement for next minor release (1.3.x). In your case, you can use release 1.2.x which won't have a problem handling so many categories I think (but much more database queries).
If you have ideas the way I can do (because I can do as in the release 1.2.x, but do not wish so).
Offline
#3 2003-11-08 11:01:55
- zjs2k
- Member
- 2003-11-08
- 1
Re: How Big Is Too Big?
Thank you for your quick reply. I am not sure how you did it in 1.2.x because I only looked into the 1.3 code. And I see you use all French comment in your scripts and I don't anything in French :(
Here is my thought (and it could be the one you used in 1.2.x): instead of reading the whole categories table, get the top level only. When a user is browsing a particular category, get the hierarchical structure for that category only, which requires recursive function callback. It's not very efficient, but it may work for me because most of my categories are less than 3 levels. In order to avoid the recursive call, you may need to alter the database structure a little bit and add the hierarchical sequence to each category. For example, you can use 1-20-9-30 to indicate the parent categories are (from top level) cat 1, cat 20, cat 9 and cat 30.
If it's still not fast enough, you can always to generate a static page for category. Update that page only when you update the category structure. For visitors, the category.php page only needs to include that static page.
May I make another suggestion? You may want to use object-oriented programming and module structure in your future development. Now I am making changes to the main code and it's gonna be a pain if I need to upgrade in the future. I know it's hard to do oo programming at the beginning, but it will save you a lot of time in the future and you may get more people to contribute.
Thanks.
Jin
Offline
#4 2003-11-08 11:57:54
- plg
- Piwigo Team
- Nantes, France, Europe
- 2002-04-05
- 13885
Re: How Big Is Too Big?
I know it's hard to do oo programming at the beginning
I'm an engineer (in computer sciences) and I've started to learn programming at school. I began with C, then C++ (so OO), Java (a lot of Java), PHP and Perl (I know many other languages but I only mention the main ones). Programming with objects and classes is not really a problem :-) It's just that at the beginning, PhpWebGallery was very small and did not need OO. Maybe (surely ?) for release 2.0
I've read the rest of your post and I'll answer after lunch :)
Offline
#5 2003-11-08 13:33:41
- plg
- Piwigo Team
- Nantes, France, Europe
- 2002-04-05
- 13885
Re: How Big Is Too Big?
Thank you for your quick reply. I am not sure how you did it in 1.2.x because I only looked into the 1.3 code. And I see you use all French comment in your scripts and I don't anything in French :(
So, it was a good thing I translated all what was in French into English... Where are you from ?
Here is my thought (and it could be the one you used in 1.2.x)
Briefly, the release 1.2.x was working this way for menu display :
1. get the top level (1 query)
2. for each expanded category get sub categories (1 query per category expanded)
The goal of the 1.3 new method was to reduce the number of queries : anywhat the number of categories to expand is, the number of queries is the same : 1.
The $page['structure'] array (N-level) is also used in many functions to retrieve the full name of a category ("party - anniversary - Pierrick 2003" instead of "Pierrick 2003") or getting the full dir...
you may need to alter the database structure a little bit and add the hierarchical sequence to each category
I do not wish so, I'll prefer to add some additional queries (the less possible).
If it's still not fast enough, you can always to generate a static page for category. Update that page only when you update the category structure. For visitors, the category.php page only needs to include that static page.
That the first thing I could do BUT you forget the system of permission / invisibility... User "paul" may not have the same permission as "mike" and categories menu will not be the same. And if I store a static menu, do I expand all categories ? only users can answer this...
Offline
#6 2003-11-09 16:17:20
- plg
- Piwigo Team
- Nantes, France, Europe
- 2002-04-05
- 13885
Re: How Big Is Too Big?
I've been working on this problem yesterday and today. Here is the solution I use :
· adding a column #categories.uppercats wich contains the ordered list of parent categories ids ("2,3,23,45" for example)
· adding a table #user_category (for a user + category) wich contains the last_date + number of sub_categories
· for creating the plain structure, I only retrieve the top level + the categories with id_uppercat contained in the expand array, I use a join query to also retrieve the user las_date and user number of sub categories for each category
This is the first point for public part. For admin panel, there is a problem : in pages where you find a select box to link a picture to another category (virtual link), I still need to create the complete structure... and generation time will still be long ! Would you find satisfactory something like this :
· for less than 100 categories, I display the complete structure
· for more than 100 categories, I display a simple text box asking the id of wished parent category
Offline
#7 2003-11-16 17:45:34
- plg
- Piwigo Team
- Nantes, France, Europe
- 2002-04-05
- 13885
Re: How Big Is Too Big?
I'm still working on this performances problem... It's all improved but there are now many calculated data that are stored in the database (calcutated once, used many times). All these improvement will be added in release 1.3.1 (don't know when it's ready)
Offline