#1 2014-07-28 06:19:31

Applepiee
Member
2014-07-11
41

[Help!] High traffic configuration tweaks

Hello/Hi/Greetings,

I was using a custom coded gallery script (albums, guest rating, guest uploads) on my website. but since yesterday night I finally made my piwigo system live, but after the first minutes the loads of my server already went up to 150 and sometimes it spikes up to 700..!

My hosting company is still looking for the problem at this moment but they can't seem to find it.
I never had load problems before, it all started the minute I made piwigo online.

My website has around 2 Million Page views a day and it always has over 1000 active connections (stats pulled form google analytics).

My dedicated server config is: Intel Xeon E3-1230 - 16GB RAM - 1TB HDD - 10TB BW

Anyone has any idea what I could do to fix this problem? Or is piwigo just not made for high traffic websites?

Regards,
Davy M

Piwigo version: 2.6.3
PHP version: 5.4
MySQL version:
Piwigo URL: **removed**

Offline

 

#2 2014-07-28 14:40:30

Applepiee
Member
2014-07-11
41

Re: [Help!] High traffic configuration tweaks

Update.

Updated to 2.7.0beta2 since I saw it had some caching improvements but It only helped a little.
At this moment I stopped using Piwigo, but I REALLY want to use it, it's such an amazing script that it's a steal to get for free.

I will need some of your Piwigo Team members to help me throubleshoot this problem. And you guys can use my website to tweak your system for high traffic websites like mine.

If any of the team members read this, please send me an e-mail or reply here. I'm eager to have this script working.

--
Applepiee

Offline

 

#3 2014-07-28 16:30:46

rvelices
Former Piwigo Team
2005-12-29
1960

Re: [Help!] High traffic configuration tweaks

Its supposed to handle well the spikes.
Just keep in mind that some plugins are not well optimized
The first time it is required, original images are converted to smaller sizes which is cpu intensive.

Offline

 

#4 2014-07-28 16:46:30

Applepiee
Member
2014-07-11
41

Re: [Help!] High traffic configuration tweaks

Hello!

You're right about the plugins, but I disabled them the first 10 minutes, that was my first try (Specially the language changer is a real killer)

About the first time, I used galleries and I synced it, I can't see any process, and I trying 20 random images and they had all sizes.

You should see when it goes online on my website.

some words from my hosting:
There are hundreds of PHP processes running on the server.  MySQL CPU usage is
through the roof.  I have gotten your Apache values as high as I can without the
server blowing up, your CPU usage hasn't dropped below 100% for some time and
your memory is almost full as well.

They have a feeling that somewhere this is missing: http://php.net/manual/en/function.mysql-close.php

My website works perfectly but when changing to your gallery script it works for 2 minutes but when I see the active users going over around 250 in google analytics then the cpu just blows up some how.

--
Applepiee

Offline

 

#5 2014-07-28 19:00:27

flop25
Piwigo Team
2006-07-06
7038

Re: [Help!] High traffic configuration tweaks

They have a feeling that somewhere this is missing: http://php.net/manual/en/function.mysql-close.php
LOL

if you don't give us any material to work on that's gonna be problematic. Like server logs. Use monitoring software or services. If you're making a huge website you are supposed to monitor deeply your server


To get a better help : Politeness like Hello-A link-Your past actions precisely described
Check my extensions : more than 30 available
who I am and what I do : http://fr.gravatar.com/flop25
My gallery : an illustration of how to integrate Piwigo in your website

Offline

 

#6 2014-07-28 19:01:45

plg
Piwigo Team
Nantes, France, Europe
2002-04-05
14087

Re: [Help!] High traffic configuration tweaks

I'm discussing with Applepiee to understand what can be wrong.

Offline

 

#7 2014-07-28 19:33:11

Applepiee
Member
2014-07-11
41

Re: [Help!] High traffic configuration tweaks

@Flop25

I'm sorry if I have offended you or your team in any way.. I'm not sure what that was, I simply quoted what they said.

I'm not sure which logs I was supposed to drop here, I was hoping you guys would tell me that.

Again, My apologies if I came over like an asshole

--
Davy M

Offline

 

#8 2014-07-29 09:35:00

ViPeR_930
Member
2012-09-09
105

Re: [Help!] High traffic configuration tweaks

I am interested in knowing what the resolution was. Please post if you've solved the issue.

Offline

 

#9 2014-07-29 13:55:50

Applepiee
Member
2014-07-11
41

Re: [Help!] High traffic configuration tweaks

plg is working on it as we speak.
I will update here with the solutions, but I think plg will post them here when he finds performance tweaks.

Offline

 

#10 2014-07-31 17:15:41

plg
Piwigo Team
Nantes, France, Europe
2002-04-05
14087

Re: [Help!] High traffic configuration tweaks

As described by Applepiee when visitors were sent to Piwigo, the server load was rising up 300 and more. In just a few seconds. Sometimes the server was also going very high in load with the old "specific" PHP script, but not for a long time.

Considering the number of "php <defunct>" processes visible on htop, I wondered how exactly the PHP scripts were handled: nginx (cache for images, reverse-proxy on Apache for PHP) => Apache => suPHP. I think suPHP is awfully slow on high traffic.

After many hours of tests, we moved the website on a new clean server, with the same kind of CPU but with another way to handle PHP execution: nginx => PHP-FPM. It is now "crazy fast". The load doesn't go higher than 7 (which is fine on a 8 threads CPU) and Piwigo page generation time never go higher than 100ms.

So now, my recommendation for high traffic on Piwigo is to use nginx + PHP-FPM. Here is the nginx conf I use:

server {
  server_name www.piwigonginx.com piwigonginx.com;
  root /var/www/piwigonginx.com;

  # with this you can remove ".php?" in Piwigo URLs
  location @rewrite {
    rewrite ^/picture((/|$).*)$ /picture.php$1 last;
    rewrite ^/index((/|$).*)$ /index.php$1 last;
    rewrite ^/i((/|$).*)$ /i.php$1 last;
  }

  location / {
    index index.php;
    try_files $uri $uri/ @rewrite;
  }

  location ~ ^(?<script_name>.+?\.php)(?<path_info>/.*)?$ {
    fastcgi_pass unix:/var/run/php5-fpm.sock;
    include fastcgi_params;
  }
}

Offline

 

#11 2014-08-01 00:56:18

mendip_discovery
Member
Mendip Hills, Somerset, UK
2012-05-19
91

Re: [Help!] High traffic configuration tweaks

So am I assuming correctly use nginx to process static items such as images, plus ask it to process the PHP files via PHP-FPM.

I am still fighting with my server so I set nginx up via Plesk so will test to see how it does.

Offline

 

Board footer

Powered by FluxBB