Announcement

#1 2016-07-05 20:52:22

palantus
Member
2016-07-05
4

Generation of custom size thumbnails

Hi

I have been experimenting with the theme Bootstrap Darkroom and plugin GThumb+ (or gdThumb).

With any combination of the plugins or theme, the thumbnails used in my albums are regenerated on the first visit, which is slow.

I have already generated images for the sizes thumbnail, square, medium and XL using the Batch Manager.

Based on the URL of the thumbnails shown in the gallery (ends with "-cu_s9999x240" when using GThumb+), the size requested is not among the ones I have generated. So I guess it makes sense, that they are generated again.

Now, my question is: How do I generate these image sizes for all of my albums?

I tried changing XXS to 9999x240, but it wont let me save that value. When viewing Details for the image sizes, s9999x240 is listed as a custom size, but I can't select it in the Batch Manager :(

Piwigo version: 2.8.1
PHP version: 7

Offline

 

#2 2016-07-05 21:25:48

teekay
Member
2013-06-12
404

Re: Generation of custom size thumbnails

The 9999 is a placeholder, not an actual pixel number. I think it's width x height, so the thumbnails get a fixed height of 240px, and the width is calculated by image's aspect ratio, plus there is some cropping if needed and enabled in plugin settings.

I don't think there is a way to pre-generate the thumbnails for gthumb+/gdthumb.

Offline

 

#3 2016-07-06 08:16:33

palantus
Member
2016-07-05
4

Re: Generation of custom size thumbnails

teekay wrote:

The 9999 is a placeholder, not an actual pixel number. I think it's width x height, so the thumbnails get a fixed height of 240px, and the width is calculated by image's aspect ratio, plus there is some cropping if needed and enabled in plugin settings.

I don't think there is a way to pre-generate the thumbnails for gthumb+/gdthumb.

Yes, but if it was possible to enter 9999x240 as resolution on eg. XXS, then I was hoping that the same would happen when generating those images in batch manager (that is: fit the image within 9999x240, taking aspect ratio into account).

However, when I try that, I get an error on two other fields saying that they are set to more than 9999 (which they aren't) :(

But even if I could enter that, I guess that there isn't any guarantee that the actual images would be used...

Oh well - guess I'll just stick with the default theme and image sizes.

Thanks.

Offline

 

#4 2016-07-06 09:26:33

teekay
Member
2013-06-12
404

Re: Generation of custom size thumbnails

Could you take a look at Administration index page -> Server environment? Which graphics library is used? (php-gd or ImageMagick)?

Actually with ImageMagick it should be pretty fast to generate the thumbnails on reasonable hardware (like: not a raspberry pi), and if you add the RV Thumbnail Scroller plugin and set thumbnails per page to something like 30 in GThumb+'s settings, then initial page load time should be acceptable. Scrolling down would load the next 30, and so on.

Bootstrap Default/Darkroom without GThumb+/gdThumb uses a fixed thumbnail size of 260x180, naming them -cu_e260x180.jpg. They are cached once they have been generated. There is no batch manager action to pre-generate them either, though.

Maybe we could implement a batch manager action to generate custom derivative sizes. I'll look into that when I find the time.

Offline

 

#5 2016-07-06 10:56:11

palantus
Member
2016-07-05
4

Re: Generation of custom size thumbnails

teekay wrote:

Could you take a look at Administration index page -> Server environment? Which graphics library is used? (php-gd or ImageMagick)?

I'm using ImageMagick.

teekay wrote:

Actually with ImageMagick it should be pretty fast to generate the thumbnails on reasonable hardware (like: not a raspberry pi), and if you add the RV Thumbnail Scroller plugin and set thumbnails per page to something like 30 in GThumb+'s settings, then initial page load time should be acceptable. Scrolling down would load the next 30, and so on.

I'm running Piwigo on a ASRock C2550D4I (quad-core Atom C2550) and it is (unfortunatly) not fast enough with that setup and a height of 180px (10+ seconds per page).

teekay wrote:

Maybe we could implement a batch manager action to generate custom derivative sizes. I'll look into that when I find the time.

That would be great! :) Will you handle the issue or should I report it?

Offline

 

#6 2016-07-06 15:51:04

teekay
Member
2013-06-12
404

Re: Generation of custom size thumbnails

Here you go.

Enable the "Personal Plugin" and LocalFiles Editor plugin.

Paste this into Administration -> Plugins -> LocalFiles Editor -> Personal Plugin

Code:

add_event_handler('loc_end_element_set_global', 'bm_gen_custom_derivatives_form');
add_event_handler('element_set_global_action', 'bm_gen_custom_derivatives_action', EVENT_HANDLER_PRIORITY_NEUTRAL, 2);
  
function bm_gen_custom_derivatives_form() {
  global $template;
  $template_add = '';
  foreach (array_keys(ImageStdParams::$custom) as $custom) {
    $template_add = $template_add . '<input type="checkbox" name="custom_sizes[]" value="' . $custom . '">' . $custom . '<br />';
  }
  
  $template->append('element_set_global_plugins_actions', array(
      'ID' => 'GenCustomDerivatives', 
      'NAME' => l10n('Pre-cache custom size derivatives'), 
      'CONTENT' => $template_add,
  ));
}

function bm_gen_custom_derivatives_action($action, $collection) {
  if ($action == 'GenCustomDerivatives'){
    global $page;
    foreach ($collection as $image_id){
      if (isset($_POST['custom_sizes'])) {
        $http_opts = array('http' => array('header' => 'Referer: '.get_absolute_root_url()));
        $http_context = stream_context_create($http_opts);
        
        $result = file_get_contents(get_absolute_root_url() .'ws.php?format=json&method=pwg.images.getInfo&image_id=' . $image_id);
        $json_result = json_decode($result);
        $sq_url = $json_result->result->derivatives->square->url;
        
        foreach ($_POST['custom_sizes'] as $size) {
          $url = str_replace('sq.jpg', 'cu_' . $size . '.jpg', $sq_url);
          file_get_contents($url, NULL, $http_context);
        }
      }
    }
  }
}

This will add a new batch manager action called "Pre-cache custom size derivatives".

Be aware that
a) there is a "chicken & egg problem", because templates can define custom derivatives, so you need to open the pages once to get the custom size registered in the database. Afterwards they show up in that list in batch manager.
b) for each selected image in batch manager, it calls the webservice API to get the URL, and then requests the custom derivative's URL. This process is DEAD SLOW, so you might have to raise "max_execution_time" parameter in php.ini up to several minutes/hours..

Disclaimer: I'm a sys admin, not a PHP programmer. There might be a way more effective solution, code whise :-)

Last edited by teekay (2016-07-06 16:04:26)

Offline

 

#7 2016-07-08 11:39:27

palantus
Member
2016-07-05
4

Re: Generation of custom size thumbnails

Thanks a lot! I will test it as soon as it get have time :)

Offline

 

#8 2016-07-19 00:46:51

chulann
Member
2016-07-17
3

Re: Generation of custom size thumbnails

I've given that a go but when I try and generate the images, I get the following error:


Warning: file_get_contents(http://SITE_ADDRESS/ws.php?format=json&method=pwg.images.getInfo&image_id=IMAGE_NUMBER): failed to open stream: HTTP request failed! HTTP/1.0 401 Access denied in /var/www/piwigo/plugins/PersonalPlugin/main.inc.php on line 36

If I go to that URL in my webrowser, it works but presumably it's getting the 401 because the HTTP request is coming from my server, rather than my logged in client? Or is it that it's only intended to work if you're running the client on the same host as the website itself?

Offline

 

#9 2016-07-19 09:53:07

teekay
Member
2013-06-12
404

Re: Generation of custom size thumbnails

chulann wrote:

I've given that a go but when I try and generate the images, I get the following error:


Warning: file_get_contents(http://SITE_ADDRESS/ws.php?format=json&method=pwg.images.getInfo&image_id=IMAGE_NUMBER): failed to open stream: HTTP request failed! HTTP/1.0 401 Access denied in /var/www/piwigo/plugins/PersonalPlugin/main.inc.php on line 36

If I go to that URL in my webrowser, it works but presumably it's getting the 401 because the HTTP request is coming from my server, rather than my logged in client? Or is it that it's only intended to work if you're running the client on the same host as the website itself?

Hi,
this is a batch manager action. So, yes, you need to be logged in as admin and use it through the site's batch manager in the admin section.

But there is a catch. Your PHP installation needs to have fopen wrappers enabled. Please check Administration (the initial overview page) -> Server environment -> PHP show info link -> Core -> allow_url_fopen. It should say "On". If that's "Off", you can try to modify the plugin code as follows:

Code:

...
function bm_gen_custom_derivatives_action($action, $collection) {
   ini_set('allow_url_fopen', '1'); //  <-- add this line here
   ...

If that doesn't work either, I could rewrite the plugin to use cURL instead of file_get_contents().

Last edited by teekay (2016-07-19 10:01:25)

Offline

 

#10 2016-07-26 00:52:55

chulann
Member
2016-07-17
3

Re: Generation of custom size thumbnails

Hi,
I did have a look and the allow_url_fopen is on.

From my point of view, it's a matter of where I'm logged in from... The ws.php script requires you to be logged into the administration panel - which sets a cookie on the browser you're accessing from. Then each time you access ws.php, the cookie is requested and given, allowing you to auth.

However, since you're using file_get_contents(), the HTTP request for ws.php comes from the server, rather than your webrowser - and there is no cookie to send. So unless you either auth from the website process or otherwise transfer the cookie, the ws.php script will never give you the details.

Unless I'm missing something?

Offline

 

#11 2016-07-26 15:41:18

teekay
Member
2013-06-12
404

Re: Generation of custom size thumbnails

chulann wrote:

Hi,
I did have a look and the allow_url_fopen is on.

From my point of view, it's a matter of where I'm logged in from... The ws.php script requires you to be logged into the administration panel - which sets a cookie on the browser you're accessing from. Then each time you access ws.php, the cookie is requested and given, allowing you to auth.

However, since you're using file_get_contents(), the HTTP request for ws.php comes from the server, rather than your webrowser - and there is no cookie to send. So unless you either auth from the website process or otherwise transfer the cookie, the ws.php script will never give you the details.

Unless I'm missing something?

I see what you mean. Are some or all of your pictures in private albums? That could cause the auth issue (mine are all public here)

Maybe passing through the client browser's cookie could help, like

Code:

....

function bm_gen_custom_derivatives_action($action, $collection) {
  if ($action == 'GenCustomDerivatives'){
    global $page;
    foreach ($collection as $image_id){
      if (isset($_POST['custom_sizes'])) {
        $http_opts = array('http' => array('header' => 'Referer: ' . get_absolute_root_url() . '\r\n' . 
                                                'Cookie: ' . $_SERVER['HTTP_COOKIE'] . '\r\n'));
        $http_context = stream_context_create($http_opts);
        
        $result = file_get_contents(get_absolute_root_url() .'ws.php?format=json&method=pwg.images.getInfo&image_id=' . $image_id, NULL, $http_context);
        $json_result = json_decode($result);
        $sq_url = $json_result->result->derivatives->square->url;
        
        foreach ($_POST['custom_sizes'] as $size) {
          $url = str_replace('sq.jpg', 'cu_' . $size . '.jpg', $sq_url);
          file_get_contents($url, NULL, $http_context);
        }
      }
    }
  }
}

Untested ^^

Last edited by teekay (2016-07-26 15:42:10)

Offline

 

#12 2016-09-04 18:05:48

Jessie
Member
2016-09-04
31

Re: Generation of custom size thumbnails

teekay wrote:

This will add a new batch manager action called "Pre-cache custom size derivatives".

Be aware that
a) there is a "chicken & egg problem", because templates can define custom derivatives, so you need to open the pages once to get the custom size registered in the database. Afterwards they show up in that list in batch manager.
b) for each selected image in batch manager, it calls the webservice API to get the URL, and then requests the custom derivative's URL. This process is DEAD SLOW, so you might have to raise "max_execution_time" parameter in php.ini up to several minutes/hours..

Disclaimer: I'm a sys admin, not a PHP programmer. There might be a way more effective solution, code whise :-)

Hello,

I am interested in generating the thumbnails as well but I haven't been able to see the "Pre-cache custom size derivatives" in action list in batch manager.
What exactly should I do about this "chicken & egg" problem ?

Thanks

Offline

 

#13 2016-09-04 20:38:10

teekay
Member
2013-06-12
404

Re: Generation of custom size thumbnails

Jessie wrote:

teekay wrote:

This will add a new batch manager action called "Pre-cache custom size derivatives".

Be aware that
a) there is a "chicken & egg problem", because templates can define custom derivatives, so you need to open the pages once to get the custom size registered in the database. Afterwards they show up in that list in batch manager.
b) for each selected image in batch manager, it calls the webservice API to get the URL, and then requests the custom derivative's URL. This process is DEAD SLOW, so you might have to raise "max_execution_time" parameter in php.ini up to several minutes/hours..

Disclaimer: I'm a sys admin, not a PHP programmer. There might be a way more effective solution, code whise :-)

Hello,

I am interested in generating the thumbnails as well but I haven't been able to see the "Pre-cache custom size derivatives" in action list in batch manager.
What exactly should I do about this "chicken & egg" problem ?

Thanks

If it doesn't show up in batch manager you either forgot to enable the "Personal Plugin" in plugins, or did something wrong with pasting the code.

For the "chicken & egg" problem, just open at least one single album once so it registers the custom derivatives.

Offline

 

#14 2016-09-04 20:46:05

Jessie
Member
2016-09-04
31

Re: Generation of custom size thumbnails

That was it thanks !

However I got the same error as above (tested with the patch you provided as well):
Warning: file_get_contents(https://piwigo.example.com/ws.php?forma … mage_id=61): failed to open stream: HTTP request failed! HTTP/1.1 401 Unauthorized in /var/www/piwigo/plugins/PersonalPlugin/main.inc.php on line 38

The website is currently behind a .htaccess password protection, could it be the issue ?

Thans

Offline

 

#15 2016-09-04 20:56:59

teekay
Member
2013-06-12
404

Re: Generation of custom size thumbnails

Jessie wrote:

That was it thanks !

However I got the same error as above (tested with the patch you provided as well):
Warning: file_get_contents(https://piwigo.example.com/ws.php?forma … mage_id=61): failed to open stream: HTTP request failed! HTTP/1.1 401 Unauthorized in /var/www/piwigo/plugins/PersonalPlugin/main.inc.php on line 38

The website is currently behind a .htaccess password protection, could it be the issue ?

Thans

The patch surely won't work with HTTP Basic Auth. It _could_ fix things when using Piwigo's standard authentication, but not sure, still untested.

Offline

 

Board footer

Powered by FluxBB

github twitter facebook google+ newsletter Donate Piwigo.org © 2002-2019 · Contact