Piwigo.org

drizbach · 2010-08-07 09:34:46

Hi,

I have spent several hour trying to upload images via pLoader to my new gallery and unfortunately without success. The pLoader tries to upload image and then window pops up (behind main window) that there was an error and the image was not uploaded. I have tried to do it via web and the flash plugin didn't work as well. The only way how I was able to upload image was via old form. After that I was able to update image via pLoader.

Piwigo version
* Piwigo 2.1.2
Environment (Hosting)
* Operating system: Linux
* PHP: 5.3.2 (Show info) [2010-08-07 09:28:59]
* mysql: 5.0.90-log [2010-08-07 09:28:59]

I did little investigation and found with wireshark that there is an error:
3f
{"stat":"fail","err":1002,"message":"Missing parameters: data"}
0

Any help would be appreciated.
Petr (drizbach)

drizbach · 2010-08-09 20:19:58

Hi,

so i did a more digging, because it looked strange to me that i do not see the data parameter in the content of the POST message. Well it was compressed, so now i can see the parameters:

position=1&type=file&data=%2F9j...method=pwg.images.addChunk&original_sum=f0ca841c61f39fe2486e4d977d495a98

After this request the already mentioned error comes back.
In the upload/buffer folder there is a block file for thumbnail.

I turned on logging in piwigo and this is content of the log:
[2010-08-09T20:14:07+02:00] [ws_images_add_chunk] welcome
[2010-08-09T20:14:07+02:00] [ws_images_add_chunk] input param "position" : "1"
[2010-08-09T20:14:07+02:00] [ws_images_add_chunk] input param "type" : "thumb"
[2010-08-09T20:14:07+02:00] [ws_images_add_chunk] input param "original_sum" : "f0ca841c61f39fe2486e4d977d495a98"
[2010-08-09T20:14:07+02:00] [ws_images_add_chunk] data length : 7255
[2010-08-09T20:14:07+02:00] [pwg.images.add] input param "name" : "DotÄ›rnĂ psy"
[2010-08-09T20:14:07+02:00] [pwg.images.add] input param "author" : "Petr MatÄ›jka"
[2010-08-09T20:14:07+02:00] [pwg.images.add] input param "categories" : "3"
[2010-08-09T20:14:07+02:00] [pwg.images.add] input param "file_sum" : "f0ca841c61f39fe2486e4d977d495a98"
[2010-08-09T20:14:07+02:00] [pwg.images.add] input param "thumbnail_sum" : "dbc0e5671508f9b5430ab31d63cf9a87"
[2010-08-09T20:14:07+02:00] [pwg.images.add] input param "original_sum" : "f0ca841c61f39fe2486e4d977d495a98"
[2010-08-09T20:14:07+02:00] [pwg.images.add] input param "original_filename" : "IMG_8720.jpg"
[2010-08-09T20:14:07+02:00] [pwg.images.add] input param "tag_ids" : ""
[2010-08-09T20:14:07+02:00] [pwg.images.add] input param "date_creation" : "2008:09:17"
[2010-08-09T20:14:07+02:00] [pwg.images.add] input param "comment" : ""
[2010-08-09T20:14:07+02:00] [pwg.images.add] input param "high_sum" : "NULL"
[2010-08-09T20:14:07+02:00] [pwg.images.add] input param "level" : "0"
[2010-08-09T20:14:07+02:00] [add_file] file_path : ./upload/2010/08/09/20100809201407-f0ca841c.jpg
[2010-08-09T20:14:07+02:00] [add_file] upload_dir : ./upload/2010/08/09
[2010-08-09T20:14:07+02:00] [merge_chunks] input parameter $output_filepath : ./upload/2010/08/09/20100809201407-f0ca841c.jpg
[2010-08-09T20:14:07+02:00] [merge_chunks] memory_get_usage before loading chunks: 1473440
[2010-08-09T20:14:07+02:00] [merge_chunks] memory_get_usage after loading chunks: 1473520

Any help would be appreciated, but i will try to dig in deeper, maybe I'll be lucky.

Petr

drizbach · 2010-08-09 20:32:42

Little more digging with wireshark and I see some php errors in last message:

Warning: chmod() [function.chmod]: Unable to access ./upload/2010/08/09/20100809202127-f0ca841c.jpg in /.../gallery/include/ws_functions.inc.php on line 1099
Warning: chmod() [function.chmod]: stat failed for ./upload/2010/08/09/20100809202127-f0ca841c.jpg in /.../gallery/include/ws_functions.inc.php on line 1099
Warning: md5_file() [function.md5-file]: Unable to access ./upload/2010/08/09/20100809202127-f0ca841c.jpg in /.../gallery/include/ws_functions.inc.php on line 1102
Warning: md5_file(./upload/2010/08/09/20100809202127-f0ca841c.jpg) [function.md5-file]: failed to open stream: No such file or directory in /.../gallery/include/ws_functions.inc.php on line 1102
Warning: Cannot modify header information - headers already sent by (output started at /.../gallery/include/ws_functions.inc.php:1099) in /.../gallery/include/functions_html.inc.php on line 483

Could this cause the problem?

Last edited by drizbach (2010-08-10 10:23:50)

drizbach · 2010-08-09 22:14:36

I did little more debugging but so far I haven't found reason why is it not working.

I have decided to log the stack trace at point at which the exception 1002. Then I added logging of the parameters in rest_handler.php and this is the output (the PwgServer details were removed).
The last enry is teh stack trace and the previous are parameters from rest_handler.php.

[2010-08-09T22:02:29+02:00] DEBUG: array (
'format' => 'json',
'method' => 'pwg.images.checkUpload',
)
[2010-08-09T22:02:29+02:00] DEBUG: array (
'method' => 'pwg.images.exist',
'md5sum_list' => 'f0ca841c61f39fe2486e4d977d495a98',
)
[2010-08-09T22:02:29+02:00] DEBUG: array (
'method' => 'pwg.images.exist',
'md5sum_list' => 'f0ca841c61f39fe2486e4d977d495a98',
)
[2010-08-09T22:02:32+02:00] DEBUG: array (
'position' => '1',
'type' => 'file',
'method' => 'pwg.images.addChunk',
'original_sum' => 'f0ca841c61f39fe2486e4d977d495a98',
)
[2010-08-09T22:02:32+02:00] DEBUG: array (
0 =>
array (
'file' => '/.../gallery/include/ws_protocols/rest_handler.php',
'line' => 53,
'function' => 'invoke',
'class' => 'PwgServer',
'object' =>
PwgServer::__set_state(),
'type' => '->',
'args' =>
array (
0 => 'pwg.images.addChunk',
1 =>
array (
'position' => '1',
'type' => 'file',
'original_sum' => 'f0ca841c61f39fe2486e4d977d495a98',
),
),
),
1 =>
array (
'file' => '/.../gallery/include/ws_core.inc.php',
'line' => 361,
'function' => 'handleRequest',
'class' => 'PwgRestRequestHandler',
'object' =>
PwgRestRequestHandler::__set_state(array(
)),
'type' => '->',
'args' =>
array (
0 =>
PwgServer::__set_state(),
),
),
2 =>
array (
'file' => '/.../gallery/ws.php',
'line' => 398,
'function' => 'run',
'class' => 'PwgServer',
'object' =>
PwgServer::__set_state(),
'type' => '->',
'args' =>
array (
),
),
)

The only thing I am not sure off, is what is causing the upload to fail.

Last edited by drizbach (2010-08-10 10:22:45)

drizbach · 2010-08-10 10:06:39

OK It seems I have found the problem. In rest_handler.php the PwgRestRequestHandler.handleRequest method is using $_POST to get the parameters. But there is the data parameter missing.
[2010-08-10T09:52:51+02:00] DEBUG REQ: array (
'position' => '1',
'type' => 'file',
'method' => 'pwg.images.addChunk',
'original_sum' => 'f0ca841c61f39fe2486e4d977d495a98',
)
[2010-08-10T09:52:51+02:00] DEBUG REQ: position=1&type=file&data=...&method=pwg.images.addChunk&original_sum=f0ca841c61f39fe2486e4d977d495a98

the first entry is generated by
ws_logfile('DEBUG REQ: '.var_export($param_array, true));
the second by
ws_logfile('DEBUG REQ: '.file_get_contents("php://input"));

I have found a bug 22427 on php.net. I will try to find or implement a script that would work around it and parse the raw data into the same array structure as $_POST uses.

Petr

drizbach · 2010-08-10 10:22:02

OK I have workaround the issue. The problem is the length of the variable data. If it is too long it will not be parsed into $_POST. So in pLoader setting the transfer chunk size to 5000 fixed the problem and the data parsed and pictures can be loaded.

I suppose the flash plugin uses same logic as pLoader and therefore it will not work for me, but I'm happy with the pLoader now working.

If I ever find a simple script that would correctly parse the raw input into the same $_POST structure I will try it and post here.

I hope my findings will help others that might suffer from similar issue.

Petr

drizbach · 2010-08-10 15:29:29

So I have found the solution. I haven't yet integrated in the rest_handler.php. I'll do that in the evening and post the whole change where I will try to detect the missing parameter in $_POST by comparing the content length.

I added following debug statement into the mentioned file after the $param_array was filled:

Code:

    if ($service->isPost()) {
      ws_logfile('DEBUG REQ: '.var_export($param_array, true));
      $newParam_array = array();
      parse_str(file_get_contents("php://input"), $newParam_array);
      ws_logfile('DEBUG REQ: '.var_export($newParam_array, true));
    }

and here are the log entries:

Code:

[2010-08-10T15:11:44+02:00] DEBUG REQ: array (
  'position' => '1',
  'type' => 'file',
  'data' => '/9j/4AAQSkZJRgABAQEASABIAAD/4TuQRXhpZgAASUkqAAgAAAAJAA8BA...',
  'method' => 'pwg.images.addChunk',
  'original_sum' => '1b46d77ec0ab75c80b35d598105658e0',
)
[2010-08-10T15:11:44+02:00] DEBUG REQ: array (
  'position' => '1',
  'type' => 'file',
  'data' => '/9j/4AAQSkZJRgABAQEASABIAAD/4TuQRXhpZgAASUkqAAgAAAAJAA8BA...',
  'method' => 'pwg.images.addChunk',
  'original_sum' => '1b46d77ec0ab75c80b35d598105658e0',
)

If the data parameter is too long it is missing in the first entry, but present in the second one.

I do not know how much this line decreases performance and increases memory usage:

Code:

      parse_str(file_get_contents("php://input"), $newParam_array);

Maybe someone else can tell me.

In case it is not by much I will just change the

Code:

    $param_array = $service->isPost() ? $_POST : $_GET;

to

Code:

      $param_array = array();
      if ($service->isPost()) {
        parse_str(file_get_contents("php://input"), $param_array);
      } else {
        $param_array = $_GET;
      }

My guess is that this will also work for the flash plugin if it is using the same approach as pLoader to upload images.

Regards,
Petr

PS: I just replaced the code and I can use chunk size 500000. Everything is working and image is uploaded in 4 seconds.

Last edited by drizbach (2010-08-10 15:41:54)

plg · 2010-08-13 12:39:22

(I'm just back from holidays and I wrote the pwg.images.add web API method)

Very interesting topic drizbach, thank you for you deep investigation!

Do you have any "secure" settings such as the suhosin PHP extension? (that limits the length of the input parameters)

drizbach wrote:
My guess is that this will also work for the flash plugin if it is using the same approach as pLoader to upload images.

No, it doesn't work the same way at all. Obviously you're skilled enough for deeper explanations, so let's go into details:

pLoader communicates with Piwigo thanks to the web API and pwg.images.add (+pwg.images.addChunk) are the most important methods pLoader calls. pLoader is "chunking" the photo into small pieces and send them in the POST parameters with multiple HTTP requests.

The Flash uploader (and the "old style" HTML form too) is using the multipart/form-data encoding to transmit the files. No chunking and the file is received in the $_FILES array, which is a much more common way for transmitting a file. The limits are different.

I have recently added a new method [wiki] pwg.images.addSimple in Piwigo API (you have to install [extension by plg] pwg.images.addSimple) that uses the same multipart/form-data encoding. pLoader won't use it because it's currently less powerful than pwg.images.add but it's much easier to use for other photo manager sch as Lightroom, iPhoto and so on.

drizbach · 2010-08-13 21:40:46

Hi plg,
you are correct, my provider uses suhosin extension and from the phpinfo page i see there is a limit on post: suhosin.post.max_value_length = 65000. You can see all the configuration on phpinfo.php page I created. It is interesting that I have found a way to work around the suhosin restriction.

Thanks for the explanation of how the flash works. I will test it over the weekend to see if it is working or if there is another limit that is causing problems.

Thanks for the addSimple method information. I'm thinking about using it in a script to migrate my old gallery to piwigo.

To be hones I am not a PHP programmer (but I know a little), I'm more into Delphi, Java, C#, Qore :) Anyway I like challenge and I try to debug things if they are not working as they should have been.

So far I really like Piwigo gallery, it is simple with a lot of features and with pLoader it is very simple to load the images. I'm already suggesting the gallery to friend of mine to use it for their pioneer organization for summer camp photos.

plg · 2010-08-13 23:12:06

drizbach wrote:
It is interesting that I have found a way to work around the suhosin restriction.

Yes, that is very interesting. I have another problem related to suhosin (on the "random pictures" page) but that's another story.

drizbach wrote:
Thanks for the explanation of how the flash works. I will test it over the weekend to see if it is working or if there is another limit that is causing problems.

Ok, keep us informed.

drizbach wrote:
Thanks for the addSimple method information. I'm thinking about using it in a script to migrate my old gallery to piwigo.

What is your old gallery? a specific coding or something similar to Piwigo?

drizbach wrote:
To be hones I am not a PHP programmer (but I know a little), I'm more into Delphi, Java, C#, Qore :) Anyway I like challenge and I try to debug things if they are not working as they should have been.

PHP is very easy to work with (too easy some coders may say) and obviously, you've found your way to debug efficiently! Thank you for the time you spent debugging pLoader/Piwigo communication, this is really not the simplest piece of code in Piwigo.

drizbach wrote:
So far I really like Piwigo gallery, it is simple with a lot of features and with pLoader it is very simple to load the images. I'm already suggesting the gallery to friend of mine to use it for their pioneer organization for summer camp photos.

Nice to read such a user feedback :-)

drizbach · 2010-08-14 00:10:29

plg wrote:
What is your old gallery? a specific coding or something similar to Piwigo?

I have used SPGM (Simple Picture Gallery Manager). The main reason was that I had only little space on my first hosting, safe-mode turned on and only limited time to take care of it and search for something better.

Recently I got more free time so I decided I need to find something better. I was going to give gallery2
(or gallery3) a try, but then I have found Piwigo and I really liked the demo so I decided to try it and so far I do not regret my decision.

plg · 2010-08-15 16:42:42

drizbach wrote:
I have used SPGM (Simple Picture Gallery Manager).

Latest release on 2007... doesn't seem to be very "living" nowadays :-/

drizbach wrote:
Recently I got more free time so I decided I need to find something better. I was going to give gallery2 (or gallery3) a try, but then I have found Piwigo and I really liked the demo so I decided to try it and so far I do not regret my decision.

Welcome to the Piwigo Community drizbach!

Piwigo.org

Announcement

#1 2010-08-07 09:34:46

pLoader unable to upload images

#2 2010-08-09 20:19:58

Re: pLoader unable to upload images

#3 2010-08-09 20:32:42

Re: pLoader unable to upload images

#4 2010-08-09 22:14:36

Re: pLoader unable to upload images

#5 2010-08-10 10:06:39

Re: pLoader unable to upload images

#6 2010-08-10 10:22:02

Re: pLoader unable to upload images

#7 2010-08-10 15:29:29

Re: pLoader unable to upload images

Code:

Code:

Code:

Code:

Code:

#8 2010-08-13 12:39:22

Re: pLoader unable to upload images

drizbach wrote:

#9 2010-08-13 21:40:46

Re: pLoader unable to upload images

#10 2010-08-13 23:12:06

Re: pLoader unable to upload images

drizbach wrote:

drizbach wrote:

drizbach wrote:

drizbach wrote:

drizbach wrote:

#11 2010-08-14 00:10:29

Re: pLoader unable to upload images

plg wrote:

#12 2010-08-15 16:42:42

Re: pLoader unable to upload images

drizbach wrote:

drizbach wrote:

Board footer