Announcement

#1 2021-02-06 12:03:05

graio
Member
2021-02-06
2

Upload using uploadAsync / piwigo mobile doesn't work most of the time

Hello,

I've updated to Piwigo 11 recently. After updating, I could no longer upload photos using the Piwigo Mobile app. When I keep trying it works sometimes, but most of the times it just won't upload and gives an 'Access Denied' error in the upload queue in the mobile app.

In my journey to identify the issue, I've started by installing a brand new installation of Piwigo 11.3.0 and Piwigo Mobile 2.5.4 (iOS).

The setup consist of:
- Piwigo 11.3
- Nginx (tried Apache before, same issue)
- PHP-FPM 7.3
- extension: piwigo-videojs

Uploading of photos and videos in the webinterface works as expected, without issues.
Uploading of photos and videos in the mobile app only work after multiple retries and with a good amount of luck.

In the PHP error log, I don't see any errors during upload.

In the Piwigo log, I see what looks likes it is waiting for chunks to be uploaded, but no errors:

Code:

[2021-02-06 10:44:06]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/d3eba4dabfdb198daece3c563833059d-u1-009of023.chunk
[2021-02-06 10:44:06]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-02-06 10:44:08]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/d3eba4dabfdb198daece3c563833059d-u1-003of023.chunk
[2021-02-06 10:44:08]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-02-06 10:44:09]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/d3eba4dabfdb198daece3c563833059d-u1-022of023.chunk
[2021-02-06 10:44:09]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now

It just keeps on waiting for chunks, which seems correct because not all chunks are uploaded at that time. However the Mobile app already shows 'Access Denied' at that point.

To find out what the Mobile app is doing, I've routed the traffic through a sniffing proxy. This shows me that the app is trying to upload all chunks, but most of them are returned a 401 error by the server:

The first four request seem to be working, they return the following response:

Code:

{"stat":"ok","result":{"message":"chunks uploaded = 1,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23"}}

The next requests are failing with the following response:

Code:

{"stat":"fail","err":401,"message":"Access denied"}

I've tried to examine the differences between the successful and failed requests. The only thing that I can see is that all successful request have a "Kept Alive: No" property, while the failed requests are showing "Kept Alive: Yes".

So i've turned off the keepalive in Nginx (keepalive_timeout 0;), which I can verify in the sniffer. Now all requests show as "Kept Alive: No", but still most of them fail. So this didn't seem to be the issue.

Right now I'm out of ideas. I don't know what I can do to debug further.

If I resume the upload enough times, at some point all chunks are uploaded and the file will be processed normally.

Maybe some of you guys have an idea what could be wrong. Since i've tried a brand new install I can't be the only one with this issue right?

Thanks!

Last edited by graio (2021-02-06 12:27:39)

Offline

 

#2 2021-02-08 19:08:36

cryopad
Piwigo Team
Grenoble, France
2015-11-01
190

Re: Upload using uploadAsync / piwigo mobile doesn't work most of the time

Hi @graio,
A new version of the iOS app is available for beta testing through TestFlight.
Could you please give it a try?


If you enjoy Piwigo for iOS…
➤ We would appreciate an AppStore review.
➤ Comments, suggestions ? Tell us on GitHub…
➤ Not yet available in your language ? Help us translate it…

Offline

 

#3 2021-02-09 00:32:33

graio
Member
2021-02-06
2

Re: Upload using uploadAsync / piwigo mobile doesn't work most of the time

Thanks for reaching out. I've tried the beta but unfortunately this version did not fixed the issue.

Using the beta app I've tried to reproduce the problem while watching the traffic in mitmproxy. I found something that I did not see before.

As an example:

Code:

Request 1:
- Request cookie: pwg_id=oe1p6a535vtpsithr6q4gue7pb
- Response: 200 {"stat":"ok","result":{"message":"chunks uploaded [...] "}}
- Response Set-cookie: pwg_id=mud6nl0ui2bol6c0knckrjqh1k; path=/; HttpOnly

Request 2: 
- Request cookie: pwg_id=oe1p6a535vtpsithr6q4gue7pb
- Response: 401 {"stat":"fail","err":401,"message":"Access denied"}
- No response Set-cookie.

Request 3:
- Request cookie: pwg_id=mud6nl0ui2bol6c0knckrjqh1k
- Response: 200 {"stat":"ok","result":{"message":"chunks uploaded [...] "}}
- Reponse Set-cookie: pwg_id=r2lhdakco29gd3evj7ld1au698; path=/; HttpOnly

And this is basically what is happening all the time. So the first request is successful, but gets a new session_id returned by Piwigo. The app still uses the old session_id on the next request(s). Maybe because those requests are already fired since it is asynchronous? And as soon as the next request uses the new session_id as a cookie, it is successful and it starts all over again.

The app only retries a certain amount of time, and if not all chunks are uploaded successful within that time, it simply fails until retry.

I also find how this problem could have occured, it seems that this change in 11.0 made this problem occur: https://github.com/Piwigo/Piwigo/pull/1178/files
The session regeneration on login was not active in PHP 7 before Piwigo 11.0. And I am using PHP 7.

As soon as I removed this session_regeneration line from the code, it all started working again.
But this was a security measure, and the fact that it wasn't working in PHP 7 before is a bug. So it should be enabled..

I think the conclusion is that Piwigo Mobile is no longer able to handle the sessions correctly. It should either use the returned session cookies in the correct order. (But I'm not sure if that is possible with the current asynchronous calls) Or it should not create a new user session (login) in Piwigo on every single request as it does now.

That last one is probably the best option, but it seems to be on the backlog already: [Github] Piwigo-Mobile issue #314

Not sure where to go from here, but at least I have workaround and can use Piwigo again :)

Offline

 

#4 2021-02-13 19:27:12

cryopad
Piwigo Team
Grenoble, France
2015-11-01
190

Re: Upload using uploadAsync / piwigo mobile doesn't work most of the time

Hi @graio,

I have crosschecked and the beta version of the iOS app sends uploadAsync requests without cookie. The chunks of each uploaded photo/video are uploaded asynchronously and for each chunk the server returns the following cookie:

"Set-Cookie" =     (
        "pwg_id=161rooo9tbhpbd4c2gjubrblv1; path=/Piwigo/; HttpOnly; secure",
        "pwg_remember=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/Piwigo/; secure"
    );

where:
- pwg_id specifies the name of the session which is used as cookie name
- pwg_remember specifies the name of the cookie used to stay logged -> deleted!

Have you checked the logs of modSecurity? or any other security tool activated on your server?


If you enjoy Piwigo for iOS…
➤ We would appreciate an AppStore review.
➤ Comments, suggestions ? Tell us on GitHub…
➤ Not yet available in your language ? Help us translate it…

Offline

 

#5 2021-03-03 19:09:13

cryopad
Piwigo Team
Grenoble, France
2015-11-01
190

Re: Upload using uploadAsync / piwigo mobile doesn't work most of the time

The issue is reported here: [Github] Piwigo issue #1344 and can be temporarily fixed after disabling the following line in LocalEditor:

Code:

$conf['guest_access'] = false;

If you enjoy Piwigo for iOS…
➤ We would appreciate an AppStore review.
➤ Comments, suggestions ? Tell us on GitHub…
➤ Not yet available in your language ? Help us translate it…

Offline

 

#6 2021-11-19 18:10:08

endrju83
Member
2020-10-11
11

Re: Upload using uploadAsync / piwigo mobile doesn't work most of the time

Hi, it seems that this problem is still not solved.

I have updated from Piwigo 11 to 12.1 hoping that it could solve the issue but not.
My iOS mobile app is also up to date with version 2.8.1.

I allso have to multiply retry the uploading of single video file so at the end of many tries all chunks gets finaly uploaded (iOS device).

I have also tried to disable this configuration

Code:

$conf['guest_access'] = false;

but it does not seems to have any effect.

I also after several re-tries of uploading my video file am not able to visit my Galery in Web browser. It tries to reload the page for several minutes until I cancel it or it returns error 503. If that happens I am also not able to re-log in to Mobile App for next XX minutes.

Is it possible that the request to upload the files consumes all avaiable PHP processes (my server gives my only 7 or 10, not sure) and then because of that it failes and gives me error 503 and also maybe to the mobile app?

This is just an example of the log:

Code:

[2021-11-19 17:54:24]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-032of186.chunk
[2021-11-19 17:54:24]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-034of186.chunk
[2021-11-19 17:54:24]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-11-19 17:54:24]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-050of186.chunk
[2021-11-19 17:54:24]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-11-19 17:54:24]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now

It is difficult with my hosting to get to the Apache/PHP error logs.
All I am able to retrieve is a log like this one (I do not understand what it means with the FCGI). The only content which repeats all the time is this (but I am suspicious that this is from my access to FTP server because I can see my FTP user name there in the brackets):

Code:

[Fri Nov 19 17:55:27.415312 2021] [proxy:error] [pid 49434:tid 140393384335104] (70007)The timeout specified has expired: AH00941: FCGI: failed to acquire connection for (my_FTP_username)
[Fri Nov 19 17:55:27.431637 2021] [proxy:error] [pid 49434:tid 140393401120512] (70007)The timeout specified has expired: AH00941: FCGI: failed to acquire connection for (my_FTP_username)
[Fri Nov 19 17:55:27.432653 2021] [proxy:error] [pid 49434:tid 140394030573312] (70007)The timeout specified has expired: AH00941: FCGI: failed to acquire connection for (my_FTP_username)
[Fri Nov 19 17:55:27.433029 2021] [proxy:error] [pid 49434:tid 140393736828672] (70007)The timeout specified has expired: AH00941: FCGI: failed to acquire connection for (my_FTP_username)
[Fri Nov 19 17:55:27.434937 2021] [proxy:error] [pid 49434:tid 140393820755712] (70007)The timeout specified has expired: AH00941: FCGI: failed to acquire connection for (my_FTP_username)
[Fri Nov 19 17:55:27.439932 2021] [proxy:error] [pid 49434:tid 140393375942400] (70007)The timeout specified has expired: AH00941: FCGI: failed to acquire connection for (my_FTP_username)
[Fri Nov 19 17:55:27.449591 2021] [proxy:error] [pid 49434:tid 140393342371584] (70007)The timeout specified has expired: AH00941: FCGI: failed to acquire connection for (my_FTP_username)
[Fri Nov 19 17:55:27.461886 2021] [proxy:error] [pid 49434:tid 140393543796480] (70007)The timeout specified has expired: AH00941: FCGI: failed to acquire connection for (my_FTP_username)
[Fri Nov 19 17:56:35.531182 2021] [proxy_fcgi:error] [pid 37482:tid 140393501832960] (70007)The timeout specified has expired: [id YZfUF8AvMYeMzA-ybGHI-AAAAB0] [client 193.165.97.84:52643] AH01075: Error dispatching request to : (reading input brigade)AH01075: Error dispatching request to : (reading input brigade)
[Fri Nov 19 17:56:47.655006 2021] [proxy_fcgi:error] [pid 45655:tid 140394156463872] (70007)The timeout specified has expired: [id YZfWdTiZk-y9iCK7zvgoaQAAAFU] [client 193.165.97.84:52649] AH01075: Error dispatching request to : (reading input brigade)AH01075: Error dispatching request to : (reading input brigade)
[Fri Nov 19 17:56:47.672241 2021] [proxy_fcgi:error] [pid 45655:tid 140394097714944] (70007)The timeout specified has expired: [id YZfWdTiZk-y9iCK7zvgoYgAAAEc] [client 193.165.97.84:52649] AH01075: Error dispatching request to : (reading input brigade)AH01075: Error dispatching request to : (reading input brigade)
[Fri Nov 19 17:56:47.698688 2021] [proxy_fcgi:error] [pid 45655:tid 140394282354432] (70007)The timeout specified has expired: [id YZfWdjiZk-y9iCK7zvgoegAAAHk] [client 193.165.97.84:52649] AH01075: Error dispatching request to : (reading input brigade)AH01075: Error dispatching request to : (reading input brigade)
[Fri Nov 19 17:57:44.675836 2021] [proxy_fcgi:error] [pid 37481:tid 140393001113344] (70007)The timeout specified has expired: [id YZfWrDRHEZ9mSG-lcD1APwAAAEE] [client 193.165.97.84:52652] AH01075: Error dispatching request to : (reading input brigade)AH01075: Error dispatching request to : (reading input brigade)
[Fri Nov 19 17:57:44.752245 2021] [proxy_fcgi:error] [pid 37481:tid 140393361999616] (70007)The timeout specified has expired: [id YZfWrTRHEZ9mSG-lcD1AUwAAAGc] [client 193.165.97.84:52652] AH01075: Error dispatching request to : (reading input brigade)AH01075: Error dispatching request to : (reading input brigade)
[Fri Nov 19 17:57:44.918028 2021] [proxy_fcgi:error] [pid 37481:tid 140393085040384] (70007)The timeout specified has expired: [id YZfWrDRHEZ9mSG-lcD1ARgAAAE0] [client 193.165.97.84:52652] AH01075: Error dispatching request to : (reading input brigade)AH01075: Error dispatching request to : (reading input brigade)

Offline

 

#7 2021-11-19 19:04:29

erAck
Only trying to help
2015-09-06
1998

Re: Upload using uploadAsync / piwigo mobile doesn't work most of the time

endrju83 wrote:

Is it possible that the request to upload the files consumes all avaiable PHP processes (my server gives my only 7 or 10, not sure) and then because of that it failes and gives me error 503 and also maybe to the mobile app?

Yes, likely. 7 or 10 processes is very restrictive. Any other requests coming in to your web server at the same time will consume some of these as well, and if after multiple retries finally all chunks get uploaded it doesn't indicate a general failure other than your server being the bottleneck.

Additionally,

Code:

[Fri Nov 19 17:55:27.461886 2021] [proxy:error] [pid 49434:tid 140393543796480] (70007)The timeout specified has expired: AH00941: FCGI: failed to acquire connection for (my_FTP_username)
[Fri Nov 19 17:56:35.531182 2021] [proxy_fcgi:error] [pid 37482:tid 140393501832960] (70007)The timeout specified has expired: [id YZfUF8AvMYeMzA-ybGHI-AAAAB0] [client 193.165.97.84:52643] AH01075: Error dispatching request to : (reading input brigade)AH01075: Error dispatching request to : (reading input brigade)

these may indicate that the Apache Timeout (and possibly ProxyTimeout) values are too low, but it may also be just a consequence of running out of processes. When happening for uploads also possible are too low RequestReadTimeout values with mod_reqtimeout being used (though the upload in chunks should circumvent that).

Talk to your hoster. But likely, with too restrictive settings you'll need a better hosting plan, or a better hoster.


Running Piwigo at https://erack.net/gallery/

Offline

 

#8 2021-11-19 20:22:03

endrju83
Member
2020-10-11
11

Re: Upload using uploadAsync / piwigo mobile doesn't work most of the time

Yea, changing hosting setting is very restricted, so I am not sure if they will be willing to give more processes.

File size for upload is limited to 128M --> reducing count of chunks by increasing file size per chunk could also help - if that is even possible to change somewhere? Do you know where it would be save to change it?
Then the number of reuploads for the same file should go down... What do you think?

Offline

 

#9 2021-11-19 21:03:17

cryopad
Piwigo Team
Grenoble, France
2015-11-01
190

Re: Upload using uploadAsync / piwigo mobile doesn't work most of the time

The iOS app uses the chunk size set in the configuration of your Piwigo. By default, the size is of 500 kB. You can change this size by adding the line:

Code:

$conf['upload_form_chunk_size'] = 1000;

in the 'Local Config' section of the LocalFiles Editor plugin. The size must be provided in kB.

Increasing the size will decrease the number of chunks. Note that if the uploaded file is too large to be merged for the server, the problem will remain.

Hope this helps.


If you enjoy Piwigo for iOS…
➤ We would appreciate an AppStore review.
➤ Comments, suggestions ? Tell us on GitHub…
➤ Not yet available in your language ? Help us translate it…

Offline

 

#10 2021-11-19 21:15:52

endrju83
Member
2020-10-11
11

Re: Upload using uploadAsync / piwigo mobile doesn't work most of the time

Thank you, I will try that.

cryopad wrote:

Note that if the uploaded file is too large to be merged for the server, the problem will remain.

What do you mean by that - I though that even when my PHP upload file limit is 128M, I could upload even bigger files - because it is uploaded chunk by chunk and then it is merged into one file and saved.
Did I got it wrong? In that case - how it works then, if you can explain please?

I have also observed - and it seems strange to me, that when I reinitiate the upload again after it fails with "Uploading... Error", it tries to uploads all the chunks again, even those which were already uploaded?
Was it implemeted and is intended to be like that?

I would expect that it tries to upload only missing chunks... If it really have to reupload every chunk over and over again with each retry, then it may never upload all the chunks.

For example chunks 001, 002, 003 and 004 of 019 were already uploaded 4 times, was it necessary?

Code:

[2021-11-20 0:13:23]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-001of019.chunk
[2021-11-20 0:13:23]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-11-20 0:13:27]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-002of019.chunk
[2021-11-20 0:13:27]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-003of019.chunk
[2021-11-20 0:13:27]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-11-20 0:13:27]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-11-20 0:13:35]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-004of019.chunk
[2021-11-20 0:13:35]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-11-20 0:13:35]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-005of019.chunk
[2021-11-20 0:13:35]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-11-20 0:13:48]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-001of019.chunk
[2021-11-20 0:13:48]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-11-20 0:13:49]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-003of019.chunk
[2021-11-20 0:13:49]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-11-20 0:13:54]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-002of019.chunk
[2021-11-20 0:13:54]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-11-20 0:16:55]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-008of019.chunk
[2021-11-20 0:16:55]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-006of019.chunk
[2021-11-20 0:16:55]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-11-20 0:16:55]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-11-20 0:16:55]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-014of019.chunk
[2021-11-20 0:16:55]  [ERROR]  ws_images_uploadAsync ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-014of019.chunk MD5 checksum mismatched
[2021-11-20 0:17:11]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-004of019.chunk
[2021-11-20 0:17:11]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-005of019.chunk
[2021-11-20 0:17:11]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-11-20 0:17:11]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-009of019.chunk
[2021-11-20 0:17:11]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-11-20 0:17:11]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-11-20 0:17:18]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-003of019.chunk
[2021-11-20 0:17:18]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-002of019.chunk
[2021-11-20 0:17:18]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-11-20 0:17:18]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-11-20 0:17:18]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-001of019.chunk
[2021-11-20 0:17:18]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-11-20 0:17:27]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-004of019.chunk
[2021-11-20 0:17:27]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-11-20 0:17:27]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-006of019.chunk
[2021-11-20 0:17:27]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-11-20 0:17:28]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-005of019.chunk
[2021-11-20 0:17:28]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-11-20 0:18:23]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-003of019.chunk
[2021-11-20 0:18:23]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-11-20 0:18:25]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-001of019.chunk
[2021-11-20 0:18:25]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-11-20 0:18:25]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-002of019.chunk
[2021-11-20 0:18:25]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now
[2021-11-20 0:18:30]  [DEBUG]  ws_images_uploadAsync uploaded ./upload/buffer/c78d95df064a3f98a8685e87f367200f-u1-004of019.chunk
[2021-11-20 0:18:30]  [DEBUG]  ws_images_uploadAsync all chunks are not uploaded yet, maybe on next chunk, exit for now

Last edited by endrju83 (2021-11-20 09:36:14)

Offline

 

#11 2021-11-20 10:34:37

cryopad
Piwigo Team
Grenoble, France
2015-11-01
190

Re: Upload using uploadAsync / piwigo mobile doesn't work most of the time

The max upload file size set on the server is one thing. The capability of the server to manipulate large files is another one. After having merging the chunks, the server prepares the thumbnails, updates the database, etc. and this can request CPU and RAM which are not available.

To answer your second question, the Piwigo iOS app splits the photo or video files into chunks and requests iOS to upload them in the background. iOS uploads up to 4 chunks in parallel. If the the user quits the Piwigo app, iOS finishes the uploads of the currently being uploaded chunks in the background. Then, if you leave the device plugged to a charger, iOS will pursue or not the uploads in the background. This is decided by iOS, not by the Piwigo app.

Chunks are not uploaded sequentially. Some chunks can be uploaded before others. This depends on the network operation, server reactivity, device use, size of the last chunk, etc. The Piwigo server collects the chunks and stores them for a limited period of time to avoid an infinite accumulation of chunks. In parallel, the iOS device may have no network access for some days (the user is hiking in high mountains for example) and some admins may delete chunks during a maintenance operation for some reasons (rare but possible).

So the iOS app does not know for sure which chunks are still on the server when it restarts uploading. That is why it retries from scratch after a failure. One may suggest that the iOS app retrieves the list of chunks which are still on the server after having re-uploaded a first chunk and cancel unnecessary upload tasks in the iOS upload queue. But how can we be certain that these chunks will still be available at the end of the remaining uploads? We enter an infinite loop.


If you enjoy Piwigo for iOS…
➤ We would appreciate an AppStore review.
➤ Comments, suggestions ? Tell us on GitHub…
➤ Not yet available in your language ? Help us translate it…

Offline

 

#12 2021-11-20 13:00:28

endrju83
Member
2020-10-11
11

Re: Upload using uploadAsync / piwigo mobile doesn't work most of the time

First, let me thanks to such thorough explanation, appreciated!

cryopad wrote:

The Piwigo server collects the chunks and stores them for a limited period of time to avoid an infinite accumulation of chunks.

Do you know how long Piwigo server keeps the chunks before deleting them? Is this time configurable?

cryopad wrote:

So the iOS app does not know for sure which chunks are still on the server when it restarts uploading. That is why it retries from scratch after a failure. One may suggest that the iOS app retrieves the list of chunks which are still on the server after having re-uploaded a first chunk and cancel unnecessary upload tasks in the iOS upload queue. But how can we be certain that these chunks will still be available at the end of the remaining uploads? We enter an infinite loop.

I get it, maybe it is not possible right now. But this could have a solution in my opinion. If Piwigo app have access to the list of chunks already on the server and uploads only those which are missing, then even if I had to reupload a few times I could be reassured that it would eventually upload all the chunks (now I had to be really lucky).
With each reupload you would know the list of chunks already on the server, so it upload only those which are missing and at the end before merging, you can verify the count of chunks on the server and do the MD5 checksum (it already does it I believe - can see it in the log). So with "just" a few additional interface commands it could be done?

Offline

 

Board footer

Powered by FluxBB

github twitter newsletter Donate Piwigo.org © 2002-2024 · Contact