Piwigo.org

You are not logged in. (Register / Login)

Announcement

#1 2013-11-06 03:31:48

jimaek
Member
Registered: 2013-05-17
Posts: 41

[resolved] [Tool] Convert all Virtual albums to Physical ones

Tags: [albums, virtual]

What the tool does is to convert all Virtual Albums to Physical ones.
All albums will be re-created as directories inside galleries/ and all the associated images will be put there as well using the original filename when the image was uploaded.
Note: The images are not moved or deleted, they are simply copied. In case that everything goes wrong you can just restore your DB backup and everything should be ok.

Technically if everything goes ok you can completely delete the upload directory.

1. Do a DB backup
2. Put the script inside your piwigo root directory
3. Using your console run the script and wait
4. Do a sync and view your website

Warning: Not for newbies!

This script was made for a personal website of mine, it worked and I have no use for it anymore. Feel free to modify it and distribute but please leave a note saying that the original was developed by jsdelivr.com.


http://www.jsdelivr.com/piwigo/cli_data_move.php.txt


DO FULL BACKUP BEFORE RUNNING IT
If you lose all data, if your DB crashes, if your server explodes it is YOUR fault.
Review the code before running it, I am NOT responsible for any damages at all and I will not be supporting the script.


I hope someone finds it useful.
Thank you!

Offline

 

#2 2013-11-06 14:59:51

flop25
Piwigo Team
Registered: 2006-07-06
Posts: 6566
Website

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

Hello
That's great ! It could be used as an export tool too! Thanks!


To get a better help : Politeness like Hello-A link-Your past actions precisely described
Check my extensions : more than 30 available
who I am and what I do : http://fr.gravatar.com/flop25
My gallery : an illustration of how to integrate Piwigo in your website

Offline

 

#3 2014-10-17 21:56:14

thanorseman
Member
Registered: 2014-10-10
Posts: 5

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

I apologize for the newb question. Are there any version restrictions with this script? I tried running it with 2.6.3 and Fatal error: Call to undefined method mysqli_result::fetch_all() in /home/usernameRemoved/public_html/images/cli_data_move.php on line 43

I'm ready to make the switch from Gallery3 to Piwigo, but i want to make sure I can export everything in a nice file structure, exactly what this script does. So I'm hopeful I'm just missing a step or not doing it correctly. Thank you!


Operating system: Linux
PHP: 5.2.17 (Show info) [2014-10-17 12:52:24]
MySQL: 5.5.37-35.0 [2014-10-17 12:52:24]
Graphics Library: External ImageMagick 6.5.4-7

Offline

 

#4 2014-10-17 21:57:54

mistic100
Piwigo Team
Location: Lyon (FR)
Registered: 2008-09-27
Posts: 3261
Website

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

http://php.net/manual/en/mysqli-result.fetch-all.php

PHP 5.3 required

you will need to upgrade your server before using it


» All my plugins  » My website
For an efficient support give the URL of your website

Offline

 

#5 2014-10-18 04:04:37

thanorseman
Member
Registered: 2014-10-10
Posts: 5

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

Thanks for the incredibly fast response. I've changed my hosting service to now use PHP 5.3 for my Piwigo installation, but I'm still getting the same error. Also, I tried updating Piwigo to 2.7.1 and that doesnt have an effect either...still the same error message

Thanks to anyone with advice, I appreciate it!


Operating system: Linux
PHP: 5.3.28 (Show info) [2014-10-18 01:45:36]
MySQL: 5.5.37-35.0 [2014-10-17 18:45:36]
Graphics Library: ImageMagick 6.5.4-7

Offline

 

#6 2014-11-04 18:20:22

mkauzlar
Member
Registered: 2014-11-04
Posts: 1

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

Check if you have mysqlnd support:

# php -m | grep mysql
mysql
mysqli
mysqlnd
pdo_mysql

If you don't install it.

On ubuntu or debian install php5-mysqlnd.

Offline

 

#7 2014-11-04 18:54:23

jnashpiwigo
Member
Registered: 2014-10-21
Posts: 129

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

This would be nice as an export tool to just copy the files appropriately, but not modify piwigo or the DB in any way.  Could be implemented as an option in this plugin?

Offline

 

#8 2014-11-04 21:17:51

thanorseman
Member
Registered: 2014-10-10
Posts: 5

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

I don't want to get too far off topic but I think my issues is that my host (Arvixe) has a default version of PHP at 5.2. I can change it to 5.3 or 5.4 etc for my sites, but no matter what when I execute the script via the console, its running with 5.2 which doesn't have mysqlnd support, which I've confirmed by executing php -m | grep MySQL, no mysqlnd. I've reached out to Arvixe support to see how I execute the 5.3 or above version of PHP from the console, but wasn't very successful. Perhaps its not possible to do as its only available as an extension. I'm not sure.

I am running my Gallery3 install out of my house, but the upload speeds are too slow and I need to save on the electricity bill so I'm switching to shared hosting. Running a r710 and sbod bay of 16 15kRPM disks is just not cost effective. Clearly some tradeoffs from running my own setup vs VPS or shared hosting.

I second the thought that it would be great if something like this tool was made into an extension. Unfortunately I do not possess the skills to make that sort of thing happen. So as to not go too far off topic on this thread than I already have, I'll put a request into the request section to see if someone is interested and has the skills.

Last edited by thanorseman (2014-11-04 21:18:19)

Offline

 

#9 2014-11-04 21:25:19

jnashpiwigo
Member
Registered: 2014-10-21
Posts: 129

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

Just as a sidenote, my php does not have mysqlnd either, yet the module works fine. 

mysqlnd conflicts with mysql for php, so I'm a tad confused.

$ php -m | grep mysql     
mysql
mysqli
pdo_mysql


In aptitude (ubuntu) upon attemped install of mysqlnd

* php5-mysql conflicts with php5-mysqlnd

Last edited by jnashpiwigo (2014-11-04 21:27:19)

Offline

 

#10 2015-01-04 15:17:59

jc
Member
Registered: 2015-01-04
Posts: 20

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

Hello piwigo and ex-gallery users :-)

Thanks for this code, it has a huge potential. It is usable for:

- V2P conversion (actually doing this)
- full V2P conversion (needs an option for delete source virtual data)
- export galleries to filesystem (needs an option to NOT alter DB but choose of destination directory)
- periodically V2P so you can upload files via web and move them automatically to regular physical structure

Extension for this functions would be nice. But i don't have enough skills.

At least i have solution for fetch_all error. For this function you must have not only PHP5.3+ but mysqlnd (native) PHP driver too. With Debian 7 you have to have php5-mysqlnd package, which is in collision with regular php5-mysql package.

But there is simplier solution (hint) and i have a simple patch:

Code:

43c43,45
<               $results = $result->fetch_all(MYSQLI_ASSOC);
---
>                 //$results = $result->fetch_all(MYSQLI_ASSOC);
>                 //for PHP with mysql driver (not mysqlnd) - without fetch_all function
>                 for ($results = array(); $tmp = $result->fetch_array(MYSQLI_ASSOC);) $results[] = $tmp;

Fullsize code is here: http://pastebin.com/jPwFNf8V

Thanks again for code from jimaek.

Offline

 

#11 2015-04-11 20:26:58

terminus
Member
Registered: 2014-11-17
Posts: 18

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

A problem that I found with the script is that it doesn't catch the case where there are several photos with the same filename.  This can happen in large albums, or when using different cameras that reuse the same common filenames (eg. DSC_0001.jpg). Related to this, an odd quirk is that where the names are unique except for case (eg. JPG and jpg), the full-size images will work, but the thumbnails won't - you'll see just one thumbnail for both.

These problems don't effect virtual albums, where unique names are enforced, but do effect the physical albums as converted by this script. The script requires a simple patch to check for and eliminate duplicates. For now I'm just flagging the problem - if I have time later, I could also contribute the patch.

Offline

 

#12 2015-04-27 12:19:37

piwok
Guest

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

Its April 2015 and i'd just like to say a huge thankyou to jimaek for writing this script. You rock! It still worked for me perfectly as of latest piwigo version. I'm sure you've helped out a tonne of people over the years with this.

Thanks again!

 

#13 2016-06-14 01:03:35

terminus
Member
Registered: 2014-11-17
Posts: 18

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

I found another bug, and I've fixed it.  The bug was that if you have more than two levels of folder, it will return them in the wrong order.  So for example if your image is in:

Fruits/Apples/fuji.jpg

Then it would be put into:

Apples/Fruits/fuji.jpg

Which is of course completely wrong.  This version fixes it:

http://pastebin.com/FRc58JPy

The diff is below:

Code:

--- 2.txt  2016-06-13 16:02:45.000000000 -0700
+++ 1.txt  2016-06-13 16:02:29.000000000 -0700
@@ -52,9 +52,14 @@
       $full_q_path = explode('///', $image['full_path']);
       $real_path = '';
       $real_path_a = array();
+      $sorted_q_path = array();
       $storage_category_id = 0;
       foreach ($full_q_path as $el) {
-        list($id, $fs_dir, $virt_dir) = explode(':::', $el);
+        $sorted_q_path[] = explode(':::', $el);
+      };
+      asort($sorted_q_path, SORT_ASC);
+      foreach ($sorted_q_path as $el) {
+        list($id, $fs_dir, $virt_dir) = $el;
         if (empty($fs_dir)) {
           $fs_dir = preg_replace("/[^\\w-_\ ]+/",'',$virt_dir);
           $fs_dir = trim($fs_dir);

Hope that helps.

Offline

 

#14 2016-12-26 01:28:20

terminus
Member
Registered: 2014-11-17
Posts: 18

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

terminus wrote:

A problem that I found with the script is that it doesn't catch the case where there are several photos with the same filename.  This can happen in large albums, or when using different cameras that reuse the same common filenames (eg. DSC_0001.jpg). Related to this, an odd quirk is that where the names are unique except for case (eg. JPG and jpg), the full-size images will work, but the thumbnails won't - you'll see just one thumbnail for both.

These problems don't effect virtual albums, where unique names are enforced, but do effect the physical albums as converted by this script. The script requires a simple patch to check for and eliminate duplicates. For now I'm just flagging the problem - if I have time later, I could also contribute the patch.

As promised, here is the patched version: http://pastebin.com/LMBmBFe5

And here the diff against my earlier bugfix version:

Code:

--- 2.txt  2016-12-25 16:22:29.000000000 -0800
+++ 1.txt  2016-12-25 16:23:46.000000000 -0800
@@ -77,8 +77,24 @@
         mkdir($real_path, 0775, true);
         echo '+';
       }
-      // copy
+      // check for duplicates
       $new_file_path = $real_path.'/'.$image['file'];
+      $dupe_file_path = $new_file_path;
+      $dupes = 1;
+      while ($dupes > 0) {
+        $sql="SELECT path FROM images WHERE path='".$dbc->real_escape_string($dupe_file_path)."'";
+        if ($result = $dbc->query($sql)) {
+          if ($result->num_rows > 0) {
+            $dupe_file_path = pathinfo($new_file_path,PATHINFO_DIRNAME) . "/" . pathinfo($new_file_path,PATHINFO_FILENAME) . $dupes++ . "." . pathinfo($new_file_path,PATHINFO_EXTENSION);
+          }
+          else {
+            $dupes = 0;
+            $new_file_path = $dupe_file_path;
+            break;
+          }
+        }
+      }
+      // copy
       copy($image['path'], $new_file_path);
       //rename($image['path'], $new_file_path);
       echo ' [v]';
@@ -87,6 +103,21 @@
       if ($storage_category_id > 0) {
         $dbc->query("UPDATE `{$prefixeTable}images` SET storage_category_id = $storage_category_id WHERE id = {$image['id']}");
       }
+      // remove old thumbnails
+      $image_dir = pathinfo($image['path'], PATHINFO_DIRNAME);
+      $thumb_dir = "./_data/i" . ltrim($image_dir,".");
+      foreach (glob($thumb_dir . "/*") as $filename) {
+        unlink($filename);
+      }
+      foreach (glob($thumb_dir . "/pwg_representative/*") as $filename) {
+        unlink($filename);
+      }
+      // try deleting thumbnail directory too
+      @rmdir($thumb_dir);
+      @rmdir($image_dir . "/pwg_representative");
+      // try deleting two parent directories (month, year)
+      @rmdir(dirname($thumb_dir));
+      @rmdir(dirname($thumb_dir,1));
     }
   }
   else {

Offline

 

Board footer

Powered by FluxBB

github twitter facebook google+ newsletter Donate Piwigo.org © 2002-2017 · Contact