Announcement

  •  » Extensions
  •  » [resolved] [Tool] Convert all Virtual albums to Physical ones

#1 2013-11-06 03:31:48

jimaek
Member
2013-05-17
41

[resolved] [Tool] Convert all Virtual albums to Physical ones

What the tool does is to convert all Virtual Albums to Physical ones.
All albums will be re-created as directories inside galleries/ and all the associated images will be put there as well using the original filename when the image was uploaded.
Note: The images are not moved or deleted, they are simply copied. In case that everything goes wrong you can just restore your DB backup and everything should be ok.

Technically if everything goes ok you can completely delete the upload directory.

1. Do a DB backup
2. Put the script inside your piwigo root directory
3. Using your console run the script and wait
4. Do a sync and view your website

Warning: Not for newbies!

This script was made for a personal website of mine, it worked and I have no use for it anymore. Feel free to modify it and distribute but please leave a note saying that the original was developed by jsdelivr.com.


http://www.jsdelivr.com/piwigo/cli_data_move.php.txt


DO FULL BACKUP BEFORE RUNNING IT
If you lose all data, if your DB crashes, if your server explodes it is YOUR fault.
Review the code before running it, I am NOT responsible for any damages at all and I will not be supporting the script.


I hope someone finds it useful.
Thank you!

Offline

 

#2 2013-11-06 14:59:51

flop25
Piwigo Team
2006-07-06
7037

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

Hello
That's great ! It could be used as an export tool too! Thanks!


To get a better help : Politeness like Hello-A link-Your past actions precisely described
Check my extensions : more than 30 available
who I am and what I do : http://fr.gravatar.com/flop25
My gallery : an illustration of how to integrate Piwigo in your website

Offline

 

#3 2014-10-17 21:56:14

thanorseman
Member
2014-10-10
5

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

I apologize for the newb question. Are there any version restrictions with this script? I tried running it with 2.6.3 and Fatal error: Call to undefined method mysqli_result::fetch_all() in /home/usernameRemoved/public_html/images/cli_data_move.php on line 43

I'm ready to make the switch from Gallery3 to Piwigo, but i want to make sure I can export everything in a nice file structure, exactly what this script does. So I'm hopeful I'm just missing a step or not doing it correctly. Thank you!


Operating system: Linux
PHP: 5.2.17 (Show info) [2014-10-17 12:52:24]
MySQL: 5.5.37-35.0 [2014-10-17 12:52:24]
Graphics Library: External ImageMagick 6.5.4-7

Offline

 

#4 2014-10-17 21:57:54

mistic100
Former Piwigo Team
Lyon (FR)
2008-09-27
3277

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

http://php.net/manual/en/mysqli-result.fetch-all.php

PHP 5.3 required

you will need to upgrade your server before using it

Offline

 

#5 2014-10-18 04:04:37

thanorseman
Member
2014-10-10
5

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

Thanks for the incredibly fast response. I've changed my hosting service to now use PHP 5.3 for my Piwigo installation, but I'm still getting the same error. Also, I tried updating Piwigo to 2.7.1 and that doesnt have an effect either...still the same error message

Thanks to anyone with advice, I appreciate it!


Operating system: Linux
PHP: 5.3.28 (Show info) [2014-10-18 01:45:36]
MySQL: 5.5.37-35.0 [2014-10-17 18:45:36]
Graphics Library: ImageMagick 6.5.4-7

Offline

 

#6 2014-11-04 18:20:22

mkauzlar
Member
2014-11-04
1

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

Check if you have mysqlnd support:

# php -m | grep mysql
mysql
mysqli
mysqlnd
pdo_mysql

If you don't install it.

On ubuntu or debian install php5-mysqlnd.

Offline

 

#7 2014-11-04 18:54:23

jnashpiwigo
Piwigo Team
2014-10-21
254

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

This would be nice as an export tool to just copy the files appropriately, but not modify piwigo or the DB in any way.  Could be implemented as an option in this plugin?

Offline

 

#8 2014-11-04 21:17:51

thanorseman
Member
2014-10-10
5

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

I don't want to get too far off topic but I think my issues is that my host (Arvixe) has a default version of PHP at 5.2. I can change it to 5.3 or 5.4 etc for my sites, but no matter what when I execute the script via the console, its running with 5.2 which doesn't have mysqlnd support, which I've confirmed by executing php -m | grep MySQL, no mysqlnd. I've reached out to Arvixe support to see how I execute the 5.3 or above version of PHP from the console, but wasn't very successful. Perhaps its not possible to do as its only available as an extension. I'm not sure.

I am running my Gallery3 install out of my house, but the upload speeds are too slow and I need to save on the electricity bill so I'm switching to shared hosting. Running a r710 and sbod bay of 16 15kRPM disks is just not cost effective. Clearly some tradeoffs from running my own setup vs VPS or shared hosting.

I second the thought that it would be great if something like this tool was made into an extension. Unfortunately I do not possess the skills to make that sort of thing happen. So as to not go too far off topic on this thread than I already have, I'll put a request into the request section to see if someone is interested and has the skills.

Last edited by thanorseman (2014-11-04 21:18:19)

Offline

 

#9 2014-11-04 21:25:19

jnashpiwigo
Piwigo Team
2014-10-21
254

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

Just as a sidenote, my php does not have mysqlnd either, yet the module works fine. 

mysqlnd conflicts with mysql for php, so I'm a tad confused.

$ php -m | grep mysql     
mysql
mysqli
pdo_mysql


In aptitude (ubuntu) upon attemped install of mysqlnd

* php5-mysql conflicts with php5-mysqlnd

Last edited by jnashpiwigo (2014-11-04 21:27:19)

Offline

 

#10 2015-01-04 15:17:59

jc
Member
2015-01-04
25

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

Hello piwigo and ex-gallery users :-)

Thanks for this code, it has a huge potential. It is usable for:

- V2P conversion (actually doing this)
- full V2P conversion (needs an option for delete source virtual data)
- export galleries to filesystem (needs an option to NOT alter DB but choose of destination directory)
- periodically V2P so you can upload files via web and move them automatically to regular physical structure

Extension for this functions would be nice. But i don't have enough skills.

At least i have solution for fetch_all error. For this function you must have not only PHP5.3+ but mysqlnd (native) PHP driver too. With Debian 7 you have to have php5-mysqlnd package, which is in collision with regular php5-mysql package.

But there is simplier solution (hint) and i have a simple patch:

Code:

43c43,45
<               $results = $result->fetch_all(MYSQLI_ASSOC);
---
>                 //$results = $result->fetch_all(MYSQLI_ASSOC);
>                 //for PHP with mysql driver (not mysqlnd) - without fetch_all function
>                 for ($results = array(); $tmp = $result->fetch_array(MYSQLI_ASSOC);) $results[] = $tmp;

Fullsize code is here: http://pastebin.com/jPwFNf8V

Thanks again for code from jimaek.

Offline

 

#11 2015-04-11 20:26:58

terminus
Member
2014-11-17
19

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

A problem that I found with the script is that it doesn't catch the case where there are several photos with the same filename.  This can happen in large albums, or when using different cameras that reuse the same common filenames (eg. DSC_0001.jpg). Related to this, an odd quirk is that where the names are unique except for case (eg. JPG and jpg), the full-size images will work, but the thumbnails won't - you'll see just one thumbnail for both.

These problems don't effect virtual albums, where unique names are enforced, but do effect the physical albums as converted by this script. The script requires a simple patch to check for and eliminate duplicates. For now I'm just flagging the problem - if I have time later, I could also contribute the patch.

Offline

 

#12 2015-04-27 12:19:37

piwok
Guest

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

Its April 2015 and i'd just like to say a huge thankyou to jimaek for writing this script. You rock! It still worked for me perfectly as of latest piwigo version. I'm sure you've helped out a tonne of people over the years with this.

Thanks again!

 

#13 2016-06-14 01:03:35

terminus
Member
2014-11-17
19

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

I found another bug, and I've fixed it.  The bug was that if you have more than two levels of folder, it will return them in the wrong order.  So for example if your image is in:

Fruits/Apples/fuji.jpg

Then it would be put into:

Apples/Fruits/fuji.jpg

Which is of course completely wrong.  This version fixes it:

http://pastebin.com/FRc58JPy

The diff is below:

Code:

--- 2.txt  2016-06-13 16:02:45.000000000 -0700
+++ 1.txt  2016-06-13 16:02:29.000000000 -0700
@@ -52,9 +52,14 @@
       $full_q_path = explode('///', $image['full_path']);
       $real_path = '';
       $real_path_a = array();
+      $sorted_q_path = array();
       $storage_category_id = 0;
       foreach ($full_q_path as $el) {
-        list($id, $fs_dir, $virt_dir) = explode(':::', $el);
+        $sorted_q_path[] = explode(':::', $el);
+      };
+      asort($sorted_q_path, SORT_ASC);
+      foreach ($sorted_q_path as $el) {
+        list($id, $fs_dir, $virt_dir) = $el;
         if (empty($fs_dir)) {
           $fs_dir = preg_replace("/[^\\w-_\ ]+/",'',$virt_dir);
           $fs_dir = trim($fs_dir);

Hope that helps.

Offline

 

#14 2016-12-26 01:28:20

terminus
Member
2014-11-17
19

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

terminus wrote:

A problem that I found with the script is that it doesn't catch the case where there are several photos with the same filename.  This can happen in large albums, or when using different cameras that reuse the same common filenames (eg. DSC_0001.jpg). Related to this, an odd quirk is that where the names are unique except for case (eg. JPG and jpg), the full-size images will work, but the thumbnails won't - you'll see just one thumbnail for both.

These problems don't effect virtual albums, where unique names are enforced, but do effect the physical albums as converted by this script. The script requires a simple patch to check for and eliminate duplicates. For now I'm just flagging the problem - if I have time later, I could also contribute the patch.

As promised, here is the patched version: http://pastebin.com/LMBmBFe5

And here the diff against my earlier bugfix version:

Code:

--- 2.txt  2016-12-25 16:22:29.000000000 -0800
+++ 1.txt  2016-12-25 16:23:46.000000000 -0800
@@ -77,8 +77,24 @@
         mkdir($real_path, 0775, true);
         echo '+';
       }
-      // copy
+      // check for duplicates
       $new_file_path = $real_path.'/'.$image['file'];
+      $dupe_file_path = $new_file_path;
+      $dupes = 1;
+      while ($dupes > 0) {
+        $sql="SELECT path FROM images WHERE path='".$dbc->real_escape_string($dupe_file_path)."'";
+        if ($result = $dbc->query($sql)) {
+          if ($result->num_rows > 0) {
+            $dupe_file_path = pathinfo($new_file_path,PATHINFO_DIRNAME) . "/" . pathinfo($new_file_path,PATHINFO_FILENAME) . $dupes++ . "." . pathinfo($new_file_path,PATHINFO_EXTENSION);
+          }
+          else {
+            $dupes = 0;
+            $new_file_path = $dupe_file_path;
+            break;
+          }
+        }
+      }
+      // copy
       copy($image['path'], $new_file_path);
       //rename($image['path'], $new_file_path);
       echo ' [v]';
@@ -87,6 +103,21 @@
       if ($storage_category_id > 0) {
         $dbc->query("UPDATE `{$prefixeTable}images` SET storage_category_id = $storage_category_id WHERE id = {$image['id']}");
       }
+      // remove old thumbnails
+      $image_dir = pathinfo($image['path'], PATHINFO_DIRNAME);
+      $thumb_dir = "./_data/i" . ltrim($image_dir,".");
+      foreach (glob($thumb_dir . "/*") as $filename) {
+        unlink($filename);
+      }
+      foreach (glob($thumb_dir . "/pwg_representative/*") as $filename) {
+        unlink($filename);
+      }
+      // try deleting thumbnail directory too
+      @rmdir($thumb_dir);
+      @rmdir($image_dir . "/pwg_representative");
+      // try deleting two parent directories (month, year)
+      @rmdir(dirname($thumb_dir));
+      @rmdir(dirname($thumb_dir,1));
     }
   }
   else {

Offline

 

#15 2018-06-02 23:26:52

georg441
Member
2018-06-01
2

Re: [resolved] [Tool] Convert all Virtual albums to Physical ones

This script doesn't work on my system. I think it gets caught in a loop right from the first iteration.

I'm getting no errors but I see the script running the same commands for the first picture in my database in MySQL without an end.

I'm not a PHP expert so I leaving this here in case anyone's interested to look into the script and find if there's a problem.

Offline

 
  •  » Extensions
  •  » [resolved] [Tool] Convert all Virtual albums to Physical ones

Board footer

Powered by FluxBB

github twitter newsletter Donate Piwigo.org © 2002-2024 · Contact