What the tool does is to convert all Virtual Albums to Physical ones.
All albums will be re-created as directories inside galleries/ and all the associated images will be put there as well using the original filename when the image was uploaded.
Note: The images are not moved or deleted, they are simply copied. In case that everything goes wrong you can just restore your DB backup and everything should be ok.
Technically if everything goes ok you can completely delete the upload directory.
1. Do a DB backup
2. Put the script inside your piwigo root directory
3. Using your console run the script and wait
4. Do a sync and view your website
Warning: Not for newbies!
This script was made for a personal website of mine, it worked and I have no use for it anymore. Feel free to modify it and distribute but please leave a note saying that the original was developed by jsdelivr.com.
http://www.jsdelivr.com/piwigo/cli_data_move.php.txt
DO FULL BACKUP BEFORE RUNNING IT
If you lose all data, if your DB crashes, if your server explodes it is YOUR fault.
Review the code before running it, I am NOT responsible for any damages at all and I will not be supporting the script.
I hope someone finds it useful.
Thank you!
Offline
Hello
That's great ! It could be used as an export tool too! Thanks!
Offline
I apologize for the newb question. Are there any version restrictions with this script? I tried running it with 2.6.3 and Fatal error: Call to undefined method mysqli_result::fetch_all() in /home/usernameRemoved/public_html/images/cli_data_move.php on line 43
I'm ready to make the switch from Gallery3 to Piwigo, but i want to make sure I can export everything in a nice file structure, exactly what this script does. So I'm hopeful I'm just missing a step or not doing it correctly. Thank you!
Operating system: Linux
PHP: 5.2.17 (Show info) [2014-10-17 12:52:24]
MySQL: 5.5.37-35.0 [2014-10-17 12:52:24]
Graphics Library: External ImageMagick 6.5.4-7
Offline
http://php.net/manual/en/mysqli-result.fetch-all.php
PHP 5.3 required
you will need to upgrade your server before using it
Offline
Thanks for the incredibly fast response. I've changed my hosting service to now use PHP 5.3 for my Piwigo installation, but I'm still getting the same error. Also, I tried updating Piwigo to 2.7.1 and that doesnt have an effect either...still the same error message
Thanks to anyone with advice, I appreciate it!
Operating system: Linux
PHP: 5.3.28 (Show info) [2014-10-18 01:45:36]
MySQL: 5.5.37-35.0 [2014-10-17 18:45:36]
Graphics Library: ImageMagick 6.5.4-7
Offline
Check if you have mysqlnd support:
# php -m | grep mysql
mysql
mysqli
mysqlnd
pdo_mysql
If you don't install it.
On ubuntu or debian install php5-mysqlnd.
Offline
This would be nice as an export tool to just copy the files appropriately, but not modify piwigo or the DB in any way. Could be implemented as an option in this plugin?
Offline
I don't want to get too far off topic but I think my issues is that my host (Arvixe) has a default version of PHP at 5.2. I can change it to 5.3 or 5.4 etc for my sites, but no matter what when I execute the script via the console, its running with 5.2 which doesn't have mysqlnd support, which I've confirmed by executing php -m | grep MySQL, no mysqlnd. I've reached out to Arvixe support to see how I execute the 5.3 or above version of PHP from the console, but wasn't very successful. Perhaps its not possible to do as its only available as an extension. I'm not sure.
I am running my Gallery3 install out of my house, but the upload speeds are too slow and I need to save on the electricity bill so I'm switching to shared hosting. Running a r710 and sbod bay of 16 15kRPM disks is just not cost effective. Clearly some tradeoffs from running my own setup vs VPS or shared hosting.
I second the thought that it would be great if something like this tool was made into an extension. Unfortunately I do not possess the skills to make that sort of thing happen. So as to not go too far off topic on this thread than I already have, I'll put a request into the request section to see if someone is interested and has the skills.
Last edited by thanorseman (2014-11-04 21:18:19)
Offline
Just as a sidenote, my php does not have mysqlnd either, yet the module works fine.
mysqlnd conflicts with mysql for php, so I'm a tad confused.
$ php -m | grep mysql
mysql
mysqli
pdo_mysql
In aptitude (ubuntu) upon attemped install of mysqlnd
* php5-mysql conflicts with php5-mysqlnd
Last edited by jnashpiwigo (2014-11-04 21:27:19)
Offline
Hello piwigo and ex-gallery users :-)
Thanks for this code, it has a huge potential. It is usable for:
- V2P conversion (actually doing this)
- full V2P conversion (needs an option for delete source virtual data)
- export galleries to filesystem (needs an option to NOT alter DB but choose of destination directory)
- periodically V2P so you can upload files via web and move them automatically to regular physical structure
Extension for this functions would be nice. But i don't have enough skills.
At least i have solution for fetch_all error. For this function you must have not only PHP5.3+ but mysqlnd (native) PHP driver too. With Debian 7 you have to have php5-mysqlnd package, which is in collision with regular php5-mysql package.
But there is simplier solution (hint) and i have a simple patch:
43c43,45 < $results = $result->fetch_all(MYSQLI_ASSOC); --- > //$results = $result->fetch_all(MYSQLI_ASSOC); > //for PHP with mysql driver (not mysqlnd) - without fetch_all function > for ($results = array(); $tmp = $result->fetch_array(MYSQLI_ASSOC);) $results[] = $tmp;
Fullsize code is here: http://pastebin.com/jPwFNf8V
Thanks again for code from jimaek.
Offline
A problem that I found with the script is that it doesn't catch the case where there are several photos with the same filename. This can happen in large albums, or when using different cameras that reuse the same common filenames (eg. DSC_0001.jpg). Related to this, an odd quirk is that where the names are unique except for case (eg. JPG and jpg), the full-size images will work, but the thumbnails won't - you'll see just one thumbnail for both.
These problems don't effect virtual albums, where unique names are enforced, but do effect the physical albums as converted by this script. The script requires a simple patch to check for and eliminate duplicates. For now I'm just flagging the problem - if I have time later, I could also contribute the patch.
Offline
Its April 2015 and i'd just like to say a huge thankyou to jimaek for writing this script. You rock! It still worked for me perfectly as of latest piwigo version. I'm sure you've helped out a tonne of people over the years with this.
Thanks again!
I found another bug, and I've fixed it. The bug was that if you have more than two levels of folder, it will return them in the wrong order. So for example if your image is in:
Fruits/Apples/fuji.jpg
Then it would be put into:
Apples/Fruits/fuji.jpg
Which is of course completely wrong. This version fixes it:
http://pastebin.com/FRc58JPy
The diff is below:
--- 2.txt 2016-06-13 16:02:45.000000000 -0700 +++ 1.txt 2016-06-13 16:02:29.000000000 -0700 @@ -52,9 +52,14 @@ $full_q_path = explode('///', $image['full_path']); $real_path = ''; $real_path_a = array(); + $sorted_q_path = array(); $storage_category_id = 0; foreach ($full_q_path as $el) { - list($id, $fs_dir, $virt_dir) = explode(':::', $el); + $sorted_q_path[] = explode(':::', $el); + }; + asort($sorted_q_path, SORT_ASC); + foreach ($sorted_q_path as $el) { + list($id, $fs_dir, $virt_dir) = $el; if (empty($fs_dir)) { $fs_dir = preg_replace("/[^\\w-_\ ]+/",'',$virt_dir); $fs_dir = trim($fs_dir);
Hope that helps.
Offline
terminus wrote:
A problem that I found with the script is that it doesn't catch the case where there are several photos with the same filename. This can happen in large albums, or when using different cameras that reuse the same common filenames (eg. DSC_0001.jpg). Related to this, an odd quirk is that where the names are unique except for case (eg. JPG and jpg), the full-size images will work, but the thumbnails won't - you'll see just one thumbnail for both.
These problems don't effect virtual albums, where unique names are enforced, but do effect the physical albums as converted by this script. The script requires a simple patch to check for and eliminate duplicates. For now I'm just flagging the problem - if I have time later, I could also contribute the patch.
As promised, here is the patched version: http://pastebin.com/LMBmBFe5
And here the diff against my earlier bugfix version:
--- 2.txt 2016-12-25 16:22:29.000000000 -0800 +++ 1.txt 2016-12-25 16:23:46.000000000 -0800 @@ -77,8 +77,24 @@ mkdir($real_path, 0775, true); echo '+'; } - // copy + // check for duplicates $new_file_path = $real_path.'/'.$image['file']; + $dupe_file_path = $new_file_path; + $dupes = 1; + while ($dupes > 0) { + $sql="SELECT path FROM images WHERE path='".$dbc->real_escape_string($dupe_file_path)."'"; + if ($result = $dbc->query($sql)) { + if ($result->num_rows > 0) { + $dupe_file_path = pathinfo($new_file_path,PATHINFO_DIRNAME) . "/" . pathinfo($new_file_path,PATHINFO_FILENAME) . $dupes++ . "." . pathinfo($new_file_path,PATHINFO_EXTENSION); + } + else { + $dupes = 0; + $new_file_path = $dupe_file_path; + break; + } + } + } + // copy copy($image['path'], $new_file_path); //rename($image['path'], $new_file_path); echo ' [v]'; @@ -87,6 +103,21 @@ if ($storage_category_id > 0) { $dbc->query("UPDATE `{$prefixeTable}images` SET storage_category_id = $storage_category_id WHERE id = {$image['id']}"); } + // remove old thumbnails + $image_dir = pathinfo($image['path'], PATHINFO_DIRNAME); + $thumb_dir = "./_data/i" . ltrim($image_dir,"."); + foreach (glob($thumb_dir . "/*") as $filename) { + unlink($filename); + } + foreach (glob($thumb_dir . "/pwg_representative/*") as $filename) { + unlink($filename); + } + // try deleting thumbnail directory too + @rmdir($thumb_dir); + @rmdir($image_dir . "/pwg_representative"); + // try deleting two parent directories (month, year) + @rmdir(dirname($thumb_dir)); + @rmdir(dirname($thumb_dir,1)); } } else {
Offline
This script doesn't work on my system. I think it gets caught in a loop right from the first iteration.
I'm getting no errors but I see the script running the same commands for the first picture in my database in MySQL without an end.
I'm not a PHP expert so I leaving this here in case anyone's interested to look into the script and find if there's a problem.
Offline