Announcement

#1 2013-10-20 23:04:28

hpsam
Member
2005-06-28
93

[resolved] pwg_high, TN- from 2.x to 2.5 how to migrate ?

Hello,

I moved to 2.0.10 to the last version 2.5.2 and I discover after update that all my "thumbail" and "pwg_high" folder are ignored.

I search and I read a lot on forum about that but I don't found solutions to migrate the old folder tree to the new structure. My gallery have more than 17000 photos in more than 200 sub folders.

Is there a solution to move all :
- galleries/myCat/thumbnail/TN-*.jpg TO _data/i/galleries/myCat/*-th.jpg
- galleries/myCat/*.jpg TO _data/i/galleries/myCat/*-me.jpg
- galleries/myCat/pwg_high/*.jpg TO galleries/myCat/*.jpg

Or to move only all "galleries/myCat/pwg_high/*.jpg" to "galleries/myCat/*.jpg" and generate all thumbails and mediums photos without waiting that visitors ask them.

I have my own debian server so I can launch shell script / command if needed.

thanks

I think that the piwigo update script must display a warning for this impact before performing upgrade.

Last edited by hpsam (2013-10-20 23:04:59)

Offline

 

#2 2013-10-21 07:59:40

flop25
Piwigo Team
2006-07-06
7037

Re: [resolved] pwg_high, TN- from 2.x to 2.5 how to migrate ?

Hi

You just didn't notice the new action in the Batch Manager to generate all 'sizes'


To get a better help : Politeness like Hello-A link-Your past actions precisely described
Check my extensions : more than 30 available
who I am and what I do : http://fr.gravatar.com/flop25
My gallery : an illustration of how to integrate Piwigo in your website

Offline

 

#3 2013-10-21 08:50:18

hpsam
Member
2005-06-28
93

Re: [resolved] pwg_high, TN- from 2.x to 2.5 how to migrate ?

Cool one point solve. thanks.

Now I have "just to" move all galleries/myCat/pwg_high/*.jpg TO galleries/myCat/*.jpg before batch operation. That is simpler that moving and renaming three sizes of file.

Is there an estimate time to generate 17000 thumbails and medium (800x800) photos from 5 megapixels photos ?

Offline

 

#4 2013-10-21 09:16:50

flop25
Piwigo Team
2006-07-06
7037

Re: [resolved] pwg_high, TN- from 2.x to 2.5 how to migrate ?

hpsam wrote:

Now I have "just to" move all galleries/myCat/pwg_high/*.jpg TO galleries/myCat/*.jpg before batch operation. That is simpler that moving and renaming three sizes of file.

this is done by the update, did you checked that?


To get a better help : Politeness like Hello-A link-Your past actions precisely described
Check my extensions : more than 30 available
who I am and what I do : http://fr.gravatar.com/flop25
My gallery : an illustration of how to integrate Piwigo in your website

Offline

 

#5 2013-10-21 10:11:16

hpsam
Member
2005-06-28
93

Re: [resolved] pwg_high, TN- from 2.x to 2.5 how to migrate ?

Update have done nothing on this.
pwg_high photos are about 2 Mio (5 megapixels) and in root gallery folders, photos are the same than before update, about 130 kio (800x600).
In the gallery, the original link display the 800x600 photo.

If It can help to find a bug in update, I use symbolic link for the galleries to share photos between two installations of Piwigo (one with all galleries and another with just some galleries) and to not backup old photos each day. Saving backup space and time.

Before update, in 2.0.10 :

Code:

www
|- galleries-old
  \- 2012 => real folder
|- galleries
  |- 2012 => sym link to "www/galleries-old/2012"
  \- 2013 => real folder
|- piwigo1
  \- galleries => sym link to "www/galleries"
|- piwigo2
  \- galleries => real folder
    |- 2012 => real folder
      \- OneGallery => sym link to "www/galleries/2012/OneGallery"
    |- 2013 => real folder
      \- AnOtherGallery => sym link to "www/galleries/2013/AnOtherGallery"

After update, in 2.5 :

Code:

www
|- galleries-i => real folder
|- galleries-old
  \- 2012 => real folder
|- galleries
  |- 2012 => sym link to "www/galleries-old/2012"
  \- 2013 => real folder
|- piwigo1
  |- _data/i/galleries => sym link to "www/galleries-i"
  \- galleries => sym link to "www/galleries"
|- piwigo2
  |- _data/i/galleries => sym link to "www/galleries-i"
  \- galleries => real folder
    |- 2012 => real folder
      \- OneGallery => sym link to "www/galleries/2012/OneGallery"
    |- 2013 => real folder
      \- AnOtherGallery => sym link to "www/galleries/2013/AnOtherGallery"

Last edited by hpsam (2013-10-21 10:12:27)

Offline

 

#6 2013-10-21 10:21:27

flop25
Piwigo Team
2006-07-06
7037

Re: [resolved] pwg_high, TN- from 2.x to 2.5 how to migrate ?

I hope you use the same size parameters because it gonna be a mess in www/galleries-i

it migth be a permission issue: what are the owner and chmod of files in www/galleries-old/2012 and all the folders leading to those files


To get a better help : Politeness like Hello-A link-Your past actions precisely described
Check my extensions : more than 30 available
who I am and what I do : http://fr.gravatar.com/flop25
My gallery : an illustration of how to integrate Piwigo in your website

Offline

 

#7 2013-10-21 11:08:52

hpsam
Member
2005-06-28
93

Re: [resolved] pwg_high, TN- from 2.x to 2.5 how to migrate ?

Yes I use the same sizes.

I think permissions issue is the good reason, but no error/warning message was shown during update.
I always put photos by FTP for the three sizes and Apache user have no write rights on the galleries folders.

I have 3 others (and smaller) galleries to migrate, for these I'm going to change write rights before upgrate and check if photos are moved during upgrade process.

Offline

 

#8 2013-10-21 11:21:22

flop25
Piwigo Team
2006-07-06
7037

Re: [resolved] pwg_high, TN- from 2.x to 2.5 how to migrate ?

you might need http://piwigo.org/dev/browser/trunk/ins … tabase.php to put as Personal plugin using LocalFiels Editor to update your installs wrongly updated


To get a better help : Politeness like Hello-A link-Your past actions precisely described
Check my extensions : more than 30 available
who I am and what I do : http://fr.gravatar.com/flop25
My gallery : an illustration of how to integrate Piwigo in your website

Offline

 

#9 2013-10-21 13:08:29

hpsam
Member
2005-06-28
93

Re: [resolved] pwg_high, TN- from 2.x to 2.5 how to migrate ?

Not usable because tn_ext, has_high, high_filesize, high_width, high_height columns where removed during upgrade process.

I'm going to write a bash script to do this, I prefer to have full control on what I'm doing that copy paste part of PHP code.

Thanks for your help and time.

Offline

 

#10 2013-10-21 13:10:59

flop25
Piwigo Team
2006-07-06
7037

Re: [resolved] pwg_high, TN- from 2.x to 2.5 how to migrate ?

okay :)
have a good day


To get a better help : Politeness like Hello-A link-Your past actions precisely described
Check my extensions : more than 30 available
who I am and what I do : http://fr.gravatar.com/flop25
My gallery : an illustration of how to integrate Piwigo in your website

Offline

 

#11 2013-10-21 20:13:30

hpsam
Member
2005-06-28
93

Re: [resolved] pwg_high, TN- from 2.x to 2.5 how to migrate ?

I'm back.

I test batch action to regenerate th and me from original. Based on my test It will took more than 8 hours at 100% CPU to create my 17.000 th and  me photos. It's not possible to slow access to all the hosted sites during 8 hours.

So I wrote a bash script to move existing files at the good places. But when I go to visit an album, Piwigo regenerate the existing th and me files, replacing those I put in place.

Is there something to do to tell Piwigo that files exist ?
I made my script to be sure that filesystem creation date of originals files are older than th and me files.

Offline

 

#12 2013-10-22 01:09:42

hpsam
Member
2005-06-28
93

Re: [resolved] pwg_high, TN- from 2.x to 2.5 how to migrate ?

After reverse engineering on i.php, I found that there is a compare between file mtime and params mtime. When I move file on linux, ctime is modified but not mtime, this is normal.
Derivative params mtime is the date of my upgrade and files mtime are from 2004 to now. So all my th and me file are regenerated because they are older than "Derivative params mtime" despite that files sizes are good.

I wrote a script to do all the actions for migration of files.
I avoid more than 8 hours of 100% working CPU on my server and done the job in few minutes.

Code:

#!/bin/bash
# WARNING SCRIPT ONLY TEST WITH MY GALLERY USE IT AT YOUR OWN RISK.
# BE SURE TO HAVE A BACKUP OF YOUR PHOTOS BEFORE USING IT.
#
# @Author Hpsam webmaster {AT} hpsam [DOT] info
# 
# Migration script from lower to 2.4 to 2.4 and upper galleries format.
# This script is here to avoid recalculation of all thumbnails and mediums photos.
# - Move web size *.jpg to *-me.jpg in new _data folder,
# - Move thumbnail/TN-*.jpg to *-th.jpg in new _data folder,
# - Move pwg_high/*.jpg to original size folder
#
# Script test with 17.114 photos in 598 physical albums
# On my server, less than 1 minute to reorganize 2.000 photos (compare to 1 hour for Piwigo to generate th and me files)


# Path without ending / of multiple sizes photos. Eg: '/home/mySite/www/_data/i/galleries'
startGalDir='/home/mySite/www/_data/i/galleries'
# Path without ending / of originals photos. Eg: '/home/mySite/www/galleries'
startDataDir='/home/mySite/www/galleries'
# Apache user:group to set on folder and file created or moved. Eg: 'www-data:www-data'
userGroup='www-data:www-data'

function recursDir {
  local tree=$1
  cd $startGalDir$tree
  local subDirs=$(find * -type d -prune)
  for subDir in $subDirs; do
    local tree=$1
    local photos=''
    local moveMe=0
    local moveTh=0
    cd $startGalDir$tree
    
    if [ "$subDir" != "thumbnail" -a "$subDir" != "pwg_high" ]; then
      cd $startGalDir$tree$subDir
      
      mkdir -p "$startDataDir$tree$subDir"
      echo "Create folder \"$startDataDir$tree$subDir\""
      [ ! -f "$startDataDir$tree$subDir/index.htm" ] && echo 'Not allowed!' > "$startDataDir$tree$subDir/index.htm"
      
      # pwg_high folder exist 
      if [ -d "$startGalDir$tree$subDir/pwg_high" ]; then
        # Rename web size photo to *-me.jpg
        photos=$(find * -maxdepth 0 -type f -name '*.jpg')
        for medium in $photos; do
          me="${medium%.jpg}-me.jpg"
          [ "$medium" = "$me" -o "$medium" = "-me.jpg" ] && continue
          mv -f "$startGalDir$tree$subDir/$medium" "$startGalDir$tree$subDir/$me"
        done
        moveMe=1
        
        # Move HR photos on level up.
        mv -f "$startGalDir$tree$subDir/pwg_high/"*.jpg "$startGalDir$tree$subDir/"
        rm -rf "$startGalDir$tree$subDir/pwg_high"
      fi
      
      # Rename thumbnails to *-th.jpg and moved them on level up.
      if [ -d "$startGalDir$tree$subDir/thumbnail" ]; then
        photos=$(find "thumbnail/"* -maxdepth 0 -type f -name 'TN-*')
        for thumb in $photos; do
          th=$(echo $thumb | sed -e 's#thumbnail/TN-\(.*\)\.jpg#\1-th.jpg#')
          [ "$thumb" = "$th" ] && continue
          mv -f "$startGalDir$tree$subDir/$thumb" "$startGalDir$tree$subDir/$th"
        done
        rm -rf "$startGalDir$tree$subDir/thumbnail"
        moveTh=1
      fi
      
      # Move mediums and thumbnails to _data/i/galleries folder
      [ $moveTh -eq 1 ] && mv -f "$startGalDir$tree$subDir/"*-th.jpg "$startDataDir$tree$subDir/"
      [ $moveMe -eq 1 ] && mv -f "$startGalDir$tree$subDir/"*-me.jpg "$startDataDir$tree$subDir/"
      
      # Touch moved files to be sure create time is newer than derivative_params last_mod_time and original create time
      [ $moveMe -eq 1 -o $moveTh -eq 1 ] && touch "$startDataDir$tree$subDir/"*
      
      recursDir $tree$subDir"/"
    fi
  done
}

cd $startDir
recursDir '/'
chown -R $userGroup "$startDataDir"
chmod -R u+rw,g+rw "$startDataDir"

Offline

 

Board footer

Powered by FluxBB

github twitter newsletter Donate Piwigo.org © 2002-2024 · Contact