Piwigo.org

wsloand · 2014-11-20 12:14:20

Hi,

I recently started using Piwigo having migrated from Menalto Gallery2. One feature that I've been looking for is support for raw file formats, and I've searched and seen that it doesn't appear to be supported (e.g. [Forum, topic 18873] Canon CR2 and piwigo).

As I've looked through the source to see what may be available to plug in support for raw files, I found the upload code has support for some specific file types (tiff, movies, and pdf files) where it would be simple to add support for raw files by adding some code to use dcraw to create a .jpg in the pws_representative folder.

I'm happy to create that patch if it would be accepted.

I wonder if it may be a longer-term solution to make a file type plugin interface where upon load a plugin would register the extensions that it supports, and the plugins would be checked in order for support of an extension-- the first one supporting an extension does the conversion. And, the last plugin in the list is a default "Unsupported file type" error.

Another advantage of this plugin-type of file upload architecture is that it could also be included in plugins like menalto2piwigo for conversion from other types of galleries or optionally in the synchronization code.

I'm happy to help with this development if it would be of interest.

Thank you,

Bill

Piwigo version: 2.7.1
PHP version: 5.5.9-1ubuntu4.5
MySQL version: 5.5.40-0ubuntu0.14.04.1
Piwigo URL: http:// (not yet public)

xbgmsharp · 2014-11-20 16:02:37

You can easily create a plugin to allow upload of RAW file.
The problem is how to displays those RAW file.
Another problem is the size of the file.
Unless you convert them on upload.

flop25 · 2014-11-20 17:01:34

Hello
by changing the local config you can uplaod what you want, then you need a representative picture

wsloand · 2014-11-20 17:28:57

My understanding is that the following would allow for display of the raw file:

1. Upload the file
2. Convert it to a .jpg using dcraw (or imagemagick or ...) and place that .jpg in the pwg_representation directory

@xbgmsharp For file sizes, it seems similar to videos, and there are several methods for including videos in piwigo.

@flop25, I've updated the config to allow for raw files to be uploaded. My thought was to automate step 2 above so that it would happen automatically on upload. After that thought, it seemed like a more general upload mechanism could improve the ability to add new file types. What I thought was:

Modify the add_uploaded_file function in admin/include/functions_upload.inc.php :

* Update the "// this photo is new" section of the script so that it called a function called "generate_representation"
* There would be some default generate_representation functions for .jpg, .png, .gif that effectively did nothing.
* There would be a new .tif generate_representation function that would make the pwg_representation
* One for ffmpeg files
* One for pdfs, ...
* One for other allowed extensions that are not otherwise supported (i.e. the way that other files are currently allowed)
* And a final result that would die for unsupported file types.

Then, for people wanting to add new file type, they could make a very simple function that registers itself and creates the representation.

Do you think that would be accepted as a patch to the main source if I wrote it?

flop25 · 2014-11-20 17:42:01

yes personally that would be a good thing since raw are widely

Sven Peters · 2014-11-20 22:54:10

Dont worry too much about generating the representative picture from a raw file, most have an embedded jpeg anyway, which is easily (and fast!) extracted with exiftool.

flop25 · 2014-11-21 09:44:28

But exif tool is not installed on shared hosting. First if would check for exif tool then elseif IM enabled convert it

wsloand · 2014-11-21 13:25:24

The way that I'm writing it, it is easy to add a new tool for extraction. Essentially, for each file extension, it works through a list of tools that support the extension with a priority order. So, for example:

jpg -> do nothing (But you could add something if you want)
gif -> do nothing
png -> do nothing
tiff -> extract a jpg of the first page
cr2 -> use dcraw

In my files, the jpg that is embedded is a thumbnail only, and so I'd prefer using dcraw to do the more intensive extraction. But, if you have full-sized jpgs embedded (and exiftool installed), you could do something like:

cr2 -> use exiftool and if that doesn't work, use dcraw

And the exiftool method could be setup to check the dimensions of the raw image. If the .jpg that comes out isn't full size, then consider it not to have worked.

I hope to have a test version this weekend. It's mostly written right now, but not tested at all.

SnotRocket · 2014-11-21 15:31:52

wsloand wrote:
The way that I'm writing it, it is easy to add a new tool for extraction. Essentially, for each file extension, it works through a list of tools that support the extension with a priority order. So, for example:

jpg -> do nothing (But you could add something if you want)
gif -> do nothing
png -> do nothing
tiff -> extract a jpg of the first page
cr2 -> use dcraw

In my files, the jpg that is embedded is a thumbnail only, and so I'd prefer using dcraw to do the more intensive extraction. But, if you have full-sized jpgs embedded (and exiftool installed), you could do something like:

cr2 -> use exiftool and if that doesn't work, use dcraw

And the exiftool method could be setup to check the dimensions of the raw image. If the .jpg that comes out isn't full size, then consider it not to have worked.

I hope to have a test version this weekend. It's mostly written right now, but not tested at all.

PM me if you'd like me to help test... I'd love to have this work!

I use FTP management only.... if that helps with your testing, let me know!

Last edited by SnotRocket (2014-11-21 15:33:14)

plg · 2014-11-22 09:54:01

(very interesting discussion, yes it would be great if Piwigo could handle raw files in a nice/simple/transparent way for end users)

wsloand · 2014-11-22 13:22:01

I've made some good progress. The upload script now will load helper functions that handle the image and return an array of:

* Did the handling of the image succeed?
* If a pwg_representative image was created, what was the representative_ext (or null if no representative was needed).

This now works to create the pwg_representative via the upload form (not yet hooked into ftp upload/synchronization), the file goes into the upload directory correctly (no change there), and the representative_ext is set in the database correctly (also no change there).

Where I'm having difficulty is that I'm not seeing the pwg_representative files in the album view. I've been trying to trace this for a bit, but I'm hoping that someone more experienced can help here. As a guess, I think that the image.class.php needs to handle the pwg_representative file, but I'm not seeing it.

Finally, as I'm preparing a patch, is it preferred to prepare it relative to git or svn (the developer docs portion of the website mentions both without saying which is preferred from what I can see)?

wsloand · 2014-11-22 14:27:10

To facilitate discussion of the code, I've opened a feature request with my current diff (relative to svn trunk) at [Bugtracker] ticket 3182

wsloand · 2014-11-24 02:12:18

I've reached a sticking point with my upload method update. I'm creating a pwg_representative using the original_to_representative function, and the representative_ext is in the database, but the representative image is not displaying.

Can one of the more experienced developers take a quick look and see if I'm doing something obviously wrong?

plg · 2014-11-24 15:50:05

Hi wsloand,

I have quickly read your very interesting patch. I don't understand why you have placed upload_helpers in the plugins directory (upload_helpers is obviously not a plugin for me, not the way it was coded, but using plugins would certainly be a good idea).

I'll give a try later and try to reprodude the problems you encounter.

wsloand · 2014-11-25 02:56:24

Thanks for looking into it Pierrick.

I didn't know the best way to setup the directories for the upload plugins. My thought was that upload plugins would make sense to be loadable where they self-register their file types. I put it in the plugins directory since that was the closest to a good place that I saw for the files. I'd be interested to hear your thought of where a better place would be to put them (and then happy to put them there).

I didn't see a straight-forward way to write this as a normal Piwigo plugin.

Piwigo.org

Announcement

#1 2014-11-20 12:14:20

Raw File SUpport

#2 2014-11-20 16:02:37

Re: Raw File SUpport

#3 2014-11-20 17:01:34

Re: Raw File SUpport

#4 2014-11-20 17:28:57

Re: Raw File SUpport

#5 2014-11-20 17:42:01

Re: Raw File SUpport

#6 2014-11-20 22:54:10

Re: Raw File SUpport

#7 2014-11-21 09:44:28

Re: Raw File SUpport

#8 2014-11-21 13:25:24

Re: Raw File SUpport

#9 2014-11-21 15:31:52

Re: Raw File SUpport

wsloand wrote:

#10 2014-11-22 09:54:01

Re: Raw File SUpport

#11 2014-11-22 13:22:01

Re: Raw File SUpport

#12 2014-11-22 14:27:10

Re: Raw File SUpport

#13 2014-11-24 02:12:18

Re: Raw File SUpport

#14 2014-11-24 15:50:05

Re: Raw File SUpport

#15 2014-11-25 02:56:24

Re: Raw File SUpport

Board footer