▲ 246 points
back
142 comments
TIFF, http://en.wikipedia.org/wiki/Tagged_Image_File_Format has been around since 1993. It supports layers. It supports compression. It supports paged(!) images. It has well-defined supported image encoding formats. It's already supported by all the image editing software you've ever used (from this millennia).
We have a Standard Layered Image Format. HURRAY. NEXT PROBLEM, PLEASE.
Image formats have come a long way. I think the author wants an image format that can replace .psd in a sensible way. TIFF sadly cannot do that. While it can contain layered raster data, it lacks a lot of other capabilities. The author explicitly mentions vector layers, but if you want parity with .PSD you're looking at also having to support adjustment layers (hue/curves/levels/etc), blending modes, opacity/fill, raster masks, clipping masks, vector masks, locks, annotations, guides, grids, gradients, tags, text, comps, channels, and countless other things I can't even think to list (I wont even dive into smart objects). Keep in mind that's just the layer data, there's tons of other metadata in the format as well. It is a behemoth of a file format, and the years of backwards compatibility inside of it have multiplied that complexity to levels of insanity. I have .psd files that if I enable backwards compatibility when saving will increase the file size by half a gig. Granted those are enormous source files to begin with, but with higher def screens on the horizon file sizes are increasing dramatically. Having something more sensible than .PSD is something we're going to have to deal with sooner or later.
Fun fact: Even though Adobe sits on the TIFF standard for more than 20 years, Photoshop doesn't support layered TIFF images.
Came here to say almost exactly this; of course, people being people, TIFF won't be "good enough", where "good enough" is either 1) invented in house (NIH syndrome), or 2) monetizable (TIFF is too open/not proprietary enough).
or if thats not good enough:
http://www.openexr.com/
has photoshop plugin aswell. No real support in the gimp though (cinepaint possibly has it though.)
FXG could have solved the single file problem by grouping all the files up and making a zip file, and picking a different extension. For example this is how Android (.apk) and iOS (.ipa) applications are distributed.
It also has the advantage that if you extract the zip you can now access whatever pieces are inside as separate files even by tools that don't understand the containing format. (eg a bunch of jpg/png inside the zip are perfectly viewable in any image viewer). Trying to do the same with SQLite is problematic as you have to write queries to grab the blobs and then dump them to disk.
Where SQLite as a format does shine is if you need to keep historical information such as an undo/redo log. That makes it good for an application format, but not for an interchange format.
> For example this is how Android (.apk) and iOS (.ipa) applications are distributed.
A more relevant example: this is how .docx files work, for very much the same reasons (one XML file at the root, and then a bunch of referenced resources.)
Does the zip format let you replace an arbitrary file inside an archive without rewriting the whole file? It's not uncommon to see Photoshop files that are 400+ MB, and rewriting the whole file on save would have some pretty poor performance. The other examples of files that use the zip approach are either distribution-only (ipa/apk/jar) or generally don't get very big (word processing).
Also done with CBR / CBZ files (comic book archives) -- basically a renamed .zip of images with filenames numbered in ascending order.
Exactly. An interchange format should be editable with standard tools included with windows, and it should be easy to write all the code needed to extract part of the file yourself. Sqlite has many benefits, but realistically the only way to interact with it is through someone else's code. That rules it out as a suitable interchange format.
Any performance benefits sqlite might have are easily outweighed by the ease of interacting with the zip-based format.
> Last summer, Adobe killed their image exchange format "FXG".
Edit: previously I had balked at this claim, but I'm wrong. While I can't find any official notice of its death, and it's still in use in Scene7 and somewhat supported in the open-sourced version of Flex, trawling message boards does indicate Adobe shying away from it in favor of SVG[1][2] and it's not supported in CS6 without an extension.
Still, his problem with FXG (or a format like it) seems superficial. Yes, FXG defines links to other assets instead of combining them into one file, but to me, that's a feature, not a problem: I can them programmatically swap out assets easily by just manipulating the references within the file. It makes it really useful for generating customized assets on demand. In practice, it's like HTML for layered documents. We don't complain that HTML is broken because it links to images instead of including them, do we?
If you don't like that you have to send a folder: zip it up and give it a custom extension if need be, a la browser extensions and Microsoft Office formats.
[1]: http://mail-archives.apache.org/mod_mbox/flex-dev/201303.mbo... [2]: http://mail-archives.apache.org/mod_mbox/flex-dev/201303.mbo...
"We don't complain that HTML is broken because it links to images instead of including them, do we?"
Actually, we do.
Saving a webpage and getting tons of file is a nightmare while the IE approach of lumping them all together is just so sensible it's hard to imagine a universe where this isn't standard.
Replacing resources is the special case. Sharing, moving or just storing an image isn't. Just get or write a tool that can replace the resources for you and you have completely satisfied the 0.001% as well.
"I want a filesystem in a single file. I'll pick the one I'm most comfortable with and pretend that it's much better than separate files".
Meanwhile, we'll ignore all the semantic issues around the data. What exactly _is_ a layer? Any given image file? What if some tools only handle e.g. alpha-layers? What if a layer is actually a filter kernel?
How are the layers composited? How are they ordered? Is it a linear order, or is there actually a tree of layers?
The reason PSD works so well is only partially that things are all in one file. It's also that the semantics of its contents are extremely well defined. (OK, if you're willing to consider PSD documentation "extremely well defined")
And Gus is almost completely punting on that part - define _that_ well enough, and it might make sense. Until then, it's just another VFS with a blob of assets that the receipient probably can't parse exactly as intended.
It's not going to happen for the same reason a standard 3D format will never happen. (and no neither FBX nor Collada fit).
The problem is, at least in the case of image editors (and 3d editors), standard formats arguably stifle innovation because you have to break the standard to add anything.
Maybe you decide it would be be better if all vector colors were stored as HSVA instead of RGBA except now no one can read your files. Maybe you'd prefer to store floating point colors with values greater than 1.0 to represent light emission. Maybe the standard only defines circles but you want eclipses. Maybe the standard says circles are defined based on rounded rect with the maximum roundness but you'd like them to be based on a center point and radius. Maybe the standard doesn't support text on a path but you want text on path. Maybe the standard doesn't support linking paths so that if the text doesn't fit on one path it bleeds into another. Maybe you'd like to justify text across multiple paths but that's not in the standard. Maybe you like columns but that's in the standard. Or you want to be able to define areas to be cut into pngs, each area with a specifiable filename. Or you want some of the settings to be per animation frame but the spec never thought about animation so your SOL. Or you want layer fx. Or you want to add a new layer effect. Or you want to be able to embed a PDF as a layer. etc.. etc.. etc..
Photoshop's basic chucky format is well known. It's not that hard to write some code to read all the chunks you care about. Putting those chunks in an SQLite format will not make it any easier to deal with new chunks your code does not understand.
Interestingly, in the GIS/Mapping space Mapbox has faced a similar kind of problem, albeit with image tiles rather than just layers. Their solution was to come up with an image format called mbtiles (http://mapbox.com/developers/mbtiles), which actually uses SQLite. It's not the same problem, but perhaps it throws some credibility to the use of SQLite as an on disk format for structured images.
This whole idea seems totally mad from a distance but as a developer I'd love to be able to open complex files as DBs.
What would be the downfalls of passing around a DB as an image file? Does the compression suck? Does the performance suck? What makes this a terrible idea?
EDIT: to clarify, by compression I'm referring to the non-lossy type - I'm assuming that within the db you'd have already processed assets.
> and Adobe, if you are listening you should really give SQLite a serious look
I'm not in the Photoshop group, but I do make liberal use of SQLite in the projects that I'm involved in at Adobe.
I'm interested in why Acorn ties the concept of a layer so tightly to a bmp or shape blob rather then segregating the two. It seems that having a library table that the layer table links to would allow for better reuse of images.
Library table <- pixel data
Layer table <- heirarchy/duplication of library items
Attributes table <- attributes of layers
Also, are the concept of Pages accounted for by the ordering of parentless layers, or does Acorn not support the notion of pages? Would that be its own table or a modification on the Layer table.
OpenEXR solved that problem years ago and it's supported by virtually all content creation tools. An open source implementation with a well designed C++ api and bindings to several other languages.
For reasons that I don't understand, it's adoption is weak outside of the VFX industry.
Won't this format leave behind traces of what was in the file before? I'm not sure that this is something that we want in a standard interchange format. Users will assume what they are sending is only what they see in an application. This is reasonable. I am concerned that using this format will lead to data leaks.
Whats wrong with SVG? Doesn't it accomplish what the OP wants to do already and is already a standard supported by a lot of applications? I must be missing something here
Zip archive containing an SVG file embedding multiple png/jpg/jpeg2000 images.
I would like to suggest this minor change:
create table layers (id text, parent_id text, sequence integer, uti text, name text, data blob);
create table layers (id text, parent_id text, sequence integer, uti text, name text, data blob, offset Vec2, bounds Vec2, rotation real);
"bounds" is the size of the axis-aligned bounding-box of the image. "offset" i a translation from the center of the axis-aligned bounding-box of the layer image data to the origin, and "rotation" is the rotation (in radians, CCW) of the layer. Order of application is rotation, then translation.
EDIT: Fixed missing def for "bounds".
What about OpenRaster? It's used in open source apps like MyPaint and Gimp.
One nice thing about using SQLite is that various compressed versions of the flattened image could just be stored as blobs in another table in the same file. For example, my camera shoots with a RAW+JPG mode that produces two files, why not store the RAW data as described here and the JPG as an easy to grab/display item for quick image viewers etc. Hell store various common versions of thumbnails and icons in there as well.
This is kind of like saying "We don't need PNG/JPEG/ETC, we just need to put image data in an XML file! It can do everything! Standardization at last!"
Which is true. Except for that little "implementation" detail.
I like the idea of using SQLite as a complex-data-in-single-file format. Hadn't really thought of that for cases like this. But it does nothing to solve the problem of standardization.
Standardizing a format/API upon a tool like SQLite was something that was proposed in the web world as WebSQL but scrapped due to not wanting an entire standard to rely on a specific code base but rather wanting to have the standard be possible to reimplementable by others and thus creating the IndexedDB standard instead.
I'm not proposing that an image format like this one should use IndexedDB, but if relying on SQLite means being reliant on a specific code base that has to be used by anyone that wants to read the file - then it's probably not such a good idea.
When discussing this I think one should also have in mind eg the move to web apps as well - a reliance on SQLite will make it hard for them to read the format. A standardized simple base format that can easily be supported in new languages would be preferable as I see it - but I'm no expert in the image processing area.
> When you want to send someone an image you want to pass them a single file, not an XML file with a folder of assets. While there are technical benefits to this, it's an incredible burden on the customer.
Or, you could just zip that whole thing up and everyone who doesn't have to care thinks it is "just one file", like .war's.
Some of your blobs seem to contain a lot of structured data: I would further specify a format for them somehow: perhaps as further tables that join to the main ones, perhaps as JSON, perhaps as protobufs. Burying application- or format-specific blobs in SQL fields just pushes the problem down a level.
The author didn't put the punchline in the title. Why did you put it in the title, HN submitter or editor?
Whatever happened to the debate around how in/efficient the blob storage format in SQLite was? Did that get changed in v3?
EDIT:
Found the official examination[0] on the performance of blob storage.
Correct me if I am wrong, but FXG files are an archive with an XML file and assets, just as xlsx files are.
How is this a problem? How could is possibly be easier to open a file in sqlite than it is to just extract the file into it's folder structure with any standard decompression software?
I don't know about you, but last time checked there is no easy way to open a file contained in a sqlite database in my editor directly. I for one would much prefer the archive method than using a sqlite database.
Also, I would rather edit xml, than edit a sqlite database. XML can be edited by any text editor. I'd say text editors are a bit more universal and widespread than sqlite editors.
And why not TIFF?
This is an interesting idea, though I cringe a little at using a relational database as a key/value store (as in the case of layer_attributes).
I wonder whether it would be more useful to just represent the whole thing as a big JSON structure with base64 blobs. Each layer can be an object in the "layers" array with the attributes as keys.
What I'd like is the ability for two editors to work on a file at the same time, so layer locks might need to be built in. A db seems to me, a novice in everything relevant(!), to be a good way to allow multiple applications access whilst controlling for collisions.
Don‘t think the discussion is complete without http://en.wikipedia.org/wiki/OpenRaster which is already supported in a number of programs.
IMHO SQLite is horrible format. using Matroska (.mkv) would be suitable for anything, vectors, bitmaps, audios and so on. It's essentially a container format like XML but for binary
WebM is based on mkv as well.
not a huge conspiracy theorist, but isn't this the kind of thing that would make us not rely on purchasing Adobe CS7 exclusively (for fear of incompatability) in the future?
Standards wise, open-source wise, and generally it's a good move (who wouldn't want to get a standard complex image format that would make sense to the common programmer?), but I don't think the finance department over at Adobe is going to like it very much?
.ora? (OpenRaster)
http://en.wikipedia.org/wiki/OpenRaster
gimp (plugin), krita, and mypaint
If someone does this right, it should go over pretty well for EDA (Electronics Design) file formats. EDA is a wasteland.
Good luck opening and modifying that 150MB psd file.
xkcd.com/927 :)
A limited subset of HTML would be nice.
How about an additional frames table?
this is like saying "i've invented a standard file format for all types of files. i call it binary. it's made up of zeroes and ones. i've solved interoperability forever".
Amen!