One question regarding this topic:

 self.db = lmdb.open(self.output_path,
                    map_size=1000000000000, # ~1TB
                    map_async=True,
                    max_dbs=0)

This 1 TB is quite a lot which is allocated for each and every dataset. Should this be made configurable?

1 TB is quite a lot

I agree it's a bit ridiculous. Has it caused a problem for you?

It does cause problems on 32-bit systems:
BVLC/caffe#1298
BVLC/caffe#2293

And it may cause problems on non-linux platforms:
https://groups.google.com/forum/#!topic/caffe-users/0RKsTTYRGpQ

But it seems to be the consensus choice for 64-bit systems:
https://lmdb.readthedocs.org/en/release/#environment-class
https://github.com/BVLC/caffe/blob/eeb428e380/src/caffe/util/db_lmdb.cpp#L9

Should this be made configurable?

That certainly seems reasonable.

Well I tried to create the mnist dataset on my machine and failed. I "only" had 1.5 tb of free space. But alone for 1 validation and 1 train set 2 TB of space are required.
I think it is unreasonable to assume that everybody can spare 2 TB for every imported dataset.
MNIST actually requires < 1 GB of memory but still would consume 2 TB.

I have also been trying to port DIGITS to windows and I can confirm that if you don't have enough space the 1TB map size fails. It seems that this is the behaviour on windows though... when I create the MNIST dataset with 1GB as map size, it creates a folder of 1Gb, whereas the leveldb version is only 47 mb.

Branched the LMDB discussion out into #206 since it's not really relevant to this pull request.