If this is of any help, the same error can be obtained just by running the tests. I recompiled caffe to use cuDNN, and wanted to run the tests: obtained this: (I skipped all the passing tests before)

[----------] 1 test from GaussianFillerTest/1, where TypeParam = double
[ RUN      ] GaussianFillerTest/1.TestFill
*** Aborted at 1425466715 (unix time) try "date -d @1425466715" if you are using GNU date ***
PC: @           0x50efd4 caffe::GaussianFillerTest_TestFill_Test<>::TestBody()
*** SIGSEGV (@0x6e0a000) received by PID 6705 (TID 0x2b18bebb3900) from PID 115384320; stack trace: ***
    @     0x2b18c4e9fd40 (unknown)
    @           0x50efd4 caffe::GaussianFillerTest_TestFill_Test<>::TestBody()
    @           0x70aa63 testing::internal::HandleExceptionsInMethodIfSupported<>()
    @           0x7016a7 testing::Test::Run()
    @           0x70174e testing::TestInfo::Run()
    @           0x701855 testing::TestCase::Run()
    @           0x704b98 testing::internal::UnitTestImpl::RunAllTests()
    @           0x704e27 testing::UnitTest::Run()
    @           0x4449fa main
    @     0x2b18c4e8aec5 (unknown)
    @           0x449b19 (unknown)
    @                0x0 (unknown)
make: *** [runtest] Segmentation fault (core dumped)

Anything I can do to help?

I tried to start bisecting to identify where this bug could have come from but after a make clean and a reinstall, I can't reproduce anymore.
@ShijianTang , might want to try to do that if you're still having this problem.

Hi,

Thanks for your help.

Now, I have solved this problem. The problem is that the the format of train.txt file for generating the lmdb is in incorrect format.

having same problem in caffe test when doing: make runtest

[----------] 1 test from LayerFactoryTest/0, where TypeParam = N5caffe8FloatCPUE
[ RUN ] LayerFactoryTest/0.TestCreateLayer
*** Aborted at 1426466145 (unix time) try "date -d @1426466145" if you are using GNU date ***
PC: @ 0x10af701b7 caffe::CuDNNConvolutionLayer<>::~CuDNNConvolutionLayer()
*** SIGSEGV (@0x0) received by PID 15366 (TID 0x7fff7eb7b300) stack trace: ***
@ 0x7fff93fcdf1a _sigtramp
@ 0x10b432b26 fatbinData
@ 0x10af7039f caffe::CuDNNConvolutionLayer<>::~CuDNNConvolutionLayer()
@ 0x10acfb314 caffe::LayerFactoryTest_TestCreateLayer_Test<>::TestBody()
@ 0x10aefe4fc testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x10aeeda9a testing::Test::Run()
@ 0x10aeee892 testing::TestInfo::Run()
@ 0x10aeeefa0 testing::TestCase::Run()
@ 0x10aef4b17 testing::internal::UnitTestImpl::RunAllTests()
@ 0x10aefed54 testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x10aef4829 testing::UnitTest::Run()
@ 0x10ab8432d main
@ 0x7fff902365c9 start
@ 0x3 (unknown)
/bin/sh: line 1: 15366 Segmentation fault: 11 .build_release/test/test_all.testbin 0 --gtest_shuffle
make: *** [runtest] Error 139

redid a:
make clean
make test
make runtest

previous failed test ran OK but,
[----------] 3 tests from SplitLayerTest/0, where TypeParam = N5caffe8FloatCPUE
[ RUN ] SplitLayerTest/0.Test
[ OK ] SplitLayerTest/0.Test (0 ms)
[ RUN ] SplitLayerTest/0.TestSetup
[ OK ] SplitLayerTest/0.TestSetup (0 ms)
[ RUN ] SplitLayerTest/0.TestGradient
[ OK ] SplitLayerTest/0.TestGradient (5 ms)
[----------] 3 tests from SplitLayerTest/0 (5 ms total)

this on failed:
[----------] 1 test from LayerFactoryTest/1, where TypeParam = N5caffe9DoubleCPUE
[ RUN ] LayerFactoryTest/1.TestCreateLayer
*** Aborted at 1426468417 (unix time) try "date -d @1426468417" if you are using GNU date ***
PC: @ 0x11479b13e cudnnDestroy
*** SIGSEGV (@0x30) received by PID 25063 (TID 0x7fff7eb7b300) stack trace: ***
@ 0x7fff93fcdf1a _sigtramp
@ 0x7fff5128dd62 (unknown)
@ 0x10ed611e8 caffe::CuDNNPoolingLayer<>::~CuDNNPoolingLayer()
@ 0x10ed6122f caffe::CuDNNPoolingLayer<>::~CuDNNPoolingLayer()
@ 0x10eae8cf4 caffe::LayerFactoryTest_TestCreateLayer_Test<>::TestBody()
@ 0x10ecec4fc testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x10ecdba9a testing::Test::Run()
@ 0x10ecdc892 testing::TestInfo::Run()
@ 0x10ecdcfa0 testing::TestCase::Run()
@ 0x10ece2b17 testing::internal::UnitTestImpl::RunAllTests()
@ 0x10ececd54 testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x10ece2829 testing::UnitTest::Run()
@ 0x10e97232d main
@ 0x7fff902365c9 start
@ 0x3 (unknown)
/bin/sh: line 1: 25063 Segmentation fault: 11 .build_release/test/test_all.testbin 0 --gtest_shuffle
make: *** [runtest] Error 139

I am encountering the same problem when using IMAGE_DATA layers:

I0426 16:48:21.890173 23626 layer_factory.hpp:74] Creating layer data
I0426 16:48:21.890197 23626 net.cpp:84] Creating Layer data
I0426 16:48:21.890213 23626 net.cpp:338] data -> data
I0426 16:48:21.890239 23626 net.cpp:338] data -> label
I0426 16:48:21.890254 23626 net.cpp:113] Setting up data
I0426 16:48:21.890269 23626 image_data_layer.cpp:36] Opening file 
I0426 16:48:21.890297 23626 image_data_layer.cpp:51] A total of 0 images.
*** Aborted at 1430081301 (unix time) try "date -d @1430081301" if you are using GNU date ***
PC: @     0x7f70e1333090 (unknown)
*** SIGSEGV (@0x0) received by PID 23626 (TID 0x7f70e224aa40) from PID 0; stack trace: ***
    @     0x7f70e0cced40 (unknown)
    @     0x7f70e1333090 (unknown)
    @     0x7f70e1b1c95c std::operator+<>()
    @     0x7f70e1b7a465 caffe::ImageDataLayer<>::DataLayerSetUp()
    @     0x7f70e1b4e986 caffe::BaseDataLayer<>::LayerSetUp()
    @     0x7f70e1b4ea89 caffe::BasePrefetchingDataLayer<>::LayerSetUp()
    @     0x7f70e1b97432 caffe::Net<>::Init()
    @     0x7f70e1b98ef2 caffe::Net<>::Net()
    @     0x7f70e1bd8260 caffe::Solver<>::InitTrainNet()
    @     0x7f70e1bd9373 caffe::Solver<>::Init()
    @     0x7f70e1bd9546 caffe::Solver<>::Solver()
    @           0x40c4b0 caffe::GetSolver<>()
    @           0x406481 train()
    @           0x404a21 main
    @     0x7f70e0cb9ec5 (unknown)
    @           0x404fcd (unknown)
Segmentation fault (core dumped)

Compiling with #USE_CUDNN := 1 and USE_CUDNN := 1 both produced the error.

anyone found a workaround ?

+1 same issue here

I did a bit of digging and it seems to me that this the "Error: (unix time) try if you are using GNU date" is unrelated to Caffe and the problem that you are encountering.

See here, it appears this is just a result of the logging library (glog) that shows this when a failure happens. So all the problems that are posted here are unrelated.

If you get this error, you should look at the stacktrace provided instead of the unix time thing.
This issue should be closed because it doesn't reflect an error in caffe.

Hi guys, I have the same problem, someone managed to solve?

@bunelr is quite right, there are many different unrelated errors here. The text which titles this issue is just a helpful hint for parsing the log message. You're welcome to open new tickets for specific, reproducible errors in master with DEBUG enabled.

I think most of this problem is due to the mis-use of cpu_data() mode and gpu_data() mode. I have encountered this problem while debugging something and I find that I should use cpu_data() instead of gpu_data(). The error is reasonable since the pointer to a gpu location doesn't mean anything in CPU. Data is in GPU. Hope this will help.

I got the same error. But after the following steps, everything is good:
make clean
make all
make test
make runtest