@siddontang I'm actually on my way to the airport to go on vacation in a couple hours so I don't have time to dive in deeper right now. I'd love to investigate further when I get back though.

I'm actually surprised that LMDB performs as well as it does for write operations. One reason why Bolt is slower than LMDB (although I'm surprised it's so much slower) is that you're using a lot of dangerous configuration options on LMDB:

https://github.com/siddontang/ledisdb/blob/master/store/mdb/mdb.go#L47

• NOSYNC & NOMETASYNC can leave your database corrupted in the event of a failure.
• WRITEMAP - really dangerous if you have any bugs in your code. You're essentially writing directly to the underlying data file.
• MAPASYNC - This flag can leave your database in a corrupted state.

I left these options out of Bolt because it makes the code more complicated (i.e. Bolt is 2KLOC, LMDB is 8KLOC) and because a database shouldn't be corruptible.

Adding a write-ahead log of some kind and grouping updates together into a single transaction can help substantially. Do you still have the coalescer integration code? I'd be curious to see how it's configured.

For a random-write-heavy workload, you're better off using something like LevelDB which writes helps to order random writes sequentially.

Thank you for the detailed reply.

I remove lmdb configuration you listed above and bench again, lmdb seems a little better but not much.

LMDB

set: 6415.51 requests per second
incr: 6356.34 requests per second
rpush: 5444.33 requests per second
lpop: 3953.94 requests per second
hset: 5919.33 requests per second
hincrby: 5922.07 requests per second
hdel: 5842.37 requests per second
zadd: 5813.88 requests per second
zincrby: 5753.64 requests per second
zrem: 5430.68 requests per second

BoltDB

set: 970.57 requests per second
incr: 3518.69 requests per second
rpush: 2789.03 requests per second
lpop: 2341.30 requests per second
hset: 3335.03 requests per second
hincrby: 3163.00 requests per second
hdel: 3404.88 requests per second
zadd: 3380.41 requests per second
zincrby: 3296.58 requests per second
zrem: 3391.19 requests per second

Because LedisDB supports replication, it may take the risk when system crashed. (Not 100% OK), I may still use these dangerous configurations.

Using coalescer is here. but the benchmark is very poor, maybe I use wrong?

Although LevelDB's random write is very good, it is not good for range get, especially for reverse range get. but LMDB or BoltDB can do it easily.

Later, like Redis, I may supply a mechanism for LMDB data saving. User can set sync at once or at every N seconds, or even not sync at all.

Can you get cpu, memory, and block profiles for the Bolt benchmarks? You can use something like Dave Cheney's profile library to make it easy.

I used net/http/pprof to profile it but found nothing performance bottleneck. BoltDB is slower than LMDB maybe Go is slower than C, I think.

heap

(pprof) top
Total: 1.0 MB
     0.5  50.0%  50.0%      0.5  50.0% allocg
     0.5  50.0% 100.0%      0.5  50.0% bufio.NewReaderSize
     0.0   0.0% 100.0%      0.5  50.0% _rt0_go
     0.0   0.0% 100.0%      0.5  50.0% bufio.NewReader

cpu

(pprof) top
Total: 797 samples
     488  61.2%  61.2%      488  61.2% runtime.mach_semaphore_wait
     137  17.2%  78.4%      137  17.2% runtime.kevent
      50   6.3%  84.7%       50   6.3% syscall.Syscall
      19   2.4%  87.1%       19   2.4% syscall.Syscall6
      18   2.3%  89.3%       18   2.3% runtime.mach_semaphore_signal
      14   1.8%  91.1%       14   1.8% runtime.usleep
       8   1.0%  92.1%        8   1.0% github.com/boltdb/bolt.(*pgids).Less
       8   1.0%  93.1%        8   1.0% runtime.memclr
       6   0.8%  93.9%        6   0.8% runtime.MSpan_Sweep
       6   0.8%  94.6%        6   0.8% scanblock
(pprof) top -cum
Total: 797 samples
       0   0.0%   0.0%      641  80.4% System
     488  61.2%  61.2%      488  61.2% runtime.mach_semaphore_wait
     137  17.2%  78.4%      137  17.2% runtime.kevent
       0   0.0%  78.4%      129  16.2% runtime.gosched0
       0   0.0%  78.4%       71   8.9% github.com/siddontang/ledisdb/server.func·003
       0   0.0%  78.4%       56   7.0% github.com/siddontang/ledisdb/server.(*respClient).run
       0   0.0%  78.4%       51   6.4% github.com/boltdb/bolt.(*DB).Update
       0   0.0%  78.4%       51   6.4% github.com/siddontang/ledisdb/ledis.(*tx).Commit
       0   0.0%  78.4%       51   6.4% github.com/siddontang/ledisdb/store/boltdb.(*DB).BatchPut
       0   0.0%  78.4%       51   6.4% github.com/siddontang/ledisdb/store/driver.(*WriteBatch).Commit

Hope you can add above dangerous configurations, then I can control how and when to sync.

Thank you!

@siddontang Can you dump the profiles out to SVG? It's easier to see the call stack from that. I'm curious why mach_semaphore_wait is taking up 61% of the CPU time.

@siddontang Also, I don't have any plans to add dangerous configuration options beside the DB.NoSync that's already there. Dangerous configuration options don't simply corrupt the database so it doesn't open or rolls back a transaction. Without an fsync, hard drives can reorder write operations so that data pages can be written after a meta page and your data is silently corrupt.

The mach_semaphore_wait issue makes me think there's lock contention somewhere but it's hard to say where (or whether it's in Bolt or Ledis) until we look at the SVG.

It's all ultimately a tradeoff. Bolt has lower random write performance but has significantly better sequential read performance than LevelDB.

Github not supports svg upload, so I share full here

Some parts:

mach_semaphore_wait is in LedisDB not BoltDB.

I think if someone cares security, reads more than writes, BoltDB is a good choice.

@siddontang Can you give me the steps to reproduce the benchmark? I found commands on the benchmark page but I didn't see how to select Bolt specifically. From your SVG, I don't think you have the Mac pprof fix. I can try it on mine and try it on Linux as well.

Ultimately, I don't think Bolt is the best choice for Ledis if a user is doing a lot of random writes unless you implement some kind of coalescing. Not sure why the coalescer isn't helping at all but we can figure that out.

Sorry that LedisDB has a painful configuration and an unclear guild, I will improve it as soon as I can.

I have not patched the mac pprof fix, sorry that too.

If you want to bench it yourself, you can do these below:

• runmake to build LedisDB, it will build ledis-server and ledis-benchmark
• copy ledisdb/etc/ledis.json to /etc/ledis.json which is ledis-server's default config path
• run ledis-server -db_name=boltdb
• run ledis-benchmark

Many users like Pure Go, and LedisDB now supports two pure Go storage: goleveldb and BoltDB, their advantages vary, user can choose the proper one based on the real situation.

or just make a local ledis.json and use -config=./ledis.json

@siddontang Here's the pprofs I got:

• w/o NoSync
• w/ NoSync

One optimization that Bolt hasn't added yet that LMDB has is using pwritev() to flush multiple pages at once. I added an issue (#238) for this so it can get added. It shouldn't be a difficult fix.

It looks like the performance is dominated by syscalls since you're doing so many tiny transactions. That optimization should help Bolt performance significantly.

One other thing to note, though, is that ledis' respClient is using a significant amount of CPU to read lines. That's probably worth optimizing on your side.

LedisDB uses Redis Protocol, respClient must read line by line to parse a request, I will try to optimize it later.

Thank you!