Skip to content

fishjoy/spark-alluxio-blockstore

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

9 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Spark-alluxio-blockstore

Apache Spark off heap cache block manager over alluxio.

Configuration

spark.externalBlockStore.blockManager=org.apache.spark.storage.AlluxioBlockManager spark.externalBlockStore.url=alluxio://localhost:19998

Details

Because the api in Alluxio1.0 is quite different with Tachyon, the old TachyonBlockManager cannot work with the Alluxio1.0. AlluxioBlockManager can replace the default TachyonBlockManager.
The detail: https://github.com/chengqiangboy/blog/blob/master/spark/spark-off_heap.md
The way using AlluxioBlockManager is the same as default.

myRdd.persist(StorageLevel.OFF_HEAP)
myDataFrame.persist(StorageLevel.OFF_HEAP)

About

Use AlluxioBlockManager to intead TachyonBlockManager as spark's off_heap.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages