Skip to content
/ distribute_crawler Public

使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现

3.3k stars 1.6k forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

gnemoug/distribute_crawler

Error
Looks like something went wrong!

About

使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现

Resources

Readme
Activity

Stars

3.3k stars

Watchers

366 watching

Forks

1.6k forks
Report repository

Releases

No releases published

Packages

No packages published

Contributors 2

Languages