Crawler/RE/2.Single-thread-crawler.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 爬虫步骤\n",
    "- 打开目标网页，先查看网页源代码\n",
    "- get网页源码\n",
    "- 找到想要的内容，找到规律，用正则表达式匹配，存储结果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Requests 收录了 python 的第三方http库\n",
    "- 完美地替代了 python 的 urllib2 模块\n",
    "- 更多的自动化，更友好的用户体验，更完善的功能"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Requests.get"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 将百度贴吧 python吧 的首页源代码获取下来\n",
    "html = requests.get('http://tieba.baidu.com/f?ie=utf-8&kw=python')\n",
    "# print html.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 这里并没有遇到取不到的情况，所以没有用到hea\n",
    "# 这个程序没有获得源代码，因为一个网站会对访问他的程序进行检查\n",
    "# hea是我们自己构造的一个字典，里面保存了user-agent\n",
    "# hea = {'User-Agent':'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36'}\n",
    "# html = requests.get('http://jp.tingroom.com/yuedu/yd300p/',headers = hea)\n",
    "\n",
    "html = requests.get('http://jp.tingroom.com/yuedu/yd300p/')\n",
    "html.encoding = 'utf-8'          #这一行是将编码转为utf-8否则中文会显示乱码。\n",
    "# print html.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "第二章 昔々、といってもせいぜい二十年ぐらい前のことなのだ...\n",
      "挪威的森林（中日对照） 内容简介： 汉堡机场一曲忧郁的《挪...\n",
      "藤野先生名文选读中日文对照 東京も格別のことはなかつた。上...\n",
      "夏目漱石 我是猫(中日对照) 吾輩は猫である 夏目漱石 一 吾輩...\n",
      "それお皿の絵柄よ 足のケガで中クラスの総合病院に入院しまし...\n",
      "あるけちん坊（ぼう）な男がおりました。 毎日毎日，ご飯どき...\n",
      "あるところに，たいへんへそまがりな息子（むすこ）がおりま...\n",
      "向こうから，お医者（いしゃ）がやってきました。そこへ店（...\n",
      "お盆休み（ぼんやすみ）に帰ってきた者（もの）同士（どうし...\n",
      "昔（むかし），三太（さんた）という，ばかな息子がおりまし...\n",
      "ある日のこと。 そこつ者が，瀬戸物（せともの）屋（や）へ，...\n",
      "植木（うえき）の大好きな旦那（だんな）がおりました。 ある...\n",
      "息子が表（おもて）で，凧（たこ）を揚（あ）げておりました...\n",
      "ある夜（や）のこと，お寺（てら）の庭（にわ）で，小僧（こ...\n",
      "はくうんしゅうしょく 一匹のトンボが夏の終わりを告げるわけ...\n",
      "先日、ある研修で聞いた言葉ですが、 「学習」の本質を端的に...\n",
      "闇夜（やみよ）に，二人の若い男が，こそこそ話しております...\n",
      "ある役人が誕生日のときに、下役たちは彼が鼠年だと聞き、お...\n",
      "東京の郊外に住む木村さんは、お酒を飲んでの失敗の多い人で...\n",
      "ある人が新調した絹の裾／裙（すそ、はかま）を着用して外出...\n"
     ]
    }
   ],
   "source": [
    "title = re.findall('color:#666666;\">(.*?)</span>',html.text,re.S)\n",
    "for each in title:\n",
    "    print each"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "300篇精选中日文对照阅读 289 挪威的森林（中日对照）第二章\n",
      "300篇精选中日文对照阅读 288  挪威的森林（中日对照）第一章\n",
      "300篇精选中日文对照阅读 287  藤野先生——名文选读\n",
      "300篇精选中日文对照阅读 286  夏目漱石 我是猫 第一章\n",
      "300篇精选中日文对照阅读 285 それお皿の絵柄よ\n",
      "300篇精选中日文对照阅读 284 つもり\n",
      "300篇精选中日文对照阅读 283 遺言（ゆいごん）\n",
      "300篇精选中日文对照阅读 282 やぶ医者\n",
      "300篇精选中日文对照阅读 281 表札\n",
      "300篇精选中日文对照阅读 280 ととの目\n",
      "300篇精选中日文对照阅读 279 つぼ\n",
      "300篇精选中日文对照阅读 278 用心\n",
      "300篇精选中日文对照阅读 277 凧揚げ\n",
      "300篇精选中日文对照阅读 276 星取り\n",
      "300篇精选中日文对照阅读 275 白云愁色\n",
      "300篇精选中日文对照阅读 274 学習とはパラダイム変換だ\n",
      "300篇精选中日文对照阅读 273 偷柿子的贼\n",
      "300篇精选中日文对照阅读 272 胃口更大\n",
      "300篇精选中日文对照阅读 271 这不是你家\n",
      "300篇精选中日文对照阅读 270 矫揉做作\n"
     ]
    }
   ],
   "source": [
    "chinese = re.findall('color: #039;\">(.*?)</a>',html.text,re.S)\n",
    "for each in chinese:\n",
    "    print each"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 爬取多个页码的网页\n",
    "爬虫只能爬网页上看得见的内容"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "url = 'https://www.crowdfunder.com/browse/deals'\n",
    "html = requests.get(url).text\n",
    "# print html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DIGITZS \n",
      "AUGMATE\n",
      "GOCOIN\n",
      "GOODWORLD\n",
      "REVL\n",
      "IECROWD\n",
      "SELFIE WITH ME\n",
      "LOOK AT YOU\n",
      "SANTO DIABLO MEZCAL\n",
      "NINJA METRICS\n"
     ]
    }
   ],
   "source": [
    "# 因为这个网站的例子，已经不用 show more，而改成页码了，所以方法和上次练习的方法一样，下面这个代码没有用了，它只能搜到第一个页面的内容\n",
    "# 构造字典 data\n",
    "# 注意这里的page后面跟的数字需要放到引号里面。\n",
    "url = 'https://www.crowdfunder.com/browse/deals&template=false'\n",
    "\n",
    "data = {\n",
    "    'entities_only':'true',\n",
    "    'page':'2'\n",
    "}\n",
    "html_post = requests.post(url,data=data)\n",
    "title = re.findall('\"card-title\">(.*?)</div>',html_post.text,re.S)\n",
    "for each in title:\n",
    "    print each"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ULTRASOUND SOLUTIONS\n",
      "BLACK FLAG ALEWERKS\n",
      "AMERIVEST LIMITED PARTNERSHIP\n",
      "ENDURING INVESTMENTS\n",
      "HURDL ENTERPRISES\n",
      "VAPETEK\n",
      "ZOGANIC DELICIOUS HEALTH\n",
      "WHICH WINERY\n",
      "PREMIER ONE LENDERS\n",
      "PAX - PERSONAL AIRLINE EXCHANGE\n",
      "LOEB'S CRUNCH\n",
      "EM&N8, CONTROLLERS INCORPORATED\n",
      "CLOUDBURST ROOM ESCAPE\n",
      "MOO MOO FARMS\n",
      "BIOFAB\n",
      "OPENDOOR COLIVING\n"
     ]
    }
   ],
   "source": [
    "# 用这个网址来取标题\n",
    "url = 'https://www.crowdfunder.com/?q=filter&page=3'\n",
    "html = requests.get(url).text\n",
    "# html = requests.get(url).text\n",
    "title = re.findall('\"card-title\">(.*?)</div>', html, re.S)\n",
    "for each in title:\n",
    "    print each"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "https://www.crowdfunder.com/?q=filter&page=1\n",
      "[]\n",
      "2\n",
      "https://www.crowdfunder.com/?q=filter&page=2\n",
      "[]\n",
      "3\n",
      "https://www.crowdfunder.com/?q=filter&page=3\n",
      "[]\n"
     ]
    }
   ],
   "source": [
    "url = 'https://www.crowdfunder.com/?q=filter&page=1'\n",
    "for i in range(1,4):\n",
    "    print i\n",
    "    \n",
    "    new_link = re.sub('page=\\d+','page=%d'%i, url, re.S)\n",
    "    print new_link\n",
    "    \n",
    "    title = re.findall('\"card-title\">(.*?)</div>', new_link, re.S)\n",
    "    print title                            # title是空的，因为规律不对了，需要自己重新找一下规律\n",
    "    \n",
    "    for each in title:\n",
    "        print each"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 爬取极客学院课程列表"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "url = 'http://www.jikexueyuan.com/course/?pageNum=2'\n",
    "html = requests.get(url).text\n",
    "# print html\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "classinfo = re.findall('(<li id=.*?</li>)', html, re.S)\n",
    "# for each in classinfo:\n",
    "#     print each"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 下面三行代码的功能，是在windows内，将命令提示符的代码强制转化成 utf－8\n",
    "# import sys\n",
    "# reload(sys)\n",
    "# sys.setdefaultencoding(\"utf-8\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 分析代码：\n",
    "下面代码，就是将 极客学院课程页的前5页里，课程的信息提取出来，存进一个txt文件中。\n",
    "先运行，看结果怎么样\n",
    "\n",
    "### 从大到小：\n",
    "- 1.产生不同的页码的链接\n",
    "- 2.每个链接，先获取网页源代码\n",
    "- 3.每个页面内，先抓每个课程的版块\n",
    "- 4.每个课程版块内，抓title，content，time，level，people，存到字典里\n",
    "- 5.最终结果保存到txt文件中\n",
    "\n",
    "- Tips：\n",
    "网页源码有时会变的，不要照搬下面代码，出不来结果时，自己找一下匹配的规律，重新写匹配规则。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class spider(object):\n",
    "    def __init__(self):\n",
    "        print u'开始爬取内容。。。'\n",
    "\n",
    "#changepage用来生产不同页数的链接\n",
    "    def changepage(self,url,total_page):\n",
    "        now_page = int(re.search('pageNum=(\\d+)',url,re.S).group(1))\n",
    "        page_group = []\n",
    "        for i in range(now_page,total_page+1):\n",
    "            link = re.sub('pageNum=\\d+','pageNum=%s'%i,url,re.S)\n",
    "            page_group.append(link)\n",
    "        return page_group\n",
    "    \n",
    "#getsource用来获取网页源代码\n",
    "    def getsource(self,url):\n",
    "        html = requests.get(url)\n",
    "        return html.text\n",
    "    \n",
    "#geteveryclass用来抓取每个课程块的信息\n",
    "    def geteveryclass(self,source):\n",
    "        #everyclass = re.findall('(<li deg=\"\".*?</li>)',source,re.S)                      # This code is old.\n",
    "        everyclass = re.findall('(<li id=.*?</li>)',source,re.S)\n",
    "        return everyclass\n",
    "    \n",
    "#getinfo用来从每个课程块中提取出我们需要的信息\n",
    "    def getinfo(self,eachclass):\n",
    "        info = {}\n",
    "        #info['title'] = re.search('target=\"_blank\">(.*?)</a>',eachclass,re.S).group(1)   # This code is old.\n",
    "        info['title'] = re.search('title=\"(.*?)\"',eachclass,re.S).group(1)\n",
    "        \n",
    "        #info['content'] = re.search('</h2><p>(.*?)</p>',eachclass,re.S).group(1)         # This code is old.\n",
    "        info['content'] = re.search('<p style=\"height: 0px; opacity: 0; display: none;\">(.*?)</p>',eachclass,re.S).group(1)\n",
    "        \n",
    "        timeandlevel = re.findall('<em>(.*?)</em>',eachclass,re.S)\n",
    "        info['classtime'] = timeandlevel[0]\n",
    "        info['classlevel'] = timeandlevel[1]\n",
    "        info['learnnum'] = re.search('\"learn-number\">(.*?)</em>',eachclass,re.S).group(1)\n",
    "        return info\n",
    "\n",
    "#saveinfo用来保存结果到info.txt文件中\n",
    "    def saveinfo(self,classinfo):\n",
    "        f = open('info3.txt','a')\n",
    "        for each in classinfo:\n",
    "            f.writelines('title:' + each['title'] + '\\n')\n",
    "            f.writelines('content:' + each['content'] + '\\n')\n",
    "            f.writelines('classtime:' + each['classtime'] + '\\n')\n",
    "            f.writelines('classlevel:' + each['classlevel'] + '\\n')\n",
    "            f.writelines('learnnum:' + each['learnnum'] +'\\n\\n')\n",
    "        f.close()\n",
    "\n",
    "if __name__ == '__main__':\n",
    "\n",
    "    classinfo = []                                          #存放最终结果\n",
    "    url = 'http://www.jikexueyuan.com/course/?pageNum=1'    \n",
    "    jikespider = spider()\n",
    "    all_links = jikespider.changepage(url,5)                #产生不同的页码的链接\n",
    "    \n",
    "    for link in all_links:\n",
    "        print u'正在处理页面：' + link\n",
    "        html = jikespider.getsource(link)                   #每个链接，先获取网页源代码\n",
    "        everyclass = jikespider.geteveryclass(html)         #每个页面内，先抓每个课程的版块\n",
    "        for each in everyclass:\n",
    "            info = jikespider.getinfo(each)                 #每个课程版块内，抓title，content，time，level，people，存到字典里\n",
    "            classinfo.append(info)\n",
    "\n",
    "    jikespider.saveinfo(classinfo)                          #最终结果保存到txt文件中\n",
    "\n",
    "print 'done'"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}