Python通过解析网页实现看报程序的方法
作者:shichen2014 发布时间:2022-07-16 14:27:12
标签:Python,解析,网页,方法
本文所述实例可以实现基于Python的查看图片报纸《参考消息》并将当天的图片报纸自动下载到本地供查看的功能,具体实现代码如下:
# coding=gbk
import urllib2
import socket
import re
import time
import os
# timeout in seconds
#timeout = 10
#socket.setdefaulttimeout(timeout)
timeout = 10
urllib2.socket.setdefaulttimeout(timeout)
home_url = "http://www.hqck.net"
home_page = ""
try:
home_page_context = urllib2.urlopen(home_url)
home_page = home_page_context.read()
print "Read home page finishd."
print "-------------------------------------------------"
except urllib2.URLError,e:
print e.code
exit()
except:
print e.code
exit()
reg_str = r'<a class="item-baozhi" href="/arc/jwbt/ckxx/\d{4}/\d{4}/\w+\.html" rel="external nofollow" ><span class.+>.+</span></a>'
news_url_reg = re.compile(reg_str)
today_cankao_news = news_url_reg.findall(home_page)
if len(today_cankao_news) == 0:
print "Cannot find today's news!"
exit()
my_news = today_cankao_news[0]
print "Latest news link = " + my_news
url_s = my_news.find("/arc/")
url_e = my_news.find(".html")
url_e = url_e + 5
print "Link index = [" + str(url_s) + "," + str(url_e) + "]"
my_news = my_news[url_s:url_e]
print "part url = " + my_news
full_news_url = home_url + my_news
print "full url = " + full_news_url
image_folder = "E:\\new_folder\\"
if (os.path.exists(image_folder) == False):
os.makedirs(image_folder)
today_num = time.strftime('%Y-%m-%d',time.localtime(time.time()))
image_folder = image_folder + today_num + "\\"
if (os.path.exists(image_folder) == False):
os.makedirs(image_folder)
print "News image folder = " + image_folder
context_uri = full_news_url[0:-5]
first_page_url = context_uri + ".html"
try:
first_page_context = urllib2.urlopen(first_page_url)
first_page = first_page_context.read()
except urllib2.HTTPError, e:
print e.code
exit()
tot_page_index = first_page.find("共")
tot_page_index = tot_page_index
tmp_str = first_page[tot_page_index:tot_page_index+10]
end_s = tmp_str.find("页")
page_num = tmp_str[2:end_s]
print page_num
page_count = int(page_num)
print "Total " + page_num + " pages:"
page_index = 1
download_suc = True
while page_index <= page_count:
page_url = context_uri
if page_index > 1:
page_url = page_url + "_" + str(page_index)
page_url = page_url + ".html"
print "News page link = " + page_url
try:
news_img_page_context = urllib2.urlopen(page_url)
except urllib2.URLError,e:
print e.reason
download_suc = False
break
news_img_page = news_img_page_context.read()
#f = open("e:\\page.html", "w")
#f.write(news_img_page)
#f.close()
reg_str = r'http://image\S+jpg'
image_reg = re.compile(reg_str)
image_results = image_reg.findall(news_img_page)
if len(image_results) == 0:
print "Cannot find news page" + str(page_index) + "!"
download_suc = False
break
image_url = image_results[0]
print "News image url = " + image_url
news_image_context = urllib2.urlopen(image_url)
image_name = image_folder + "page_" + str(page_index) + ".jpg"
imgf = open(image_name, 'wb')
print "Getting image..."
try:
while True:
date = news_image_context.read(1024*10)
if not date:
break
imgf.write(date)
imgf.close()
except:
download_suc = False
print "Save image " + str(page_index) + " failed!"
print "Unexpected error: " + sys.exc_info()[0] + sys.exc_info()[1]
else:
print "Save image " + str(page_index) + " succeed!"
page_index = page_index + 1
if download_suc == True:
print "News download succeed! Path = \"" + str(image_folder) + "\""
print "Enjoy it! ^^"
else:
print "news download failed!"
0
投稿
猜你喜欢
- 对想要在可视化的环境下制作复杂网页的专业网页制作者来说,Dreamweaver 已经渐渐在网页编辑工具市场中展露头角,成为专业人士
- 打过了趟深圳回来后,已经快半个月,在广州购书中心逛了下,发现2本前端书《重构HTML-改善WEB应用的设计》、《CSS3 实战》,看了一半《
- 一、property() 函数讲解了解 @property 装饰器之前,我们首先要了解内置函数的 property()。class prop
- 问题:在论坛发表回复时出现“The table is full”的提示,字面意义上是数据表已满的意思。因为很少有开发者遭遇单一表超过4G的情
- 这篇文章主要介绍了简单了解Python3 bytes和str类型的区别和联系,文中通过示例代码介绍的非常详细,对大家的学习或者工作具有一定的
- 本文实例讲述了Python爬虫之正则表达式基本用法。分享给大家供大家参考,具体如下:一、简介正则表达式,又称正规表示式、正规表示法、正规表达
- 环境准备Python3.6pip install Django==2.0.1pip install celery==4.1.0pip ins
- 这个主要应用于,获取用户输入的时候,防止用户不小心,多输入了一个空格,导致验证无法通过,多用于用户名跟密码的,好多情况下,大家复制的winr
- 随着因特网的发展,在网络环境中,数据库应用渐渐向操作简单、功能实用的方向发展。本文介绍如何利用ASP技术实现对数据库进行在线维护的方法,并给
- 一、目的之前在博文SQL Server数据库最小宕机迁移方案中提到了使用了完全备份+差异备份的功能完成了数据库的转移,但是这个方法在遇到了7
- 如何让图片自动缩放以适合界面大小,拿出你的Editplus,打开c_function.asp文件,找到UBBCode函数,在第417行有如下
- UCHOME的代码还是很不错的,学习一下! <?php /** * 定义一些常量 */ @define('IN_UCHOME&
- 有如下 Pandas DataFrame:import pandas as pdinp = [{'c1':10, '
- 例子:http.Handle("/tmpfiles/", http.StripPrefix("/tmpfile
- 终于能出来透口气,写点东西了。前段太忙,也很郁闷,现在调整过来点了。我的设计原则就是在满足需求的前提下,尽可能的简化,简化,再简化。有一次跟
- 1、IIS为一个死循的执行过程设定执行时间(缺省为90秒)超时事件:<%response.buffer=true%><BO
- 本文实例讲述了phpmyadmin中禁止外网使用的方法。分享给大家供大家参考。具体方法如下:首先,在phpmyadmin文件夹中找到 php
- Application Name(应用程序名称):应用程序的名称。如果没有被指定的话,它的值为.NET SqlClient Data Pro
- 问题描述:被搜索名字为:andy这时搜索andy正常,但是搜索a就搜不到。解决办法,在索引配置文件中的index中添加min_infix_l
- 本文实例讲述了Python实现的读取文件内容并写入其他文件操作。分享给大家供大家参考,具体如下:文件目录结构,如图:read_file.py