用python实现爬取奥特曼图片实例

爬取网址：http://www.ultramanclub.com/allultraman/

使用工具：pycharm，requests

进入网页

打开开发者工具

点击 Network

 刷新网页，获取信息

其中的Request URL就是我们所爬取的网址

滑到最下有一个User-Agent，复制

 向服务器发送请求

200意味着请求成功

使用 response.text 获取文本数据

 可以看到有些乱码

使用encode转换

import requests

url = 'http://www.ultramanclub.com/allultraman/'

headers = {
   'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36'
}

response = requests.get(url = url,headers=headers)
html = response.text
Html=html.encode('iso-8859-1').decode('gbk')
print(Html)

 接下来开始爬取需要的数据

使用Xpath获得网页链接

要使用Xpath必须先导入parsel包

import requests
import parsel

def get_response(html_url):
   headers = {
       'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36'
   }

response = requests.get(url = html_url,headers=headers)
   return response

url = 'http://www.ultramanclub.com/allultraman/'
response = get_response(url)
html=response.text.encode('iso-8859-1').decode('gbk')
selector = parsel.Selector(html)

period_hrefs = selector.xpath('//div[@class="btn"]/a/@href')  #获取三个时代的网页链接

for period_href in period_hrefs:
   print(period_href.get())

可以看到网页链接不完整，我们手动给它添加上去period_href = 'http://www.ultramanclub.com/allultraman/' + period_href.get()

 进入其中一个网页

跟之前的操作一样，用Xpath获取奥特曼的网页信息

for period_href in period_hrefs:
   period_href = 'http://www.ultramanclub.com/allultraman/' + period_href.get()
   # print(period_href)
   period_response = get_response(period_href).text
   period_html = parsel.Selector(period_response)
   lis = period_html.xpath('//div[@class="ultraheros-Contents_Generations"]/div/ul/li/a/@href')
   for li in lis:
       print(li.get())

运行后同样发现链接不完整

li = 'http://www.ultramanclub.com/allultraman/' + li.get().replace('./','')

拿到网址后继续套娃操作,就可以拿到图片数据

png_url = 'http://www.ultramanclub.com/allultraman/' + li_selector.xpath('//div[@class="left"]/figure/img/@src').get().replace('../','')

完整代码

import requests
import parsel
import os

dirname = "奥特曼"
if not os.path.exists(dirname):     #判断是否存在名称为奥特曼的文件夹，没有就创建
   os.mkdir(dirname)

def get_response(html_url):
   headers = {
       'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36'
   }

response = requests.get(url = html_url,headers=headers)
   return response

url = 'http://www.ultramanclub.com/allultraman/'
response = get_response(url)
html=response.text.encode('iso-8859-1').decode('gbk')
selector = parsel.Selector(html)

period_hrefs = selector.xpath('//div[@class="btn"]/a/@href')  #获取三个时代的网页链接

for period_href in period_hrefs:
   period_href = 'http://www.ultramanclub.com/allultraman/' + period_href.get()

period_html = get_response(period_href).text
   period_selector = parsel.Selector(period_html)
   lis = period_selector.xpath('//div[@class="ultraheros-Contents_Generations"]/div/ul/li/a/@href')
   for li in lis:
       li = 'http://www.ultramanclub.com/allultraman/' + li.get().replace('./','')     #获取每个奥特曼的网址
       # print(li)
       li_html = get_response(li).text
       li_selector = parsel.Selector(li_html)
       url = li_selector.xpath('//div[@class="left"]/figure/img/@src').get()
       # print(url)

if url:
           png_url = 'http://www.ultramanclub.com/allultraman/' + url.replace('.', '')
           png_title =li_selector.xpath('//ul[@class="lists"]/li[3]/text()').get()
           png_title = png_title.encode('iso-8859-1').decode('gbk')
           # print(li,png_title)
           png_content = get_response(png_url).content
           with open(f'{dirname}\\{png_title}.png','wb') as f:
               f.write(png_content)
           print(png_title,'图片下载完成')
       else:
           continue

当爬到 奈克斯特奥特曼的时候，就会返回None，调了半天，也没搞懂，所以用if url：语句跳过了奈克斯特奥特曼，有没有大佬知道原因

url = li_selector.xpath('//div[@class="left"]/figure/img/@src').get()

来源：https://blog.csdn.net/K4239527/article/details/122842708