Java爬虫实现Jsoup利用dom方法遍历Document对象
作者:Cyril_KI 发布时间:2023-06-15 07:52:36
标签:Java,Jsoup,遍历,Document对象
先给出网页地址:
https://wall.alphacoders.com/featured.php?lang=Chinese
主要步骤:
利用Jsoup的connect方法获取Document对象
String html = "https://wall.alphacoders.com/featured.php?lang=Chinese";
Document doc = Jsoup.connect(html).get();
内容过长,就不再显示。
我们以这部分为例:
<ul class="nav nav-pills">
<li><a href="https://alphacoders.com/site/about-us" rel="external nofollow" rel="external nofollow" >About Us</a></li>
<li><a href="https://alphacoders.com/site/faq" rel="external nofollow" rel="external nofollow" >FAQ</a></li>
<li><a href="https://alphacoders.com/site/privacy" rel="external nofollow" rel="external nofollow" >Privacy Policy</a></li>
<li><a href="https://alphacoders.com/site/tos" rel="external nofollow" rel="external nofollow" >Terms Of Service</a></li>
<li><a href="https://alphacoders.com/site/acceptable_use" rel="external nofollow" rel="external nofollow" >Acceptable Use</a></li>
<li><a href="https://alphacoders.com/site/etiquette" rel="external nofollow" rel="external nofollow" >Etiquette</a></li>
<li><a href="https://alphacoders.com/site/advertising" rel="external nofollow" rel="external nofollow" >Advertise With Us</a></li>
<li><a id="change_consent">Change Consent</a></li>
</ul>
我们先找到所有的ul:
Elements elements = doc.getElementsByTag("ul");
输出如下:
<ul class="nav navbar-nav center">
<li> <a title="Submit Wallpapers" href="https://alphacoders.com/site/submit-wallpaper" rel="external nofollow" ><i class="el el-circle-arrow-up"></i> 提交</a> </li>
<li> <a href="https://alphacoders.com/contest" rel="external nofollow" ><i class="el el-gift"></i> 精美奖品</a> </li>
</ul>
<ul class="nav navbar-nav navbar-right center">
<li> <a href="language.php?lang=Chinese" rel="external nofollow" > <img src="https://static.alphacoders.com/wa/Chinese-flag.png" alt="Chinese-flag"> 中文 </a> </li>
<li> <a href="https://alphacoders.com/users/login" rel="external nofollow" ><i class="el el-user"></i> 登录</a> </li>
<li> <a href="https://alphacoders.com/users/register" rel="external nofollow" ><i class="el el-edit"></i> 注册</a> </li>
</ul>
<ul class="pagination">
<li class="active"><a id="prev_page" href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >< 上一页</a></li>
<li class="active"><a>1</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >2</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=3" rel="external nofollow" rel="external nofollow" rel="external nofollow" >3</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=4" rel="external nofollow" rel="external nofollow" rel="external nofollow" >4</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=5" rel="external nofollow" rel="external nofollow" rel="external nofollow" >5</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=6" rel="external nofollow" rel="external nofollow" rel="external nofollow" >6</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=7" rel="external nofollow" rel="external nofollow" rel="external nofollow" >7</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=8" rel="external nofollow" >8</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=9" rel="external nofollow" >9</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=10" rel="external nofollow" >10</a></li>
<li><a>...</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=319" rel="external nofollow" rel="external nofollow" rel="external nofollow" >319</a></li>
<li><a id="next_page" href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >下一页 ></a></li>
</ul>
<ul class="pagination">
<li class="active"><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >< 上一页</a></li>
<li class="active"><a>1</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >2</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=3" rel="external nofollow" rel="external nofollow" rel="external nofollow" >3</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=4" rel="external nofollow" rel="external nofollow" rel="external nofollow" >4</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=5" rel="external nofollow" rel="external nofollow" rel="external nofollow" >5</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=6" rel="external nofollow" rel="external nofollow" rel="external nofollow" >6</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=7" rel="external nofollow" rel="external nofollow" rel="external nofollow" >7</a></li>
<li><a>...</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=319" rel="external nofollow" rel="external nofollow" rel="external nofollow" >319</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >下一页 ></a></li>
</ul>
<ul class="pagination">
<li class="active"><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" ><< </a></li>
<li class="active"><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >< 上一页</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >下一页 ></a></li>
<li><a title="末页 (319)" href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=319" rel="external nofollow" rel="external nofollow" rel="external nofollow" > >></a></li>
</ul>
<ul class="pagination">
<li class="active"><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >1</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >2</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=3" rel="external nofollow" rel="external nofollow" rel="external nofollow" >3</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=4" rel="external nofollow" rel="external nofollow" rel="external nofollow" >4</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=5" rel="external nofollow" rel="external nofollow" rel="external nofollow" >5</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=6" rel="external nofollow" rel="external nofollow" rel="external nofollow" >6</a></li>
<li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=7" rel="external nofollow" rel="external nofollow" rel="external nofollow" >7</a></li>
</ul>
<ul class="nav nav-pills">
<li><a href="https://alphacoders.com/site/about-us" rel="external nofollow" rel="external nofollow" >About Us</a></li>
<li><a href="https://alphacoders.com/site/faq" rel="external nofollow" rel="external nofollow" >FAQ</a></li>
<li><a href="https://alphacoders.com/site/privacy" rel="external nofollow" rel="external nofollow" >Privacy Policy</a></li>
<li><a href="https://alphacoders.com/site/tos" rel="external nofollow" rel="external nofollow" >Terms Of Service</a></li>
<li><a href="https://alphacoders.com/site/acceptable_use" rel="external nofollow" rel="external nofollow" >Acceptable Use</a></li>
<li><a href="https://alphacoders.com/site/etiquette" rel="external nofollow" rel="external nofollow" >Etiquette</a></li>
<li><a href="https://alphacoders.com/site/advertising" rel="external nofollow" rel="external nofollow" >Advertise With Us</a></li>
<li><a id="change_consent">Change Consent</a></li>
</ul>
可以发现class为"nav nav-pills"的只有一个,我们找到它:
Elements elements = doc.getElementsByTag("ul");
//System.out.println(elements);
Element tempElement = null;
for(Element element : elements) {
if (element.className().equals("nav nav-pills")) {
tempElement = element;
//System.out.println(element.className());
break;
}
}
循环遍历这个ul,输出其中每一个li里每一个a的href和rel属性:
Elements li = tempElement.getElementsByTag("li");
for(Element element : li) {
Elements element2 = element.getElementsByTag("a");
for(Element element3 : element2) {
String hrefString = element3.attr("href");
String relString = element3.attr("rel");
if(hrefString != "" && relString != "") {
System.out.println("href=" + hrefString + " rel="external nofollow" rel="external nofollow" " + "rel=" + relString);
}
}
}
最终结果:
href=https://alphacoders.com/site/about-us rel=nofollow
href=https://alphacoders.com/site/faq rel=nofollow
href=https://alphacoders.com/site/privacy rel=nofollow
href=https://alphacoders.com/site/tos rel=nofollow
href=https://alphacoders.com/site/acceptable_use rel=nofollow
href=https://alphacoders.com/site/etiquette rel=nofollow
href=https://alphacoders.com/site/advertising rel=nofollow
完整代码:
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.IOException;
import org.jsoup.Jsoup;
/**
* @ClassName: Jsoup_Test
* @description:
* @author: KI
* @Date: 2020年8月17日 下午8:15:14
*/
public class Jsoup_Test {
public static void main(String[] args) throws IOException {
// TODO 自动生成的方法存根
String html = "https://wall.alphacoders.com/featured.php?lang=Chinese";
Document doc = Jsoup.connect(html).get();
System.out.println(doc);
Elements elements = doc.getElementsByTag("ul");
//System.out.println(elements);
Element tempElement = null;
for(Element element : elements) {
if (element.className().equals("nav nav-pills")) {
tempElement = element;
//System.out.println(element.className());
break;
}
}
System.out.println(tempElement);
Elements li = tempElement.getElementsByTag("li");
for(Element element : li) {
Elements element2 = element.getElementsByTag("a");
for(Element element3 : element2) {
String hrefString = element3.attr("href");
String relString = element3.attr("rel");
if(hrefString != "" && relString != "") {
System.out.println("href=" + hrefString + " rel="external nofollow" rel="external nofollow" " + "rel=" + relString);
}
}
}
}
}
来源:https://blog.csdn.net/Cyril_KI/article/details/108073933
0
投稿
猜你喜欢
- 方法参数public String listFireEvent(@Valid FireSearch fireSearch, Ht
- 本文实例为大家分享了java实现随机数生成器的具体代码,供大家参考,具体内容如下自己编的随机数生成器,比较简陋,功能也单一,当作练手。App
- PostMapping接收json参数后返回404问题描述js中传递json数据给后端,后端可以正常接收参数,但返回404。js
- 当你使用synchronized关键字的时候,是通过互斥器来保障线程安全以及对共享资源的同步访问。线程间也经常需要更进一步的协调
- 1.以ApplocationContext上下文单例模式装配bean为例,深入探讨bean的生命周期:(1).生命周期图:(2).具体事例:
- 在spring运行时,动态的添加bean,dapeng框架在解析xml的字段时,使用到了动态注册,注册了一个实现了FactoryBean类!
- 一、使用ThreadLocal实现当前登录信息的存取在项目中我们增加一个员工有一些信息是需要我们自己填入的,有一些信息不需要我们自己填写,例
- 一、链表的介绍什么是链表链表是一种物理存储单元上非连续、非顺序的存储结构,数据元素的逻辑顺序是通过链表中的指针链接次序实现的。链表由一系列结
- 一个打砖块游戏算法,供大家参考,具体内容如下这里有一个打砖块游戏:小明面前有很多砖块,每个砖块上有一个字符,小明每击中一个砖块,会产生一个分
- 用java实现循环队列的方法:1、添加一个属性size用来记录眼下的元素个数。目的是当head=rear的时候。通过size=0还是size
- thymeleaf介绍简单说, Thymeleaf 是一个跟 Velocity、FreeMarker 类似的模板引擎,它可以完全替代 JSP
- 前言今天遇到一个问题,我们有个ip.dat二进制文件,通过里面内容可以解析ip所在的地域信息,本地单元测试都是OK的,部署到测试环境后,发现
- 本文实例为大家分享了java实现图片分割指定大小的具体代码,供大家参考,具体内容如下1.使用工具:ThumbnailsThumbnails
- 首先微信公众号开发网页授权登录使用环境:开发工具:eclipse;服务器:tomcat8,开发语言:JAVA。我写的网页授权登录时用开发者模
- Java 8支持动态语言,看到了很酷的Lambda表达式,对一直以静态类型语言自居的Java,让人看到了Java虚拟机可以支持动态语言的目标
- 本文研究的主要是Java回调函数与观察者模式的实现,具体介绍和实现代码如下。观察者模式(有时又被称为发布(publish )-订阅(Subs
- Unity IPostBuildPlayerScriptDLLsUnity IPostBuildPlayerScriptDLLs是Unity
- 简介常见的4种使用线程的方法:1实现 Runnable 接口;2实现 Callable 接口;3继承 Thread 类。4匿名内部类的写法。
- 数据库事务是被当作单个工作单元的操作序列。这些操作要么全部完成或全部不成功。事务管理是面向企业应用程序,以确保数据的完整性和一致性RDBMS
- 前言先简单介绍下我们的使用场景,线上5台Broker节点的kafka承接了所有binlog订阅的数据,用于Flink组件接收数据做数据中台的