Java爬虫实现Jsoup利用dom方法遍历Document对象
先给出网页地址:
https://wall.alphacoders.com/featured.php?lang=Chinese
主要步骤:
利用Jsoup的connect方法获取Document对象
String html = "https://wall.alphacoders.com/featured.php?lang=Chinese"; Document doc = Jsoup.connect(html).get();
内容过长,就不再显示。
我们以这部分为例:
<ul class="nav nav-pills"> <li><a href="https://alphacoders.com/site/about-us" rel="external nofollow" rel="external nofollow" >About Us</a></li> <li><a href="https://alphacoders.com/site/faq" rel="external nofollow" rel="external nofollow" >FAQ</a></li> <li><a href="https://alphacoders.com/site/privacy" rel="external nofollow" rel="external nofollow" >Privacy Policy</a></li> <li><a href="https://alphacoders.com/site/tos" rel="external nofollow" rel="external nofollow" >Terms Of Service</a></li> <li><a href="https://alphacoders.com/site/acceptable_use" rel="external nofollow" rel="external nofollow" >Acceptable Use</a></li> <li><a href="https://alphacoders.com/site/etiquette" rel="external nofollow" rel="external nofollow" >Etiquette</a></li> <li><a href="https://alphacoders.com/site/advertising" rel="external nofollow" rel="external nofollow" >Advertise With Us</a></li> <li><a id="change_consent">Change Consent</a></li> </ul>
我们先找到所有的ul:
Elements elements = doc.getElementsByTag("ul");
输出如下:
<ul class="nav navbar-nav center"> <li> <a title="Submit Wallpapers" href="https://alphacoders.com/site/submit-wallpaper" rel="external nofollow" ><i class="el el-circle-arrow-up"></i> 提交</a> </li> <li> <a href="https://alphacoders.com/contest" rel="external nofollow" ><i class="el el-gift"></i> 精美奖品</a> </li> </ul> <ul class="nav navbar-nav navbar-right center"> <li> <a href="language.php?lang=Chinese" rel="external nofollow" > <img src="https://static.alphacoders.com/wa/Chinese-flag.png" alt="Chinese-flag"> 中文 </a> </li> <li> <a href="https://alphacoders.com/users/login" rel="external nofollow" ><i class="el el-user"></i> 登录</a> </li> <li> <a href="https://alphacoders.com/users/register" rel="external nofollow" ><i class="el el-edit"></i> 注册</a> </li> </ul> <ul class="pagination"> <li class="active"><a id="prev_page" href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >< 上一页</a></li> <li class="active"><a>1</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >2</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=3" rel="external nofollow" rel="external nofollow" rel="external nofollow" >3</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=4" rel="external nofollow" rel="external nofollow" rel="external nofollow" >4</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=5" rel="external nofollow" rel="external nofollow" rel="external nofollow" >5</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=6" rel="external nofollow" rel="external nofollow" rel="external nofollow" >6</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=7" rel="external nofollow" rel="external nofollow" rel="external nofollow" >7</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=8" rel="external nofollow" >8</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=9" rel="external nofollow" >9</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=10" rel="external nofollow" >10</a></li> <li><a>...</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=319" rel="external nofollow" rel="external nofollow" rel="external nofollow" >319</a></li> <li><a id="next_page" href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >下一页 ></a></li> </ul> <ul class="pagination"> <li class="active"><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >< 上一页</a></li> <li class="active"><a>1</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >2</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=3" rel="external nofollow" rel="external nofollow" rel="external nofollow" >3</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=4" rel="external nofollow" rel="external nofollow" rel="external nofollow" >4</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=5" rel="external nofollow" rel="external nofollow" rel="external nofollow" >5</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=6" rel="external nofollow" rel="external nofollow" rel="external nofollow" >6</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=7" rel="external nofollow" rel="external nofollow" rel="external nofollow" >7</a></li> <li><a>...</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=319" rel="external nofollow" rel="external nofollow" rel="external nofollow" >319</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >下一页 ></a></li> </ul> <ul class="pagination"> <li class="active"><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" ><< </a></li> <li class="active"><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >< 上一页</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >下一页 ></a></li> <li><a title="末页 (319)" href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=319" rel="external nofollow" rel="external nofollow" rel="external nofollow" > >></a></li> </ul> <ul class="pagination"> <li class="active"><a href="#" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >1</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=2" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" >2</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=3" rel="external nofollow" rel="external nofollow" rel="external nofollow" >3</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=4" rel="external nofollow" rel="external nofollow" rel="external nofollow" >4</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=5" rel="external nofollow" rel="external nofollow" rel="external nofollow" >5</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=6" rel="external nofollow" rel="external nofollow" rel="external nofollow" >6</a></li> <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&page=7" rel="external nofollow" rel="external nofollow" rel="external nofollow" >7</a></li> </ul> <ul class="nav nav-pills"> <li><a href="https://alphacoders.com/site/about-us" rel="external nofollow" rel="external nofollow" >About Us</a></li> <li><a href="https://alphacoders.com/site/faq" rel="external nofollow" rel="external nofollow" >FAQ</a></li> <li><a href="https://alphacoders.com/site/privacy" rel="external nofollow" rel="external nofollow" >Privacy Policy</a></li> <li><a href="https://alphacoders.com/site/tos" rel="external nofollow" rel="external nofollow" >Terms Of Service</a></li> <li><a href="https://alphacoders.com/site/acceptable_use" rel="external nofollow" rel="external nofollow" >Acceptable Use</a></li> <li><a href="https://alphacoders.com/site/etiquette" rel="external nofollow" rel="external nofollow" >Etiquette</a></li> <li><a href="https://alphacoders.com/site/advertising" rel="external nofollow" rel="external nofollow" >Advertise With Us</a></li> <li><a id="change_consent">Change Consent</a></li> </ul>
可以发现class为"nav nav-pills"的只有一个,我们找到它:
Elements elements = doc.getElementsByTag("ul"); //System.out.println(elements); Element tempElement = null; for(Element element : elements) { if (element.className().equals("nav nav-pills")) { tempElement = element; //System.out.println(element.className()); break; } }
循环遍历这个ul,输出其中每一个li里每一个a的href和rel属性:
Elements li = tempElement.getElementsByTag("li"); for(Element element : li) { Elements element2 = element.getElementsByTag("a"); for(Element element3 : element2) { String hrefString = element3.attr("href"); String relString = element3.attr("rel"); if(hrefString != "" && relString != "") { System.out.println("href=" + hrefString + " rel="external nofollow" rel="external nofollow" " + "rel=" + relString); } } }
最终结果:
href=https://alphacoders.com/site/about-us rel=nofollow
href=https://alphacoders.com/site/faq rel=nofollow
href=https://alphacoders.com/site/privacy rel=nofollow
href=https://alphacoders.com/site/tos rel=nofollow
href=https://alphacoders.com/site/acceptable_use rel=nofollow
href=https://alphacoders.com/site/etiquette rel=nofollow
href=https://alphacoders.com/site/advertising rel=nofollow
完整代码:
import org.jsoup.nodes.Document; import org.jsoup.nodes.Element; import org.jsoup.select.Elements; import java.io.IOException; import org.jsoup.Jsoup; /** * @ClassName: Jsoup_Test * @description: * @author: KI * @Date: 2020年8月17日 下午8:15:14 */ public class Jsoup_Test { public static void main(String[] args) throws IOException { // TODO 自动生成的方法存根 String html = "https://wall.alphacoders.com/featured.php?lang=Chinese"; Document doc = Jsoup.connect(html).get(); System.out.println(doc); Elements elements = doc.getElementsByTag("ul"); //System.out.println(elements); Element tempElement = null; for(Element element : elements) { if (element.className().equals("nav nav-pills")) { tempElement = element; //System.out.println(element.className()); break; } } System.out.println(tempElement); Elements li = tempElement.getElementsByTag("li"); for(Element element : li) { Elements element2 = element.getElementsByTag("a"); for(Element element3 : element2) { String hrefString = element3.attr("href"); String relString = element3.attr("rel"); if(hrefString != "" && relString != "") { System.out.println("href=" + hrefString + " rel="external nofollow" rel="external nofollow" " + "rel=" + relString); } } } } }
到此这篇关于Java爬虫实现Jsoup利用dom方法遍历Document对象的文章就介绍到这了,更多相关Java Jsoup遍历Document对象内容请搜索猪先飞以前的文章或继续浏览下面的相关文章希望大家以后多多支持猪先飞!
原文出处:https://blog.csdn.net/Cyril_KI/article/details/108073933
相关文章
- 这篇文章主要介绍了如何利用java语言实现经典《复杂迷宫》游戏,文中采用了swing技术进行了界面化处理,感兴趣的小伙伴可以动手试一试...2022-02-01
java 运行报错has been compiled by a more recent version of the Java Runtime
java 运行报错has been compiled by a more recent version of the Java Runtime (class file version 54.0)...2021-04-01- 这篇文章主要介绍了在java中获取List集合中最大的日期时间操作,具有很好的参考价值,希望对大家有所帮助。一起跟随小编过来看看吧...2020-08-15
- 这篇文章主要介绍了教你怎么用Java获取国家法定节假日,文中有非常详细的代码示例,对正在学习java的小伙伴们有非常好的帮助,需要的朋友可以参考下...2021-04-23
- 这篇文章主要介绍了Java如何发起http请求的实现,文中通过示例代码介绍的非常详细,对大家的学习或者工作具有一定的参考学习价值,需要的朋友们下面随着小编来一起学习学习吧...2021-03-31
- 说起C#和Java这两门语言(语法,数据类型 等),个人以为,大概有90%以上的相似,甚至可以认为几乎一样。但是在工作中,我也发现了一些细微的差别...2020-06-25
- 下面小编就为大家带来一篇js遍历json的key和value的实例。小编觉得挺不错的,现在就分享给大家,也给大家做个参考。一起跟随小编过来看看吧...2017-01-26
- 这篇文章主要介绍了解决Java处理HTTP请求超时的问题,具有很好的参考价值,希望对大家有所帮助。一起跟随小编过来看看吧...2021-03-29
- 这篇文章主要介绍了java 判断两个时间段是否重叠的案例,具有很好的参考价值,希望对大家有所帮助。一起跟随小编过来看看吧...2020-08-15
- 这篇文章主要介绍了超简洁java实现双色球若干注随机号码生成(实例代码),本文通过实例代码给大家介绍的非常详细,对大家的学习或工作具有一定的参考借鉴价值,需要的朋友可以参考下...2021-04-02
JavaScript中的数组遍历forEach()与map()方法以及兼容写法介绍
下面小编就为大家带来一篇JavaScript中的数组遍历forEach()与map()方法以及兼容写法介绍。小编觉得挺不错的,现在分享给大家,也给大家做个参考。一起跟随小编过来看看吧...2016-05-20- 这篇文章主要介绍了Java生成随机姓名、性别和年龄的实现示例,文中通过示例代码介绍的非常详细,对大家的学习或者工作具有一定的参考学习价值,需要的朋友们下面随着小编来一起学习学习吧...2020-10-01
java 画pdf用itext调整表格宽度、自定义各个列宽的方法
这篇文章主要介绍了java 画pdf用itext调整表格宽度、自定义各个列宽的方法,具有很好的参考价值,希望对大家有所帮助。一起跟随小编过来看看吧...2021-01-31- 这篇文章主要介绍了java正则表达式判断前端参数修改表中另一个字段的值,需要的朋友可以参考下...2021-05-07
Java使用ScriptEngine动态执行代码(附Java几种动态执行代码比较)
这篇文章主要介绍了Java使用ScriptEngine动态执行代码,并且分享Java几种动态执行代码比较,需要的朋友可以参考下...2021-04-15- 这篇文章主要介绍了Java开发实现人机猜拳游戏,文中示例代码介绍的非常详细,具有一定的参考价值,感兴趣的小伙伴们可以参考一下...2020-08-03
Java 8 Stream 的终极技巧——Collectors 功能与操作方法详解
这篇文章主要介绍了Java 8 Stream Collectors 功能与操作方法,结合实例形式详细分析了Java 8 Stream Collectors 功能、操作方法及相关注意事项,需要的朋友可以参考下...2020-05-20- 这篇文章主要介绍了Java List集合返回值去掉中括号('[ ]')的操作,具有很好的参考价值,希望对大家有所帮助。一起跟随小编过来看看吧...2020-08-29
jQuery遍历DOM的父级元素、子级元素和同级元素的方法总结
借助jQuery我们可以轻松地堆DOM元素进行向上、向下遍历以及同级的遍历,本文我们即来整理jQuery遍历DOM的父级元素、子级元素和同级元素的方法总结:...2016-07-25Java中lombok的@Builder注解的解析与简单使用详解
这篇文章主要介绍了Java中lombok的@Builder注解的解析与简单使用,本文给大家介绍的非常详细,对大家的学习或工作具有一定的参考借鉴价值,需要的朋友可以参考下...2021-01-06