powershell网络蜘蛛解决乱码问题
2019/7/10 21:30:02
本文主要是介绍powershell网络蜘蛛解决乱码问题,对大家解决编程问题具有一定的参考价值,需要的程序猿们随着小编来一起学习吧!
抓取(爬取)网上信息的脚本程序,俗称网络蜘蛛。
powershell中自带了这样的两个命令,【Invoke-WebRequest】和【Invoke-RestMethod】,但这两个命令有时候会乱码。
现在转帖分享, 某个【歪果仁】写的脚本。来源于 墙外出处: https://gist.github.com/angel-vladov/9482676
核心代码
function Read-HtmlPage { param ([Parameter(Mandatory=$true, Position=0, ValueFromPipeline=$true)][String] $Uri) # Invoke-WebRequest and Invoke-RestMethod can't work properly with UTF-8 Response so we need to do things this way. [Net.HttpWebRequest]$WebRequest = [Net.WebRequest]::Create($Uri) [Net.HttpWebResponse]$WebResponse = $WebRequest.GetResponse() $Reader = New-Object IO.StreamReader($WebResponse.GetResponseStream()) $Response = $Reader.ReadToEnd() $Reader.Close() # Create the document class [mshtml.HTMLDocumentClass] $Doc = New-Object -com "HTMLFILE" $Doc.IHTMLDocument2_write($Response) # Returns a HTMLDocumentClass instance just like Invoke-WebRequest ParsedHtml $Doc #powershell 传教士 转帖并修改的文章 2016-01-01, 允许再次转载,但必须保留名字和出处,否则追究法律责任 }
原文函数
function Read-HtmlPage { param ([Parameter(Mandatory=$true, Position=0, ValueFromPipeline=$true)][String] $Uri) # Invoke-WebRequest and Invoke-RestMethod can't work properly with UTF-8 Response so we need to do things this way. [Net.HttpWebRequest]$WebRequest = [Net.WebRequest]::Create($Uri) [Net.HttpWebResponse]$WebResponse = $WebRequest.GetResponse() $Reader = New-Object IO.StreamReader($WebResponse.GetResponseStream()) $Response = $Reader.ReadToEnd() $Reader.Close() # Create the document class [mshtml.HTMLDocumentClass] $Doc = New-Object -com "HTMLFILE" $Doc.IHTMLDocument2_write($Response) # Returns a HTMLDocumentClass instance just like Invoke-WebRequest ParsedHtml $Doc }
PowerShell function you can use for reading UTF8 encoded HTML pages content. The built in Invoke-WebRequest and Invoke-RestMethod fail miserably.
这篇关于powershell网络蜘蛛解决乱码问题的文章就介绍到这儿,希望我们推荐的文章对大家有所帮助,也希望大家多多支持为之网!
- 2022-09-14SharePoint 2019 用 PowerShell将启用发布功能站点网站集另存为模板
- 2022-08-30PowerShell教程 - 程序性能和BUG分析工具
- 2022-08-30PowerShell教程 - 模块管理(Modules Management)
- 2022-08-29PowerShell教程 - Web requests(Web请求)
- 2022-08-26PowerShell教程 - 日期时间管理(Date & Time Management)
- 2022-08-25PowerShell教程 - 磁盘与硬件管理(Disk & Hardware Management)
- 2022-08-25PowerShell教程 - 系统事件管理(System Event Management)
- 2022-08-25PowerShell教程 - 文件系统管理(File System Management)
- 2022-08-24PowerShell教程 - 网络管理(Network Management)
- 2022-08-24PowerShell