作者mikemlbb (M)
看板R_Language
標題[問題] 抓取PTT網頁，請問此程式碼的錯誤在哪？
時間Tue Oct 18 15:23:22 2016
[問題類型]:
程式諮詢(我想用R 做某件事情，但是我不知道要怎麼用R 寫出來)
[軟體熟悉度]:
使用者(已經有用R 做過不少作品)
[問題敘述]:
我照著書本輸入以下程式碼，
想嘗試抓取笨版中之文章文字內容，
但程式碼執行完後卻出現：
Error in regexpr("www", line) :
argument "line" is missing, with no default
了解regexpr語法的用法後，
發現此程式的"line"字詞與該語法之用法不同，
然而若是在此範例中，
要怎麼修改才能抓取到笨版當中的文章呢？
謝謝大家解惑
[程式範例]:
install.packages("XML")
install.packages("RCurl")
library(XML)
library(RCurl)
data <- list()
for(i in 1058:1118){
tmp <- paste(i, '.html', sep = '')
url <- paste('
https://webptt.com/m.aspx?n=bbs/StupidClown/index',tmp,sep = '')
get_url <- getURL(url,ssl.verifypeer = FALSE)
html <- htmlParse(get_url)
url.list <- xpathSApply(html,"//div[@class='title']/a[@href]",xmlAttrs)
data <- rbind(data, paste('
https://webptt.com/m.aspx?n=',url.list,sep = ''))
}
data <- unlist(data)
getdoc <- function(line){
start <- regexpr('www', line)[1]
end <- regexpr('html', line)[1]
if(start != -1 & end != -1){
url <- substr(line, start, end+3)
html <- htmlParse(getURL(url,ssl.verifypeer = FALSE),encoding = 'UTF-8')
doc <- xpathSApply(html, "//div[@id='main-container']",xmlValue)
name <- strsplit(url,'/')[[1]][4]
write(doc,gsub('html','txt',name))
}
}
getdoc()
sapply(data, getdoc)
setwd("C://Documents and Settings//12345//桌面//R_textmining")
write.table(getdoc,file = "getdoc.txt",row.names = F,quote = F)
[環境敘述]:
R version 3.3.1 (2016-06-21)
Platform: i386-w64-mingw32/i386 (32-bit)
Running under: Windows XP (build 2600) Service Pack 3
[關鍵字]:
regexpr、xpathSApply、PTT爬蟲
--
※ 發信站: 批踢踢實業坊(ptt.cc), 來自: 118.163.143.251
※ 文章網址: https://webptt.com/m.aspx?n=bbs/R_Language/M.1476775409.A.321.html
※ 編輯: mikemlbb (118.163.143.251), 10/18/2016 15:29:00
1F：推 clansoda: Hi，I am trying to solve your problem. 10/19 09:41
2F：→ clansoda: Would you tell me what your expected output is 10/19 09:41
3F：→ clansoda: The "data" dataframe contains 1220 URL characters 10/19 09:42
4F：→ mikemlbb: I'm trying to crawl the content of StupidClown site 10/21 02:23
5F：→ mikemlbb: XIncluding article title and content by no.1058 to11 10/21 02:25
6F：→ mikemlbb: But the code seem to be wrong.When I run "getdoc()" 10/21 02:27
7F：→ mikemlbb: The error will emerge then say "line" is not defined 10/21 02:29