作者celestialgod (天)
看板R_Language
標題Re: [問題] parsing的中文顯示
時間Tue Jun 21 21:07:13 2016
1F：→ clansoda: 請問C大，這種網頁是不是完全爬不了06/21 16:20
3F：→ clansoda: 外面能爬的那個網頁，他大概有幾百個缺漏值，可能他後來06/21 16:21
4F：→ clansoda: 就不曾更新了，因此有些新開的他就沒加進去了06/21 16:21
好讀版：
http://pastebin.com/zNjkb0uh
csv結果：
http://www.mediafire.com/download/a3d7aat94yhz16n/familyMart_loc.csv
我用查郵遞區號方式去對inquery.aspx查詢
這裡最重要的是用curl的referer
我一開始找不到httr的方法去設定，所以先用了RCurl去做
後面補上用httr的GET的做法
library(httr)
library(xml2)
library(pipeR)
library(purrr)
library(stringi)
library(stringr)
library(RCurl)
library(jsonlite)
url_postCodes <- "
http://www.easytravel.com.tw/postid_search.asp"
all_postCodes <- url_postCodes %>>% GET %>>% content(encoding = "big5") %>>%
xml_find_all("//td/font") %>>% xml_text %>>%
stri_conv(from = "UTF-8", to = "Big5") %>>% str_extract("\\d{3}") %>>%
`[`(!is.na(.)) %>>% as.integer
# method with RCurl::getURL
familyMartLoc <- lapply(all_postCodes, function(postCode){
url_toPrint <- str_c("
http://api.map.com.tw/net/GraphicsXY.aspx?",
"search_class=Zip&Zip=%i&fun=getCityByZipReturn")
city_area <- sprintf(url_toPrint, postCode) %>>%
getURL(referer = "
http://www.family.com.tw/marketing/inquiry.aspx") %>>%
str_replace_all("getCityByZipReturn\\(|\\)", "") %>>% fromJSON
url_toPrint <- str_c("
http://api.map.com.tw/net/familyShop.aspx?",
"searchType=ShopList&type=&city=%s&area=%s&road=&",
"fun=showStoreList&key=6F30E8BF706D653965BDE302661D1241F8BE9EBC")
sprintf(url_toPrint, city_area$COUNTY, city_area$TOWN) %>>%
getURL(referer = "
http://www.family.com.tw/marketing/inquiry.aspx") %>>%
str_replace_all("showStoreList\\(|\\)", "") %>>% fromJSON
}) %>>% do.call(what = rbind)
# method with httr::GET
familyMartLoc2 <- lapply(all_postCodes, function(postCode){
url_toPrint <- str_c("
http://api.map.com.tw/net/GraphicsXY.aspx?",
"search_class=Zip&Zip=%i&fun=getCityByZipReturn")
city_area <- sprintf(url_toPrint, postCode) %>>%
GET(config(referer = "
http://www.family.com.tw/marketing/inquiry.aspx")) %>>%
content("text") %>>% str_replace_all("getCityByZipReturn\\(|\\)", "") %>>%
fromJSON
url_toPrint <- str_c("
http://api.map.com.tw/net/familyShop.aspx?",
"searchType=ShopList&type=&city=%s&area=%s&road=&",
"fun=showStoreList&key=6F30E8BF706D653965BDE302661D1241F8BE9EBC")
sprintf(url_toPrint, city_area$COUNTY, city_area$TOWN) %>>%
GET(config(referer = "
http://www.family.com.tw/marketing/inquiry.aspx")) %>>%
content("text") %>>% str_replace_all("showStoreList\\(|\\)", "") %>>%
fromJSON
}) %>>% do.call(what = rbind)
partial results:
NAME TEL POSTel px py
1 全家新德店 02-23214351 02-77463163 121.5316 25.04447
2 全家台醫店 02-23278841 02-23279603 121.5185 25.04079
3 全家南機場店 02-23039420 02-77095865 121.5058 25.02840
4 全家新惠安店 02-23036887 02-77300717 121.5065 25.02953
5 全家新龍口店 02-23047184 02-77300715 121.5104 25.02723
6 全家新廈門店 02-23681443 02-77397215 121.5176 25.02532
--
R資料整理套件系列文：
magrittr #1LhSWhpH (R_Language) http://tinyurl.com/j3ql84c
data.table #1LhW7Tvj (R_Language) http://tinyurl.com/hr77hrn
dplyr(上) #1LhpJCfB (R_Language) http://tinyurl.com/jtg4hau
dplyr(下) #1Lhw8b-s (R_Language)
tidyr #1Liqls1R (R_Language) http://tinyurl.com/jq3o2g3
--
※ 發信站: 批踢踢實業坊(ptt.cc), 來自: 211.76.63.212
※ 文章網址: https://webptt.com/m.aspx?n=bbs/R_Language/M.1466514438.A.696.html
※ 編輯: celestialgod (211.76.63.212), 06/21/2016 21:19:40
5F：推 clansoda: C大可以請您說明一下原理嗎 我只看得懂取得郵遞區號那邊 06/22 15:51
6F：→ clansoda: 後面的這串API網址是哪裡來的呀 點下去是奇怪的網頁 06/22 15:52
7F：→ clansoda: 看不太懂中間的來龍去脈 06/22 15:52
8F：→ celestialgod: 按F12去看查詢時的網頁 06/22 17:36
9F：→ celestialgod: network那個標籤可以看到你查詢到你網頁送出的reque 06/22 17:47
10F：→ celestialgod: st 06/22 17:47
11F：推 clansoda: 我今天試著去搞懂他 06/23 09:10
12F：推 clansoda: str_c跟paste0有不同嗎 我寫起來的效果好像一樣 06/23 09:59
13F：→ celestialgod: 效果一樣 06/23 10:00