drysrape安装Ubuntu服务器16.04

问题描述：

我在执行dryscrape和ubuntu 16.04服务器（干净安装在数字海洋上）时出现问题 - 目的是刮掉JS人口稠密的网站。drysrape安装Ubuntu服务器16.04

我下面dryscrape从here安装说明：

apt-get update 
apt-get install qt5-default libqt5webkit5-dev build-essential \ 
        python-lxml python-pip xvfb 

pip install dryscrape

，然后运行下面的Python脚本，我发现here以及在同一链路测试HTML页面。（它返回HTML或JS）

的Python

import dryscrape 
from bs4 import BeautifulSoup 
session = dryscrape.Session() 
my_url = 'http://www.example.com/scrape.php' 
session.visit(my_url) 
response = session.body() 
soup = BeautifulSoup(response) 
soup.find(id="intro-text")

HTML - scrape.php

<!DOCTYPE html> 
<html> 
<head> 
    <meta charset="utf-8"> 
    <title>Javascript scraping test</title> 
</head> 
<body> 
    <p id='intro-text'>No javascript support</p> 
    <script> 
    document.getElementById('intro-text').innerHTML = 'Yay! Supports javascript'; 
    </script> 
</body> 
</html>

当我这样做，我似乎无法得到预期的回报数据，相反，它只是错误。

我想知道是否有任何明显的我失踪？

注意：我浏览了很多安装指南/线程，似乎无法使其工作。我也尝试过使用硒，但似乎也无法使用硒。非常感谢。

输出

Traceback (most recent call last): 
    File "js.py", line 3, in <module> 
    session = dryscrape.Session() 
    File "/usr/local/lib/python2.7/dist-packages/dryscrape/session.py", line 22, in __init__ 
    self.driver = driver or DefaultDriver() 
    File "/usr/local/lib/python2.7/dist-packages/dryscrape/driver/webkit.py", line 30, in __init__ 
    super(Driver, self).__init__(**kw) 
    File "/usr/local/lib/python2.7/dist-packages/webkit_server.py", line 230, in __init__ 
    self.conn = connection or ServerConnection() 
    File "/usr/local/lib/python2.7/dist-packages/webkit_server.py", line 507, in __init__ 
    self._sock = (server or get_default_server()).connect() 
    File "/usr/local/lib/python2.7/dist-packages/webkit_server.py", line 450, in get_default_server 
    _default_server = Server() 
    File "/usr/local/lib/python2.7/dist-packages/webkit_server.py", line 424, in __init__ 
    raise NoX11Error("Could not connect to X server. " 
webkit_server.NoX11Error: Could not connect to X server. Try calling dryscrape.start_xvfb() before creating a session.

工作脚本

import dryscrape 
from bs4 import BeautifulSoup 

dryscrape.start_xvfb() 
session = dryscrape.Session() 
my_url = 'https://www.example.com/scrape.php' 
session.visit(my_url) 
response = session.body() 
soup = BeautifulSoup(response, "html.parser") 
print soup.find(id="intro-text").text

答

你有没有X服务器上运行。线索是

尝试调用dryscrape.start_xvfb（）创建会话之前

见http://dryscrape.readthedocs.io/en/latest/usage.html

if 'linux' in sys.platform: 
    # start xvfb in case no X is running. Make sure xvfb 
    # is installed, otherwise this won't work! 
    dryscrape.start_xvfb()

http://dryscrape.readthedocs.io/en/latest/installation.html

xvfb_（仅需如果没有其他X服务器可用）

所以，你可以再补充：

dryscrape.start_xvfb()

前：

session = dryscrape.Session()

感谢这个，我在更新/工作python脚本已经添加到我的答案的底部。我只需要添加额外的东西就是在'soup = BeautifulSoup（response，“html.parser”）内指定html解析器，因为我花了4个小时阅读并试图在昨天解决问题，所以我非常欣赏这个帮助。 – denski

drysrape安装Ubuntu服务器16.04

相关推荐