Python的phantomjs加载网页不正确

Question

我有一个问题，从这个链接

http://www.bursamalaysia.com/market/listed-companies/company-announcements/#/?category=FA&sub_category=FA1&alphabetical=All&company=5250

提取带给我从这个链接，而不是它本身的主要页面的数据。 http://www.bursamalaysia.com/market/listed-companies/company-announcements/#/?category=all

任何想法为什么发生这种情况？ 我使用PhantomJS硒和美丽的汤来帮助我。

# The standard library modules 
import os 
import sys 
import re 
import sqlite3 
import locale 
# The wget module 
import wget 
import time 
import calendar 
from datetime import datetime 
# The BeautifulSoup module 
from bs4 import BeautifulSoup 

# The selenium module 
from selenium import webdriver 
from selenium.webdriver.common.keys import Keys 
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC 
from selenium.webdriver.common.by import By 


def getURLS(url): 
    driver = webdriver.PhantomJS(service_args=['--ignore-ssl-errors=true']) 
    driver.get(url) # load the web page 
    src = driver.page_source 
    #Get text and split it 
    soup = BeautifulSoup(src, 'html5lib') 

    print soup 

link ='http://www.bursamalaysia.com/market/listed-companies/company-announcements/#/?category=FA&sub_category=FA1&alphabetical=All&company=5250' 
getURLS(link) 

亚历克斯Lucaci解决方案

def getURLS(url): 
    driver = webdriver.PhantomJS(service_args=['--ignore-ssl-errors=true']) 
    driver.get(url) # load the web page 
    src = driver.page_source 
    category_select = Select(driver.find_element_by_xpath('//*[@id="bm_announcement_types"]')) 
    category_select.select_by_visible_text("Financial Results") 
    category_select2 = Select(driver.find_element_by_xpath('//*[@id="bm_sub_announcement_types"]')) 
    category_select2.select_by_visible_text("Financial Results") 
    category_select3 = Select(driver.find_element_by_xpath('//*[@id="bm_company_list"]')) 
    category_select3.select_by_visible_text("7-ELEVEN MALAYSIA HOLDINGS BERHAD (5250)") 
    driver.find_element_by_xpath('//*[@id="bm_company_announcements_search_form"]/input[1]').click() 
    src = driver.page_source 
    soup = BeautifulSoup(src, 'html5lib') 
    link="http://www.bursamalaysia.com/market/listed-companies/company-announcements/#/?category=all" 
    getURLS(link) 

Answer 1

当你保存页面没有完全加载您提交后的源，以便尝试抓取的网页源之前等待一对夫妇秒：

def getURLS(url): 
driver = webdriver.PhantomJS(service_args=['--ignore-ssl-errors=true']) 
driver.get(url) # load the web page 
time.sleep(5)# waiting for 5 seconds before fetching the source 
src = driver.page_source 
#Get text and split it 
soup = BeautifulSoup(src, 'html5lib') 

print soup 

要执行下拉菜单中选择你导入的Select类如下：from selenium.webdriver.support.ui import Select，然后你必须选择这样的下拉元素：

category_select = Select(driver.find_element_by_xpath('//*[@id="bm_announcement_types"]')) 
category_select.select_by_visible_text('Financial Results') 

在我的例子我已经做到了为 - 分类 - 下拉列表，按照每个类别的具体步骤。 请注意，通过xpath选择下拉列表是最好的方法，您可以通过使用谷歌浏览器 - >点击元素 - >检查 - >右键点击<select>出现的右侧菜单 - >复制 - >复制Xpath

当您选择所有元素时，您必须单击提交并等待几秒钟才能加载，之后您将获取源代码。

让我知道如果我的答案帮助你。