美丽的汤刮图案？

Question

我试图刮包含以下HTML代码网站：

<div class="content-sidebar-wrap"><main class="content"><article 
class="post-773 post type-post status-publish format-standard has-post- 
thumbnail category-money entry" itemscope 
itemtype="http://schema.org/CreativeWork"> 

这包含数据我感兴趣......我一直在使用BeautifulSoup解析它尝试过，但以下回报：

<div class="content-sidebar-wrap"><main class="content"><article 
class="entry"> 
<h1 class="entry-title">Not found, error 404</h1><div class="entry-content 
"><p>"The page you are looking for no longer exists. Perhaps you can return 
back to the site's "<a href="http://www.totalsportek.com/">homepage</a> and 
see if you can find what you are looking for. Or, you can try finding it 
by using the search form below.</p><form 
action="http://www.totalsportek.com/" class="search-form" 
itemprop="potentialAction" itemscope="" 
itemtype="http://schema.org/SearchAction" method="get" role="search"> 

# I've made small modifications to make it readable 

美丽的汤元素不包含我想要的代码。我不太熟悉html，但我假设这会调用一些外部服务来返回数据..？我读过这个与Schema有关的东西。

无论如何我可以访问这些数据吗？

Answer 1

您在发出请求时需要指定User-Agent标头。打印文章标题和内容的工作示例：

import requests 
from bs4 import BeautifulSoup 

url = "http://www.totalsportek.com/money/barcelona-player-salaries/" 

response = requests.get(url, headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.78 Safari/537.36"}) 
soup = BeautifulSoup(response.content, "html.parser") 

article = soup.select_one(".content article.post.entry.status-publish") 
header = article.header.get_text(strip=True) 
content = article.select_one(".entry-content").get_text(strip=True) 

print(header) 
print(content)