Come posso ottenere tutto il testo normale da un sito Web con Scrapy?

Question

Mi piacerebbe avere tutto il testo visibile da un sito Web, dopo il rendering dell'HTML. Sto lavorando in Python con Scrapy framework. Con xpath('//body//text()') Sono in grado di ottenerlo, ma con i tag HTML, e voglio solo il testo. Qualche soluzione per questo? Grazie !

Answer 1

L'opzione più semplice sarebbe quella di extract//body//text() e join tutto trovato:

''.join(sel.select("//body//text()").extract()).strip() 

dove sel è un'istanza Selector.

Un'altra opzione è quella di utilizzare nltk 's clean_html():

>>> import nltk 
>>> html = """ 
... <div class="post-text" itemprop="description"> 
... 
...   <p>I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework. 
... With <code>xpath('//body//text()')</code> I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !</p> 
... 
...  </div>""" 
>>> nltk.clean_html(html) 
"I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework.\nWith xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks !" 

Un'altra opzione è quella di utilizzare BeautifulSoup' s get_text():

get_text()

If you only want the text part of a document or tag, you can use the get_text() method. It returns all the text in a document or beneath a tag, as a single Unicode string.

>>> from bs4 import BeautifulSoup 
>>> soup = BeautifulSoup(html) 
>>> print soup.get_text().strip() 
I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework. 
With xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks ! 

Un'altra opzione è quella di utilizzare lxml.html s' text_content() :

.text_content()

Returns the text content of the element, including the text content of its children, with no markup.

>>> import lxml.html 
>>> tree = lxml.html.fromstring(html) 
>>> print tree.text_content().strip() 
I would like to have all the text visible from a website, after the HTML is rendered. I'm working in Python with Scrapy framework. 
With xpath('//body//text()') I'm able to get it, but with the HTML tags, and I only want the text. Any solution for this? Thanks ! 

Answer 2

Hai provato?

xpath('//body//text()').re('(\w+)') 

O

xpath('//body//text()').extract()