Scrapy exceptions tutorial

Question

Sto seguendo la documentazione del tutorial Scrapy al http://media.readthedocs.org/pdf/scrapy/0.14/scrapy.pdf e ho verificato che items.py e dmoz_spider.py siano stati digitati (non tagliati & incollati correttamente).

Il primo "hmmm ..." parte per me era questa istruzione:

Questo è il codice per il nostro primo Spider; salvarlo in un file di nome dmoz_spider.py nella directory DMOZ/ragni

Sto utilizzando l'ultima versione di Ubuntu e non c'era una cartella dmoz creato, così ho messo questo codice in ~/esercitazione/tutorial/ragni. (Era questo il mio primo errore?)

Quindi, ecco il mio script dmoz_spider.py:

from scrapy.spider import BaseSpider 

class DmozSpider(BaseSpider): 
    name = "dmoz" 
    allowed_domains = ["dmoz.org"] 
    start_urls = [ 
    "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/", 
    "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/" 
    ] 

def parse(self, response): 
    filename = response.url.split("/")[-2] 
    open(filename, 'wb').write(response.body) 

Nel mio terminale digito

scrapy crawl dmoz 

e ottengo questo:

2012-10-08 13:20:22-0700 [scrapy] INFO: Scrapy 0.12.0.2546 started (bot: tutorial) 
2012-10-08 13:20:22-0700 [scrapy] DEBUG: Enabled extensions: TelnetConsole, SpiderContext, WebService, CoreStats, MemoryUsage, CloseSpider 
2012-10-08 13:20:22-0700 [scrapy] DEBUG: Enabled scheduler middlewares: DuplicatesFilterMiddleware 
2012-10-08 13:20:22-0700 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpCompressionMiddleware, DownloaderStats 
2012-10-08 13:20:22-0700 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware 
2012-10-08 13:20:22-0700 [scrapy] DEBUG: Enabled item pipelines: 
2012-10-08 13:20:22-0700 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023 
2012-10-08 13:20:22-0700 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080 
2012-10-08 13:20:22-0700 [dmoz] INFO: Spider opened 
2012-10-08 13:20:22-0700 [dmoz] DEBUG: Crawled (200) <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/> (referer: None) 
2012-10-08 13:20:22-0700 [dmoz] ERROR: Spider error processing <http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/> (referer: <None>) 
Traceback (most recent call last): 
    File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1178, in mainLoop 
    self.runUntilCurrent() 
    File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 800, in runUntilCurrent 
    call.func(*call.args, **call.kw) 
    File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 362, in callback 
    self._startRunCallbacks(result) 
    File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 458, in _startRunCallbacks 
    self._runCallbacks() 
--- <exception caught here> --- 
    File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 545, in _runCallbacks 
    current.result = callback(current.result, *args, **kw) 
    File "/usr/lib/python2.7/dist-packages/scrapy/spider.py", line 62, in parse 
    raise NotImplementedError 
exceptions.NotImplementedError: 

2012-10-08 13:20:22-0700 [dmoz] DEBUG: Crawled (200) <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> (referer: None) 
2012-10-08 13:20:22-0700 [dmoz] ERROR: Spider error processing <http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> (referer: <None>) 
Traceback (most recent call last): 
    File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1178, in mainLoop 
    self.runUntilCurrent() 
    File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 800, in runUntilCurrent 
    call.func(*call.args, **call.kw) 
    File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 362, in callback 
    self._startRunCallbacks(result) 
    File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 458, in _startRunCallbacks 
    self._runCallbacks() 
--- <exception caught here> --- 
    File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 545, in _runCallbacks 
    current.result = callback(current.result, *args, **kw) 
    File "/usr/lib/python2.7/dist-packages/scrapy/spider.py", line 62, in parse 
    raise NotImplementedError 
exceptions.NotImplementedError: 

2012-10-08 13:20:22-0700 [dmoz] INFO: Closing spider (finished) 
2012-10-08 13:20:22-0700 [dmoz] INFO: Spider closed (finished) 

Nella mia ricerca, ho visto che qualcun altro aveva detto che twisted probabilmente non era installato ... ma non sarebbe installato se ho usato il Installer di pacchetti Ubuntu per Scrapy?

Grazie in anticipo!

Answer 1

Il metodo di analisi in BaseSpider viene richiamato al posto di uno perché non è stato sovrascritto correttamente il metodo di analisi. Il tuo rientro è sbagliato, quindi l'analisi viene dichiarata come funzione al di fuori della classe DmozSpider. Benvenuti in Python :)

Non ha nulla a che vedere con il twistato, vedo che è contorto nei traceback, quindi è chiaramente installato.