Errore durante il trasferimento di file da ec2 a s3 utilizzando boto

Question

Sto seguendo questa procedura link per caricare il mio mongodump su s3.

script bash

#!/bin/sh 

MONGODB_SHELL='/usr/bin/mongo' 

DUMP_UTILITY='/usr/bin/mongodump' 
DB_NAME='amicus' 

date_now=`date +%Y_%m_%d_%H_%M_%S` 
dir_name='db_backup_'${date_now} 
file_name='db_backup_'${date_now}'.bz2' 

log() { 
    echo $1 
} 

do_cleanup(){ 
    rm -rf db_backup_2010* 
    log 'cleaning up....' 
} 

do_backup(){ 
    log 'snapshotting the db and creating archive' && \ 
    ${MONGODB_SHELL} admin fsync_lock.js && \ 
    ${DUMP_UTILITY} -d ${DB_NAME} -o ${dir_name} && tar -jcf $file_name ${dir_name} 
    ${MONGODB_SHELL} admin unlock.js && \ 
    log 'data backd up and created snapshot' 
} 

save_in_s3(){ 
    log 'saving the backup archive in amazon S3' && \ 
    python aws_s3.py set ${file_name} && \ 
    log 'data backup saved in amazon s3' 
} 

do_backup && save_in_s3 && do_cleanup 

aws_s3.py

ACCESS_KEY='' 
SECRET='' 
BUCKET_NAME='s3:///s3.amazonaws.com/database-backup' #note that you need to create this bucket first 

from boto.s3.connection import S3Connection 
from boto.s3.key import Key 

def save_file_in_s3(filename): 
    conn = S3Connection(ACCESS_KEY, SECRET) 
    bucket = conn.get_bucket(BUCKET_NAME) 
    k = Key(bucket) 
    k.key = filename 
    k.set_contents_from_filename(filename) 

def get_file_from_s3(filename): 
    conn = S3Connection(ACCESS_KEY, SECRET) 
    bucket = conn.get_bucket(BUCKET_NAME) 
    k = Key(bucket) 
    k.key = filename 
    k.get_contents_to_filename(filename) 

def list_backup_in_s3(): 
    conn = S3Connection(ACCESS_KEY, SECRET) 
    bucket = conn.get_bucket(BUCKET_NAME) 
    for i, key in enumerate(bucket.get_all_keys()): 
     print "[%s] %s" % (i, key.name) 

def delete_all_backups(): 
    #FIXME: validate filename exists 
    conn = S3Connection(ACCESS_KEY, SECRET) 
    bucket = conn.get_bucket(BUCKET_NAME) 
    for i, key in enumerate(bucket.get_all_keys()): 
     print "deleting %s" % (key.name) 
     key.delete() 

if __name__ == '__main__': 
    import sys 
    if len(sys.argv) < 3: 
     print 'Usage: %s <get/set/list/delete> <backup_filename>' % (sys.argv[0]) 
    else: 
     if sys.argv[1] == 'set': 
      save_file_in_s3(sys.argv[2]) 
     elif sys.argv[1] == 'get': 
      get_file_from_s3(sys.argv[2]) 
     elif sys.argv[1] == 'list': 
      list_backup_in_s3() 
     elif sys.argv[1] == 'delete': 
      delete_all_backups() 
     else: 
      print 'Usage: %s <get/set/list/delete> <backup_filename>' % (sys.argv[0]) 

Ma continuo a ricevere questo errore:

Traceback (most recent call last): 
    File "aws_s3.py", line 42, in <module> 
    save_file_in_s3(sys.argv[2]) 
    File "aws_s3.py", line 13, in save_file_in_s3 
    k.set_contents_from_filename(filename) 
    File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 1362, in set_contents_from_filename 
    encrypt_key=encrypt_key) 
    File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 1293, in set_contents_from_file 
    chunked_transfer=chunked_transfer, size=size) 
    File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 750, in send_file 
    chunked_transfer=chunked_transfer, size=size) 
    File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 951, in _send_file_internal 
    query_args=query_args 
    File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 664, in make_request 
    retry_handler=retry_handler 
    File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1071, in make_request 
    retry_handler=retry_handler) 
    File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1030, in _mexe 
    raise ex 
socket.error: [Errno 104] Connection reset by peer 

fatto un po 'della mia ricerca e ha scoperto che la sua una sorta di bug in boto. Come procedere ulteriormente con questo?

Answer 1

Come non ho ricevuto alcun aggiornamento su come farlo funzionare ho usato s3cmd nel mio script bash. Ma devo ancora testarlo per i file> 1 GB.

Ecco il codice aggiornato -

#!/bin/sh 

MONGODB_SHELL='/usr/bin/mongo' 

DUMP_UTILITY='/usr/bin/mongodump' 
DB_NAME='amicus' 

date_now=`date +%Y_%m_%d_%H_%M_%S` 
dir_name='db_backup_'${date_now} 
file_name='db_backup_'${date_now}'.bz2' 

log() { 
    echo $1 
} 

do_cleanup(){ 
    rm -rf db_backup_2010* 
    log 'cleaning up....' 
} 

do_backup(){ 
    log 'snapshotting the db and creating archive' && \ 

    ${DUMP_UTILITY} -d ${DB_NAME} -o ${dir_name} && tar -jcf $file_name ${dir_name} 

    log 'data backd up and created snapshot' 
} 

save_in_s3(){ 
    log 'saving the backup archive in amazon S3' && \ 
    python aws_s3.py set ${file_name} && \ 
    s3cmd put ${file_name} s3://YOURBUCKETNAME 
    log 'data backup saved in amazon s3' 
} 

do_backup && save_in_s3 && do_cleanup 

Answer 2

Questo probabilmente ha a che fare con la dimensione del file caricato.

"Connessione ripristinata dal peer" indica in genere che il server remoto ha chiuso la connessione (non pensare che sia un problema di boto). Anche andando a indovinare, questa è una sorta di timeout che stai colpendo per fare la richiesta (per un file di grandi dimensioni il trasferimento richiede molto tempo).

Raccomanderei di esaminare i caricamenti multipart se si desidera eseguire da soli. Vedi questo esempio: https://gist.github.com/chrishamant/1556484

s3cmd fa questo nella parte posteriore in base alla dimensione del file.