Transfer files quickly from Amazon S3
It may be that you’re running a migration for someone, or you’ve decided to switch to using Digital Ocean volumes for shared disk storage across a cluster. Whatever the reason, downloading files from S3 using rsync and fuse is very slow. Here’s how to transfer files quickly.
The commands below install the older AWS CLI v1 via pip. On Ubuntu 24.04 the current tool is AWS CLI v2, installed from the official bundle at awscli.amazonaws.com. Either version runs the same aws s3 sync command shown here.
apt install python-pip
pip install awscli --upgrade --user
pip install --upgrade pip
apt install awscliNow that AWS is installed, you need to configure it with your access key and secret key.
aws configureIt’ll ask you for your access key and secret key, so paste them into the window. Once done, you can download your bucket.
aws s3 sync s3://mybucket /root/s3dest/aws s3 sync transfers many objects in parallel, which is why it comfortably beats a fuse-mounted bucket for a migration. In my experience it is about 4 or 5 times faster than fuse plus rsync.
That’s it – it’ll download the entire bucket now. Performance-wise, I have not benchmarked this precisely, but it’s about 4 or 5 times faster than using fuse and rsync.
Comments