Simple way to migrate s3 buckets across AWS accounts

Method - I

We are not able to create buckets with the identical name on the source account. Since across accounts, s3 bucket names are unique for the entire s3 namespace. Create a new temporary bucket to copy contents to the destination. After moving contents to the destination, remove the s3 bucket as the source and create it to destination AWS account, to move content from the temporary bucket.

Step 1: Create IAM users at the origin and target server with reading and write access to the s3 bucket.

Step 2: Create an EC2 instance to copy content from the source bucket. You can also do it from any Linux machine, even from your Linux PC. However, I suggest an instance in the same AWS account to speed up (network speed should be good) the migration process.

Step 3: Configure those IAM users on your Linux instance.

Step 4: Check the permission of IAM user.

You can check this by listing buckets from your Linux instance.

aws s3 ls

Step 5: Copy contents from source bucket to the local EC2 instance. You can use the following command:

aws s3 cp s3://mybucket /path/to/local/ --recursive

Run the above command in a screen.

Step 6: After the process is completed, start the copy from local folder to remote s3 bucket.

aws s3 cp /path/to/local/ s3://mybucket --recursive --profile user2

Here the user2 is the IAM user who has access to the remote s3 bucket.

Step 7: Cross check all the content migrated correctly to the remote bucket.

You can use the following command to check files are copied correctly to remote bucket:

Source bucket:

aws s3 ls --summarize --human-readable --recursive s3://bucket-name|tail -n 2

Destination bucket:

aws s3 ls --summarize --human-readable --recursive s3://bucket-name --profile user2|tail -n 2

Method - II

By using single IAM user which has access to source and destination s3 buckets. Follow these steps:

Step 1: Create an IAM user on source AWS account which has the privilege to access s3 buckets.

Step 2: Give access to that IAM user on destinations AWS bucket.

To set the permission to an IAM user once s3 bucket is removed

Use the following policy, to add the permission to buckets :

----
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AddPerm",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::bucket-name/*"
        }
    ]
}
----

Step 3: Check the IAM user, whether he has access to both buckets

Use the following to check it by listing buckets from our Linux instance.

aws s3 ls s3://source-bucket

aws s3 ls s3://destination-bucket

Step 4: Start copying process

aws s3 cp s3://source-bucket/ s3://estination-bucket/ --recursive

You can also use the “sync” option to copy/sync content between buckets.

aws s3 sync s3://source-bucket/ s3://estination-bucket/ --recursive

As the migration is completed, remove the source bucket, create the same on destination, move contents also remove the temporary bucket.