Skip to main content

Migrating data from AWS S3 to KakaoCloud Object Storage

You can migrate data from AWS S3 to KakaoCloud Object Storage using Rclone, an open-source command-line tool that supports cloud storage.
This document explains some usage examples of Rclone. For more details on Rclone's features, refer to the Rclone Reference documentation.

info

About this scenario​

This scenario explains how to migrate data from AWS S3 to KakaoCloud Object Storage. The migration process utilizes Rclone, which supports data transfer between various cloud storage platforms.
The main tasks in this scenario are as follows:

  • Install Rclone: Install Rclone to manage cloud storage.
  • Configure Rclone: Configure Rclone to manage both AWS S3 and KakaoCloud Object Storage.
  • Migrate data: Transfer data from AWS S3 to KakaoCloud Object Storage and verify data integrity.

Scenario Architecture

caution

Since the data migration is conducted over the internet, ensure that the VM instance for data migration has an active internet connection to KakaoCloud Object Storage.

Step 1. Access the VM instance for data migration​

Access the intermediate VM instance via SSH to install the Rclone package.

chmod 400 ${PRIVATE_KEY}.pem # Grant read permissions
ssh -i ${PRIVATE_KEY}.pem ubuntu@${VM_PUBLIC_IP}
Environment VariableDescription
PRIVATE_KEY🖌︎Key file name
VM_PUBLIC_IP🖌︎Check public IP in the Virtual Machine > Instance tab

Step 2. Install Rclone​

Install the Rclone package on the intermediate VM instance for data migration. For detailed instructions, refer to the Rclone Install documentation.

Install Rclone
## Install Rclone  
curl https://rclone.org/install.sh | sudo bash

## Confirm the installation 
rclone --version

Step 3. Configure Rclone​

You can manage both KakaoCloud Object Storage and AWS S3 using Rclone on the intermediate VM instance. Rclone needs to be configured for this.
For detailed instructions, refer to the Rclone config documentation.

  1. Add the KakaoCloud and AWS configuration details to the rclone.conf file.

    cat <<EOL > ~/.config/rclone/rclone.conf

    # KakaoCloud Object Storage configuration
    [kakaocloud-obj]
    type = s3
    provider = Ceph
    env_auth = True
    access_key_id = ${CREDENTIAL_ACCESS_KEY}
    secret_access_key = ${CREDENTIAL_SECRET_ACCESS_KEY}
    region = kr-central-2
    endpoint = https://objectstorage.kr-central-2.kakaocloud.com

    # AWS S3 configuration
    [aws-s3]
    type = s3
    provider = AWS
    access_key_id = ${AWS_ACCESS_KEY}
    secret_access_key = ${AWS_SECRET_ACCESS_KEY}
    region = ${AWS_REGION}
    EOL
    Environment VariableDescription
    CREDENTIAL_ACCESS_KEY🖌︎ Access key for S3 API
    CREDENTIAL_SECRET_ACCESS_KEY🖌︎ Secret access key for S3 API
    AWS_ACCESS_KEY🖌︎AWS access key
    AWS_SECRET_ACCESS_KEY🖌︎AWS secret access key
    AWS_REGION🖌︎AWS region
    info

    To configure KakaoCloud Object Storage, you need to issue credentials required for using the S3 API.
    For detailed steps, refer to the Issuing credentials for S3 API documentation.

  2. Use the lsd command to list all the buckets in Object Storage. This will verify that Rclone is correctly connected to Object Storage.

    Verify connection between Rclone and Object Storage
    # Verify KakaoCloud Object Storage connection
    rclone lsd kakaocloud-obj:/

    # Verify AWS S3 connection
    rclone lsd aws-s3:/

Step 4. Migrate data​

Use the sync command to synchronize the data between the source and destination. This allows you to migrate specific bucket data from AWS S3 to KakaoCloud Object Storage. For more details, refer to the Rclone sync documentation.

# Test data migration from AWS S3 to KakaoCloud Object Storage 
rclone sync aws-s3:/"${AWS_BUCKET}" kakaocloud-obj:/"${KAKAO_BUCKET}" --progress -v \
--create-empty-src-dirs --metadata --log-file "${LOGFILE}" --log-format date,time,KST,longfile --dry-run

# Migrate data from AWS S3 to KakaoCloud Object Storage
rclone sync aws-s3:/"${AWS_BUCKET}" kakaocloud-obj:/"${KAKAO_BUCKET}" --progress -v \
--create-empty-src-dirs --metadata --log-file "${LOGFILE}" --log-format date,time,KST,longfile
Environment VariableDescription
AWS_BUCKET🖌︎AWS Bucket name
KAKAO_BUCKET🖌︎KakaoCloud Bucket name
LOGFILE🖌︎Log file name
caution

When using the sync command for data migration, the data at the destination will be modified. Therefore, before proceeding, ensure that the source and destination are correct. If they are reversed, data loss may occur.

  • Source: AWS S3
  • Destination: KakaoCloud Object Storage
info

The --metadata option in the sync example is supported from Rclone 1.66 onwards.

Step 5. Verify data integrity​

You can verify the integrity of the data to ensure that the migration was successful.

# Verify data integrity
rclone check aws-s3:/"${AWS_BUCKET}" kakaocloud-obj:/"${KAKAO_BUCKET}" --progress -v 

# Check the number of objects and capacity in both Object Storages
rclone size aws-s3:/"${AWS_BUCKET}"
rclone size kakaocloud-obj:/"${KAKAO_BUCKET}"
Environment VariableDescription
AWS_BUCKET🖌︎AWS S3 Bucket name
KAKAO_BUCKET🖌︎KakaoCloud Object Storage Bucket name