Migrating data from AWS S3 to KakaoCloud Object Storage

You can migrate data from AWS S3 to KakaoCloud Object Storage using Rclone, an open-source command-line tool that supports cloud storage.
This document explains some usage examples of Rclone. For more details on Rclone's features, refer to the Rclone Reference documentation.

info

• Estimated time required: 30 minutes
  
• User environment
  • Recommended operating systems: Rocky, Ubuntu
    
  • Region: kr-central-2
    
• Prerequisites
  
  • KakaoCloud
    • Access key
    • Credential
  • AWS
    • AWS S3
    • AWS S3 access key

About this scenario

This scenario explains how to migrate data from AWS S3 to KakaoCloud Object Storage. The migration process utilizes Rclone, which supports data transfer between various cloud storage platforms.
The main tasks in this scenario are as follows:

• Install Rclone: Install Rclone to manage cloud storage.
• Configure Rclone: Configure Rclone to manage both AWS S3 and KakaoCloud Object Storage.
• Migrate data: Transfer data from AWS S3 to KakaoCloud Object Storage and verify data integrity.

caution

Since the data migration is conducted over the internet, ensure that the VM instance for data migration has an active internet connection to KakaoCloud Object Storage.

Step 1. Access the VM instance for data migration

Access the intermediate VM instance via SSH to install the Rclone package.

chmod 400 ${PRIVATE_KEY}.pem # Grant read permissions
ssh -i ${PRIVATE_KEY}.pem ubuntu@${VM_PUBLIC_IP}

Environment Variable	Description
PRIVATE_KEY🖌︎	Key file name
VM_PUBLIC_IP🖌︎	Check public IP in the Virtual Machine > Instance tab

Step 2. Install Rclone

Install the Rclone package on the intermediate VM instance for data migration. For detailed instructions, refer to the Rclone Install documentation.

Install Rclone

## Install Rclone  
curl https://rclone.org/install.sh | sudo bash

## Confirm the installation 
rclone --version

Step 3. Configure Rclone

You can manage both KakaoCloud Object Storage and AWS S3 using Rclone on the intermediate VM instance. Rclone needs to be configured for this.
For detailed instructions, refer to the Rclone config documentation.

1. Add the KakaoCloud and AWS configuration details to the rclone.conf file.

   cat <<EOL > ~/.config/rclone/rclone.conf
   
   # KakaoCloud Object Storage configuration
   [kakaocloud-obj]
   type = s3
   provider = Ceph
   env_auth = True
   access_key_id = ${CREDENTIAL_ACCESS_KEY}
   secret_access_key = ${CREDENTIAL_SECRET_ACCESS_KEY}
   region = kr-central-2
   endpoint = https://objectstorage.kr-central-2.kakaocloud.com
   
   # AWS S3 configuration
   [aws-s3]
   type = s3
   provider = AWS
   access_key_id = ${AWS_ACCESS_KEY}
   secret_access_key = ${AWS_SECRET_ACCESS_KEY}
   region = ${AWS_REGION}
   EOL

   Environment Variable	Description
   CREDENTIAL_ACCESS_KEY🖌︎	Access key for S3 API
   CREDENTIAL_SECRET_ACCESS_KEY🖌︎	Secret access key for S3 API
   AWS_ACCESS_KEY🖌︎	AWS Access key
   AWS_SECRET_ACCESS_KEY🖌︎	AWS secret access key
   AWS_REGION🖌︎	AWS region

   info

   To configure KakaoCloud Object Storage, you need to issue credentials required for using the S3 API.
   For detailed steps, refer to the Issuing credentials for S3 API documentation.

2. Use the lsd command to list all the buckets in Object Storage. This will verify that Rclone is correctly connected to Object Storage.

   Verify connection between Rclone and Object Storage

   # Verify KakaoCloud Object Storage connection
   rclone lsd kakaocloud-obj:/
   
   # Verify AWS S3 connection
   rclone lsd aws-s3:/

Step 4. Migrate data

Use the sync command to synchronize the data between the source and destination. This allows you to migrate specific bucket data from AWS S3 to KakaoCloud Object Storage. For more details, refer to the Rclone sync documentation.

# Test data migration from AWS S3 to KakaoCloud Object Storage 
rclone sync aws-s3:/"${AWS_BUCKET}" kakaocloud-obj:/"${KAKAO_BUCKET}" --progress -v \
--create-empty-src-dirs --metadata --log-file "${LOGFILE}" --log-format date,time,KST,longfile --dry-run

# Migrate data from AWS S3 to KakaoCloud Object Storage
rclone sync aws-s3:/"${AWS_BUCKET}" kakaocloud-obj:/"${KAKAO_BUCKET}" --progress -v \
--create-empty-src-dirs --metadata --log-file "${LOGFILE}" --log-format date,time,KST,longfile

Environment Variable	Description
AWS_BUCKET🖌︎	AWS Bucket name
KAKAO_BUCKET🖌︎	KakaoCloud Bucket name
LOGFILE🖌︎	Log file name

caution

When using the sync command for data migration, the data at the destination will be modified. Therefore, before proceeding, ensure that the source and destination are correct. If they are reversed, data loss may occur.

• Source: AWS S3
• Destination: KakaoCloud Object Storage

info

The --metadata option in the sync example is supported from Rclone 1.66 onwards.

Step 5. Verify data integrity

You can verify the integrity of the data to ensure that the migration was successful.

# Verify data integrity
rclone check aws-s3:/"${AWS_BUCKET}" kakaocloud-obj:/"${KAKAO_BUCKET}" --progress -v 

# Check the number of objects and capacity in both Object Storages
rclone size aws-s3:/"${AWS_BUCKET}"
rclone size kakaocloud-obj:/"${KAKAO_BUCKET}"

Environment Variable	Description
AWS_BUCKET🖌︎	AWS S3 Bucket name
KAKAO_BUCKET🖌︎	KakaoCloud Object Storage Bucket name