Migrate data between KakaoCloud Object Storage buckets

This tutorial explains how to migrate data between Object Storage buckets using Rclone, an open-source command-line tool that supports cloud storage. It supports migrations between Classic/General buckets in the kr-gov-central-1 region and buckets in the kr-central-2 region.

Basic information

• Estimated time: Varies depending on the size of data
• Recommended OS: Rocky, Ubuntu
  
• Prerequisites
  
  • Issue S3 access key

Tutorial use cases

• KakaoCloud Object Storage supports both Swift API and S3 API, and this scenario shows how to migrate data between Object Storage buckets.
  
• You can specify the source/target bucket using either Classic or General buckets in the kr-gov-central-1 region, or buckets in the kr-central-2 region.
• If you are using other storage services that support Swift or S3 APIs, this tutorial can also serve as a reference for migrating your data to KakaoCloud Object Storage.

About this scenario

This scenario demonstrates how to migrate data from a source Object Storage bucket to a target Object Storage bucket using Rclone, which supports data transfers between various cloud storages. The main steps include:

• Installing the Rclone package
• Configuring Rclone to manage source/target Object Storage buckets
• Verifying data integrity after migration

caution

• All data transfers occur over the internet, so the VM instance used for migration must be able to connect to Object Storage via the internet.
  
• The migration VM instance can be created in any region regardless of the region of the target or source bucket.

Before you start

For this scenario, create a relay VM instance in the kr-central-2 region and an Object Storage bucket in the same region as the migration destination.

• Refer to Create a VM instance for instructions on creating the VM.
• For more details on managing Object Storage buckets, see Create and manage Object Storage buckets.

Also, since data transfer occurs over the internet, ensure that the network environment allows proper internet connectivity between the VM and Object Storage.

Getting started

The steps below walk you through the entire process of migrating data between Object Storage buckets.

Step 1. Access VM instance for data migration

Access the VM instance via SSH to install the Rclone package.

chmod 400 ${PRIVATE_KEY}.pem # Grant read permission
ssh -i ${PRIVATE_KEY}.pem ubuntu@${VM_PUBLIC_IP}

환경변수	설명
PRIVATE_KEY🖌︎	your-key-file-name
VM_PUBLIC_IP🖌︎	Check public IP from the VM instance tab

Step 2. Install Rclone

Install Rclone on the relay VM instance for data migration. For more details, see the Rclone Install guide.

Install Rclone

## Install Rclone 
curl https://rclone.org/install.sh | sudo bash
## Verify Rclone installation 
rclone --version

Step 3. Configure Rclone

Configure Rclone to manage KakaoCloud Object Storage from the relay VM instance.
Classic buckets in the kr-gov-central-1 region use User ID-based authentication, while General buckets in kr-gov-central-1 and all buckets in kr-central-2 use S3 access key-based authentication.
Enter the appropriate information for your source and target buckets. For details, refer to Rclone config.

info

For General buckets in kr-gov-central-1 and all buckets in kr-central-2, you must provide S3 access key credentials.
See Issue S3 access key for more information.

1. Add the configuration for the source/target buckets in the rclone.conf file:

   cat <<EOL > ~/.config/rclone/rclone.conf
   # Classic bucket in kr-gov-central-1 (Swift type)
   [gov-classic-obj]
   type = swift
   env_auth = true
   user = ${USER_ID}
   key = ${USER_PASSWORD}
   auth = https://iam.kakaocloud-kr-gov.com/identity/v3
   user_id = ${USER_ID}
   domain = ${DOMAIN_NAME}
   tenant_id = ${PROJECT_ID}
   tenant = ${PROJECT_NAME}
   tenant_domain = ${DOMAIN_NAME}
   region = kr-gov-central-1
   endpoint_type = 
   auth_version = 3
   # General bucket in kr-gov-central-1 (S3 type)
   [gov-obj]
   type = s3
   provider = Ceph
   env_auth = True
   access_key_id = ${kr-gov-central-1_CREDENTIAL_S3_ACCESS_KEY}
   secret_access_key = ${kr-gov-central-1_CREDENTIAL_S3_SECRET_ACCESS_KEY}
   region = kr-gov-central-1
   endpoint = https://objectstorage.kr-gov-central-1.kakaocloud-kr-gov.com
   # Bucket in kr-central-2 (S3 type)
   [kr2-obj]
   type = s3
   provider = Ceph
   env_auth = True
   access_key_id = ${kr-central-2_S3_CREDENTIAL_ACCESS_KEY}
   secret_access_key = ${kr-central-2_S3_CREDENTIAL_SECRET_ACCESS_KEY}
   region = kr-central-2
   endpoint = https://objectstorage.kr-central-2.kakaocloud.com
   EOL

2. Use the lsd command to list all buckets and verify that Rclone is connected correctly:

   Verify connection between Rclone and Object Storage

   # Classic bucket in kr-gov-central-1
   rclone lsd gov-classic-obj:/
   # General bucket in kr-gov-central-1
   rclone lsd gov-obj:/
   # Bucket in kr-central-2
   rclone lsd kr2-obj:/

Step 4. Migrate data

Use the sync command to synchronize data between the source and destination buckets.
The following example migrates data from a General bucket in kr-gov-central-1 to a bucket in kr-central-2. See the Rclone sync documentation for details.

# Dry-run before actual migration
rclone sync gov-obj:/"${kr-gov-central-1_BUCKET}" kr2-obj:/"${kr-central-2_BUCKET}" --progress -v \
--create-empty-src-dirs --log-file "${LOGFILE}" --log-format date,time,KST,longfile --fast-list --dry-run
# Actual migration
rclone sync gov-obj:/"${kr-gov-central-1_BUCKET}" kr2-obj:/"${kr-central-2_BUCKET}" --progress -v \
--create-empty-src-dirs --log-file "${LOGFILE}" --log-format date,time,KST,longfile --fast-list

환경변수	설명
kr-gov-central-1_BUCKET🖌︎	name of general bucket in kr-gov-central-1
kr-central-2_BUCKET🖌︎	name of target bucket in kr-central-2
LOGFILE🖌︎	log file name

Warning

Using the sync command will overwrite the destination data. Make sure to verify the source and destination settings beforehand.
Reversing them may lead to data loss.

Step 5. Verify data integrity

Check the integrity of the migrated data to ensure everything transferred correctly.

# Verify data consistency
rclone check gov-obj:/"${kr-gov-central-1_BUCKET}" kr2-obj:/"${kr-central-2_BUCKET}" --progress -v
# Compare object count and size between source and destination
rclone size gov-obj:/"${kr-gov-central-1_BUCKET}"
rclone size kr2-obj:/"${kr-central-2_BUCKET}"

환경변수	설명
kr-gov-central-1_BUCKET🖌︎	name of general bucket in kr-gov-central-1
kr-central-2_BUCKET🖌︎	name of target bucket in kr-central-2

info

If issues occur during the integrity check, please contact Helpdesk.