Data migration from AWS ElastiCache to MemStore

This document provides a step-by-step guide on how to migrate data from AWS ElastiCache to MemStore.

Basic Information

• Estimated Time: Approximately 30 minutes
• Recommended OS: MacOS, Ubuntu
• Region: kr-central-2
• Prerequisites
  • Issue an access key
  • Obtain credentials for S3 API usage

Attention

Currently, the supported Redis OSS engine versions for backup in KakaoCloud MemStore are 6.2.5 and 5.0.6. If you are using Redis OSS version 7 or higher on AWS ElastiCache, restoration is not supported, and the migration cannot proceed.

About this scenario

This guide outlines the steps for migrating data from AWS ElastiCache to MemStore. We use Rclone to facilitate data transfer between cloud storage services, and the major steps are as follows:

• Backup the AWS ElastiCache Redis cluster: Generate a backup file of your Redis cluster and store it in Amazon S3.
• Transfer backup file with rclone: Move the backup file from Amazon S3 to KakaoCloud Object Storage, verifying data consistency.
• Restore MemStore cluster: Use the transferred backup file to restore data into the KakaoCloud MemStore cluster.

Preparation in KakaoCloud

To proceed with the data migration, configure the necessary resources in KakaoCloud.

1. Create an Object Storage bucket: Set up a bucket in KakaoCloud Object Storage for data migration. Refer to the Object Storage documentation for bucket creation instructions.
2. Create a VM instance for data migration: Launch a VM instance with internet access to install and use Rclone. Refer to the VM instance creation guide for details.

Preparation in AWS

Prepare the resources in AWS needed for backup and data migration.

1. Prepare the AWS ElastiCache Redis cache: Set up an AWS ElastiCache Redis OSS cache for migration, ensuring the engine version is 6.2.6 or lower.
2. Create an Amazon S3 bucket: Create a bucket in Amazon S3 to store the backup and migration data.
3. Issue an Amazon S3 access key: Obtain an access key for S3 bucket access.

Step-by-Step Guide

Follow the steps below to migrate data from AWS ElastiCache to KakaoCloud MemStore.

Step 1. Backup AWS ElastiCache Redis cluster

1. In the AWS console, navigate to Amazon ElastiCache > Redis OSS Cache and select the cache to back up.
2. Click [Actions] in the upper-right corner, then choose [Back up].
3. In the backup creation page, enter a backup name, verify the default options, and click [Create backup].
4. Verify the newly created backup under the Backup tab.

Step 2. Export backup file to Amazon S3 bucket

1. In the AWS console, go to Amazon ElastiCache > Backups and select the backup you want to export to S3.
2. Click [Actions] and select [Export].
3. In the export page, specify the backup name and S3 bucket, then click [Export].
4. Confirm that the backup file appears in your S3 bucket.

Step 3. Transfer backup file from Amazon S3 to Object Storage

Use Rclone to transfer data. For more details, refer to the Data migration tutorial from AWS S3 to Object Storage.

1. Install the rclone package on a VM instance. For detailed installation instructions, refer to the Rclone installation guide.

   Rclone installation

   ## Rclone installation 
   curl https://rclone.org/install.sh | sudo bash
   
   ## Verify Rclone installation status 
   rclone --version

2. Configure the rclone settings file (rclone.conf) to include KakaoCloud and AWS configurations. The following settings are required for data transfer between KakaoCloud Object Storage and AWS S3.

   cat <<EOL > ~/.config/rclone/rclone.conf
   
   # KakaoCloud Object Storage settings
   [${KAKAO_BUCKET}]
   type = s3
   provider = Ceph
   env_auth = True
   access_key_id = ${CREDENTIAL_ACCESS_KEY}
   secret_access_key = ${CREDENTIAL_SECRET_ACCESS_KEY}
   region = kr-central-2
   endpoint = https://objectstorage.kr-central-2.kakaocloud.com
   
   # AWS S3 settings
   [${AWS_BUCKET}]
   type = s3
   provider = AWS
   access_key_id = ${AWS_ACCESS_KEY}
   secret_access_key = ${AWS_SECRET_ACCESS_KEY}
   region = ${AWS_REGION}
   EOL

   환경변수	설명
   KAKAO_BUCKET🖌︎	KakaoCloud Object Storage bucket name
   CREDENTIAL_ACCESS_KEY🖌︎	S3 API access key
   CREDENTIAL_SECRET_ACCESS_KEY🖌︎	Secure access key for S3 API
   AWS_BUCKET🖌︎	AWS S3 bucket name
   AWS_ACCESS_KEY🖌︎	AWS access key
   AWS_SECRET_ACCESS_KEY🖌︎	AWS secure access key
   AWS_REGION🖌︎	AWS region

   info

   Note: Enter the access_key_id and secret_access_key values from each cloud's S3 API credentials. For KakaoCloud, refer to Issuing Credentials for S3 API.

3. Use the lsd command to list all buckets in Object Storage. This allows you to verify that rclone is properly connected to Object Storage.

   Verify Rclone Connection to Object Storage

   # Verify connection to KakaoCloud Object Storage
   rclone lsd ${KAKAO_BUCKET}:/
   
   # Verify connection to AWS S3
   rclone lsd ${AWS_BUCKET}:/
   

   환경변수	설명
   KAKAO_BUCKET🖌︎	KakaoCloud Object Storage bucket name
   AWS_BUCKET🖌︎	AWS S3 bucket name

4. Transfer backup files from AWS S3 to KakaoCloud Object Storage.

   # Pre-transfer test from AWS S3 to KakaoCloud Object Storage 
   rclone sync aws-s3:/"${AWS_BUCKET}" kakaocloud-obj:/"${KAKAO_BUCKET}" --progress -v \
   --create-empty-src-dirs --metadata --log-file "${LOGFILE}" --log-format date,time,KST,longfile --dry-run
   
   # Transfer from AWS S3 to KakaoCloud Object Storage
   rclone sync aws-s3:/"${AWS_BUCKET}" kakaocloud-obj:/"${KAKAO_BUCKET}" --progress -v \
   --create-empty-src-dirs --metadata --log-file "${LOGFILE}" --log-format date,time,KST,longfile

   환경변수	설명
   AWS_BUCKET🖌︎	AWS bucket name
   KAKAO_BUCKET🖌︎	KakaoCloud bucket name
   LOGFILE🖌︎	Log file name

5. Check the data integrity of the backup files.

   # Verify data integrity
   rclone check aws-s3:/"${AWS_BUCKET}" kakaocloud-obj:/"${KAKAO_BUCKET}" --progress -v 
   
   # Check the number of objects and capacity in both Object Storages
   rclone size aws-s3:/"${AWS_BUCKET}"
   rclone size kakaocloud-obj:/"${KAKAO_BUCKET}"

   환경변수	설명
   AWS_BUCKET🖌︎	AWS S3 bucket name
   KAKAO_BUCKET🖌︎	KakaoCloud Object Storage bucket name

6. Check the objects in the bucket in the KakaoCloud console under the Object Storage service.

Step 4. Restore MemStore cluster using transferred backup files

caution

• Restoration is only possible in Cluster Not in Use mode.

info

• Redis OSS engine version 7 or higher cannot be restored. This tutorial uses the AWS ElastiCache with Redis OSS engine 6.2.6.
• If the cache size of the source backup exceeds 60% of the memory of the cluster being restored, restoration may fail.

1. Navigate to KakaoCloud Console > MemStore and click the [Create Cluster] button.

2. On the Create Cluster details page, configure the items according to the table below, and set the remaining items as needed.

   Item	Setting
   Configuration Option	Cluster Restore
   Backup Type	User Backup File
   Backup File Path	Enter the Object Storage path of the backup file (ex. BucketName/FolderName/FileName)

3. After clicking the [Create] button, verify that the cluster is created successfully.

4. Use the access port of the MemStore cluster to connect and check the data. For detailed instructions, please refer to the Cluster Management document.

   Connection Command

   redis-cli -h {ENDPOINT} -p {Port Number} {Command}

   Item	Description
   ENDPOINT	Endpoint information obtained from the Details tab
   Port Number	Port number for Redis connection