Skip to main content

Data migration from AWS ElastiCache to MemStore

This document provides a step-by-step guide on how to migrate data from AWS ElastiCache to MemStore.

Basic Information
caution
  • To ensure the stability of the MemStore service, restoration is only possible with the same or higher engine version used during the backup.
  • When restoring from an existing MemStore backup, restoration may fail if the cache size of the source backup exceeds 60% of the memory of the cluster flavor being restored.

About this scenario​

This guide outlines the steps for migrating data from AWS ElastiCache to MemStore. We use Rclone to facilitate data transfer between cloud storage services, and the major steps are as follows:

  • Backup the AWS ElastiCache Redis cluster: Generate a backup file of your Redis cluster and store it in Amazon S3.
  • Transfer backup file with rclone: Move the backup file from Amazon S3 to KakaoCloud Object Storage, verifying data consistency.
  • Restore MemStore cluster: Use the transferred backup file to restore data into the KakaoCloud MemStore cluster.

Scenario architecture

Preparation in KakaoCloud​

To proceed with the data migration, configure the necessary resources in KakaoCloud.

  1. Create an Object Storage bucket: Set up a bucket in KakaoCloud Object Storage for data migration. Refer to the Object Storage documentation for bucket creation instructions.
  2. Create a VM instance for data migration: Launch a VM instance with internet access to install and use Rclone. Refer to the VM instance creation guide for details.

Preparation in AWS​

Prepare the resources in AWS needed for backup and data migration.

  1. Prepare the AWS ElastiCache Redis cache: Set up an AWS ElastiCache Redis OSS cache for migration, ensuring the engine version is 6.2.6 or lower.
  2. Create an Amazon S3 bucket: Create a bucket in Amazon S3 to store the backup and migration data.
  3. Issue an Amazon S3 access key: Obtain an access key for S3 bucket access.

Step-by-Step Guide​

Follow the steps below to migrate data from AWS ElastiCache to KakaoCloud MemStore.

Step 1. Backup AWS ElastiCache Redis cluster​

  1. In the AWS console, navigate to Amazon ElastiCache > Redis OSS Cache and select the cache to back up.
  2. Click [Actions] in the upper-right corner, then choose [Back up].
  3. In the backup creation page, enter a backup name, verify the default options, and click [Create backup].
  4. Verify the newly created backup under the Backup tab.

Step 2. Export backup file to Amazon S3 bucket​

  1. In the AWS console, go to Amazon ElastiCache > Backups and select the backup you want to export to S3.
  2. Click [Actions] and select [Export].
  3. In the export page, specify the backup name and S3 bucket, then click [Export].
  4. Confirm that the backup file appears in your S3 bucket.

Step 3. Transfer backup file from Amazon S3 to Object Storage​

Use Rclone to transfer data. For more details, refer to the Data migration tutorial from AWS S3 to Object Storage.

  1. Install the rclone package on a VM instance. For detailed installation instructions, refer to the Rclone installation guide.

    Rclone installation
    ## Rclone installation 
    curl https://rclone.org/install.sh | sudo bash

    ## Verify Rclone installation status 
    rclone --version

  2. Configure the rclone settings file (rclone.conf) to include KakaoCloud and AWS configurations. The following settings are required for data transfer between KakaoCloud Object Storage and AWS S3.

    cat <<EOL > ~/.config/rclone/rclone.conf

    # KakaoCloud Object Storage settings
    [${KAKAO_BUCKET}]
    type = s3
    provider = Ceph
    env_auth = True
    access_key_id = ${CREDENTIAL_ACCESS_KEY}
    secret_access_key = ${CREDENTIAL_SECRET_ACCESS_KEY}
    region = kr-central-2
    endpoint = https://objectstorage.kr-central-2.kakaocloud.com

    # AWS S3 settings
    [${AWS_BUCKET}]
    type = s3
    provider = AWS
    access_key_id = ${AWS_ACCESS_KEY}
    secret_access_key = ${AWS_SECRET_ACCESS_KEY}
    region = ${AWS_REGION}
    EOL
    환경변수설명
    KAKAO_BUCKET🖌︎ KakaoCloud Object Storage bucket name
    CREDENTIAL_ACCESS_KEY🖌︎ S3 API access key
    CREDENTIAL_SECRET_ACCESS_KEY🖌︎Secure access key for S3 API
    AWS_BUCKET🖌︎ AWS S3 bucket name
    AWS_ACCESS_KEY🖌︎AWS access key
    AWS_SECRET_ACCESS_KEY🖌︎AWS secure access key
    AWS_REGION🖌︎AWS region
    info

    Note: Enter the access_key_id and secret_access_key values from each cloud's S3 API credentials. For KakaoCloud, refer to Issuing Credentials for S3 API.

  3. Use the lsd command to list all buckets in Object Storage. This allows you to verify that rclone is properly connected to Object Storage.

    Verify Rclone Connection to Object Storage
    # Verify connection to KakaoCloud Object Storage
    rclone lsd ${KAKAO_BUCKET}:/

    # Verify connection to AWS S3
    rclone lsd ${AWS_BUCKET}:/

    환경변수설명
    KAKAO_BUCKET🖌︎ KakaoCloud Object Storage bucket name
    AWS_BUCKET🖌︎ AWS S3 bucket name

  4. Transfer backup files from AWS S3 to KakaoCloud Object Storage.

    # Pre-transfer test from AWS S3 to KakaoCloud Object Storage 
    rclone sync aws-s3:/"${AWS_BUCKET}" kakaocloud-obj:/"${KAKAO_BUCKET}" --progress -v \
    --create-empty-src-dirs --metadata --log-file "${LOGFILE}" --log-format date,time,KST,longfile --dry-run

    # Transfer from AWS S3 to KakaoCloud Object Storage
    rclone sync aws-s3:/"${AWS_BUCKET}" kakaocloud-obj:/"${KAKAO_BUCKET}" --progress -v \
    --create-empty-src-dirs --metadata --log-file "${LOGFILE}" --log-format date,time,KST,longfile
    환경변수설명
    AWS_BUCKET🖌︎AWS bucket name
    KAKAO_BUCKET🖌︎KakaoCloud bucket name
    LOGFILE🖌︎Log file name

  5. Check the data integrity of the backup files.

    # Verify data integrity
    rclone check aws-s3:/"${AWS_BUCKET}" kakaocloud-obj:/"${KAKAO_BUCKET}" --progress -v 

    # Check the number of objects and capacity in both Object Storages
    rclone size aws-s3:/"${AWS_BUCKET}"
    rclone size kakaocloud-obj:/"${KAKAO_BUCKET}"
    환경변수설명
    AWS_BUCKET🖌︎AWS S3 bucket name
    KAKAO_BUCKET🖌︎KakaoCloud Object Storage bucket name

  6. Check the objects in the bucket in the KakaoCloud console under the Object Storage service.

Step 4. Restore MemStore cluster using transferred backup files​

caution
  • Restoration is only possible in Cluster Not in Use mode.
info
  • Redis OSS engine version 7 or higher cannot be restored. This tutorial uses the AWS ElastiCache with Redis OSS engine 6.2.6.
  • If the cache size of the source backup exceeds 60% of the memory of the cluster being restored, restoration may fail.

  1. Navigate to KakaoCloud Console > MemStore and click the [Create Cluster] button.

  2. On the Create Cluster details page, configure the items according to the table below, and set the remaining items as needed.

    ItemSetting
    Configuration OptionCluster Restore
    Backup TypeUser Backup File
    Backup File PathEnter the Object Storage path of the backup file (ex. BucketName/FolderName/FileName)

  3. After clicking the [Create] button, verify that the cluster is created successfully.

  4. Use the access port of the MemStore cluster to connect and check the data. For detailed instructions, please refer to the Cluster Management document.

    Connection Command
    redis-cli -h {ENDPOINT} -p {Port Number} {Command}
    ItemDescription
    ENDPOINTEndpoint information obtained from the Details tab
    Port NumberPort number for Redis connection