Automating web server log storage to Object Storage

This guide provides a script to automate the process of storing web server logs from a Virtual Machine to Object Storage.

info

• Estimated time: 15 minutes
• Recommended Operating system: MacOS, Ubuntu
• Region: kr-central-1

Prework

Before starting this tutorial, you need to have Access Keys and VM access key pair ready from the KakaoCloud Console.

Step 1. Set up a VM-based web service

Follow the previous tutorial VM-based web service setup to configure a web server environment. The resource specifications provided in that document can be adjusted as needed, except for the operating system (Ubuntu 20.04).

Step 2. SSH into the web service

Use ssh to connect to the web server. If the public IP is not configured, connect via a Bastion host or other methods. This step applies to all instances where the web server is provisioned.

# web-1
ssh -i ${PRIVATE_KEY} ubuntu@${WEB1_ENDPOINT}

# web-2
ssh -i ${PRIVATE_KEY} ubuntu@${WEB1_ENDPOINT}

Step 3. Write the Object Storage environment setup script

1. Install the jq package in shell to easily handle json formatted data.

   sudo apt-get update -y
   sudo apt-get install -y jq

2. After connecting to the instance via ssh, create an environment variable file to store the logs. Refer to the table below and input the Access Key and Secret Access Key in the environment variables.

   cat << \EOF | sudo tee /tmp/env.sh
   #!/bin/bash
   
   export ACCESS_KEY="${ACCESS_KEY}"
   export ACCESS_SECRET_KEY="${ACCESS_SECRET_KEY}"
   
   # Don't Edit
   export BUCKET_NAME="hands-on"
   export FILE="/var/log/nginx/access.log"
   export TOPATH="/log/nginx/date_id=$(date +%Y-%m-%d)/host_id=$(hostname -s)/access.log"
   EOF

   Environment Variable Key	Environment Variable Value
   ${ACCESS_KEY}	Access Key
   ${ACCESS_SECRET_KEY}	Secret Access Key

   • You can obtain an API authentication token and Project ID using the ‘Access Key’ and ‘User Security Access Key’ written in the environment variable file.
   • For detailed information on token issuance, please refer to API authentication token.

   cat << \EOF | sudo tee /tmp/token.sh
   #!/bin/bash
   
   export TOKEN=$(curl -s -X POST -i https://iam.kakaocloud.com/identity/v3/auth/tokens -H "Content-Type: application/json" -d \
   '{
       "auth": {
           "identity": {
               "methods": [
                   "application_credential"
               ],
               "application_credential": {
                   "id": "'${ACCESS_KEY}'",
                   "secret": "'${ACCESS_SECRET_KEY}'"
               }
           }
       }
   }' | grep x-subject-token | awk -v RS='\r\n' '{print $2}')
   
   export PROJECT_ID=$(curl -s -X POST https://iam.kakaocloud.com/identity/v3/auth/tokens -H "Content-Type: application/json" -d \
   '{
       "auth": {
           "identity": {
               "methods": [
                   "application_credential"
               ],
               "application_credential": {
                   "id": "'${ACCESS_KEY}'",
                   "secret": "'${ACCESS_SECRET_KEY}'"
               }
           }
       }
   }' | jq -r ".token.project.id")
   
   if [ -z $TOKEN ]; then
           echo "TOKEN is null..."
           exit 0
   fi
   
   if [ -z $PROJECT_ID ]; then
           echo "PROJECT_ID is null..."
           exit 0
   fi
   EOF

3. Create an Object Storage bucket to load logs using environment variables and token issuance script. Check the table below for information about the Object Storage bucket being created.

   cat << \EOF | sudo tee /tmp/init.sh
   #!/bin/bash
   
   . /tmp/env.sh
   . /tmp/token.sh
   
   function create_bukkit() {
           echo "create bukkit."
           curl -s -X PUT 'https://objectstorage.kr-central-1.kakaocloud.com/v1_ext/bucket' \\
           -H "X-Auth-Token:${TOKEN}" \\
           -H 'Content-Type: application/json' \\
           -d '{
   "name": "'${BUCKET_NAME}'",
   "type": "hot",
   "use_encryption": true
   }'
           echo "create end: ${BUCKET_NAME}"
   }
   
   create_bukkit
   
   EOF
   bash /tmp/init.sh
   

   Item	Information
   Bucket type	hot
   bucket name	hands-on
   Cryptography	Disabled

4. Write a script to upload log files to Object Storage according to the information defined in the environment variables file.

   cat << \EOF | sudo tee /tmp/upload.sh
   #!/bin/bash
   
   . /tmp/env.sh
   . /tmp/token.sh
   . /tmp/preupload.sh
   
   function upload_file() {
           echo "upload file: ${FILE}"
           curl -s -X PUT 'https://objectstorage.kr-central-1.kakaocloud.com/v1/'${PROJECT_ID}'/'${BUCKET_NAME}''${TOPATH}'' \
           -H 'X-Auth-Token: '${TOKEN}'' \
           -H 'Content-Type: application/octet-stream' \
           -T ${FILE}
           echo "upload end: '${BUCKET_NAME}' '${TOPATH}'"
   }
   
   upload_file
   . /tmp/postupload.sh
   EOF

5. Write a script to run before and after uploading the log files. If you uploaded a log file, create a backup file and set it to create a new log file.

   cat << EOF | sudo tee /tmp/preupload.sh
   #!/bin/bash
   
   EOF
   
   cat << EOF | sudo tee /tmp/postupload.sh
   #!/bin/bash
   
   sudo mv /var/log/nginx/access.log /var/log/nginx/access.log.backup
   sudo kill -USR1 `cat /var/run/nginx.pid`
   EOF

Step 4. Automatic upload environment settings

1. Use the cron package to automatically upload log files at regular intervals.

   sudo apt update -y
   sudo apt install -y cron

2. Write a command in crontab to run the upload script every midnight.

   sudo rm /etc/logrotate.d/nginx
   cat << EOF > tmp_crontab
   0 0 * * * /bin/bash /tmp/upload.sh
   EOF
   sudo crontab tmp_crontab
   rm tmp_crontab

3. Verify that the cron job is registered. Then run the upload script to upload the logs to Object Storage.

   sudo crontab -l
   bash /tmp/upload.sh

4. Verify that the logs are saved in the Object Storage console.