Storing Web Server Logs in Object Storage

This guide provides a script to automate the storage of web server logs deployed on a Virtual Machine into Object Storage.

info

Estimated time: 15 minutes
Recommended Environment
- OS: MacOS, Ubuntu
- Region: kr-central-2
Prerequisites
- Access Key
- Key Pair for VM Access

Introduction

In this scenario, you will learn how to automatically store logs from a web server deployed on a Virtual Machine into KakaoCloud Object Storage. Using an automated script simplifies the log upload process and improves the efficiency of log management and data retention.

Getting started

This scenario consists of the following steps to set up an environment for automatically storing web server logs in Object Storage.

Step 1. Set Up VM-based Web Service

Refer to the tutorial Set Up a VM-based Web Service to configure your web server environment. You may adjust the resource specifications as needed, except for the recommended OS (Ubuntu 20.04).

Step 2. SSH into the Web Service

Access the web server using ssh. If a public IP is not configured, use a Bastion host or similar method to connect. Repeat this process for all provisioned instances.

# web-1
ssh -i ${PRIVATE_KEY} ubuntu@${WEB1_ENDPOINT}

# web-2
ssh -i ${PRIVATE_KEY} ubuntu@${WEB2_ENDPOINT}

Step 3. Write Object Storage Environment Setup Script

Install the jq package to handle JSON data in the shell.

sudo apt-get update -y
sudo apt-get install -y jq

After accessing the instance, create an environment variable file for log storage.

cat << \EOF | sudo tee /tmp/env.sh
#!/bin/bash

export ACCESS_KEY="${ACCESS_KEY}"
export ACCESS_SECRET_KEY="${ACCESS_SECRET_KEY}"

# Don't Edit
export BUCKET_NAME="hands-on"
export FILE="/var/log/nginx/access.log"
export TOPATH="/log/nginx/date_id=$(date +%Y-%m-%d)/host_id=$(hostname -s)/access.log"
EOF

Generate the API Authentication Token and Project ID.

cat << \EOF | sudo tee /tmp/token.sh
#!/bin/bash

export TOKEN=$(curl -s -X POST -i https://iam.kakaocloud.com/identity/v3/auth/tokens -H "Content-Type: application/json" -d '{
    "auth": {
        "identity": {
            "methods": [
                "application_credential"
            ],
            "application_credential": {
                "id": "'${ACCESS_KEY}'",
                "secret": "'${ACCESS_SECRET_KEY}'"
            }
        }
    }
}' | grep x-subject-token | awk -v RS='\r\n' '{print $2}')

export PROJECT_ID=$(curl -s -X POST https://iam.kakaocloud.com/identity/v3/auth/tokens -H "Content-Type: application/json" -d '{
    "auth": {
        "identity": {
            "methods": [
                "application_credential"
            ],
            "application_credential": {
                "id": "'${ACCESS_KEY}'",
                "secret": "'${ACCESS_SECRET_KEY}'"
            }
        }
    }
}' | jq -r ".token.project.id")

if [ -z $TOKEN ]; then
        echo "TOKEN is null..."
        exit 0
fi

if [ -z $PROJECT_ID ]; then
        echo "PROJECT_ID is null..."
        exit 0
fi
EOF

Create an Object Storage bucket using the environment and token scripts.

cat << 'EOF' | sudo tee /tmp/init.sh
#!/bin/bash

. /tmp/env.sh
. /tmp/token.sh

function create_bucket() {
    echo "Creating bucket: ${BUCKET_NAME}..."

    if [ -z "$TOKEN" ]; then
        echo "ERROR: TOKEN is not set!"
        exit 1
    fi

    RESPONSE=$(curl -s -X PUT "https://objectstorage.kr-central-2.kakaocloud.com/v1_ext/bucket"         -H "X-Auth-Token: ${TOKEN}"         -H "Content-Type: application/json"         -d "{
            \"name\": \"${BUCKET_NAME}\",
            \"type\": \"STANDARD\",
            \"use_encryption\": true
        }")

    echo "API Response: $RESPONSE"

    if echo "$RESPONSE" | grep -q "Error"; then
        echo "ERROR: Bucket creation failed!"
        exit 1
    else
        echo "Bucket created successfully: ${BUCKET_NAME}"
    fi
}

create_bucket
EOF
bash /tmp/init.sh

Write a script to upload log files using the defined environment variables.

cat << \EOF | sudo tee /tmp/upload.sh
#!/bin/bash

. /tmp/env.sh
. /tmp/token.sh
. /tmp/preupload.sh

function upload_file() {
        echo "upload file: ${FILE}"
        curl -s -X PUT 'https://objectstorage.kr-central-2.kakaocloud.com/v1/'${PROJECT_ID}'/'${BUCKET_NAME}''${TOPATH}''         -H 'X-Auth-Token: '${TOKEN}''         -H 'Content-Type: application/octet-stream'         -T ${FILE}
        echo "upload end: '${BUCKET_NAME}' '${TOPATH}'"
}

upload_file
. /tmp/postupload.sh
EOF

Create scripts to execute before and after uploading the logs.

cat << EOF | sudo tee /tmp/preupload.sh
#!/bin/bash

EOF

cat << EOF | sudo tee /tmp/postupload.sh
#!/bin/bash

sudo mv /var/log/nginx/access.log /var/log/nginx/access.log.backup
sudo kill -USR1 `cat /var/run/nginx.pid`
EOF

Step 4. Set up Automatic Uploading

Install the cron package to schedule the log uploads.
```
sudo apt update -y
sudo apt install -y cron
```

Configure crontab to run the upload script at midnight.

sudo rm /etc/logrotate.d/nginx
cat << EOF > tmp_crontab
0 0 * * * /bin/bash /tmp/upload.sh
EOF
sudo crontab tmp_crontab
rm tmp_crontab

Verify the cron job and run the script manually.
```
sudo crontab -l
bash /tmp/upload.sh
```
Confirm the logs are saved in Object Storage via the console.

Introduction​

Getting started​

Step 1. Set Up VM-based Web Service​

Step 2. SSH into the Web Service​

Step 3. Write Object Storage Environment Setup Script​

Step 4. Set up Automatic Uploading​