Skip to main content

Create and manage crawler

Create crawler​

A crawler in Data Catalog scans MySQL data, extracts metadata, and automatically updates Data Catalog to simplify data discovery.
To create a crawler, ensure that both the database and MySQL are already created. Follow these steps to create a crawler in the Data Catalog service.

info

For guidance on creating databases and MySQL, refer to the following guides:
How to create a database / How to create MySQL

  1. Go to the KakaoCloud Console > Analytics > Data Catalog.

  2. In the Crawler menu, click the [Create crawler] button.

  3. Enter the required information in Create crawler and click [Create].

    FieldDescription
    DatabaseName of the database to which the table will be added
    - Only databases with an Active status are shown in the list.
    - When selecting a database, network/subnet information can be checked.
    Crawler nameName of the crawler
    - Must consist of lowercase letters, numbers, or underscores (4–64 characters)
    MySQL full pathSelect the MySQL instance and enter the database name of the MySQL instance
    - Only MySQL instances in Available status are displayed in the list.
    MySQL accountEnter the username and password set during MySQL creation
    - Connection test: After entering the MySQL full path and account information, test the connection using the [Test connection] button.
    * If the connection test fails, you cannot create the crawler.
    Description (optional)Add additional details about the crawler
    Table prefix (optional)Prefix for the table name; creates tables in the format of Prefix+MySQL database name_table name
    - Must consist of lowercase letters, numbers, or underscores (1–64 characters)
    ScheduleManage when the crawler runs
    - Choose from on-demand, hourly, daily, weekly, or monthly schedules.
    - On-demand means the crawler runs only when manually triggered.
info

Only databases and MySQL instances in Active/Available status will be displayed in the list.

Manage crawler​

This section explains how to manage crawlers in the Data Catalog service.

View crawler list​

You can view the list of active crawlers in Data Catalog.

  1. Go to the KakaoCloud Console > Analytics > Data Catalog.

  2. Click the Crawler menu to view the crawler list.

    ColumnDescription
    NameName of the crawler entered during creation
    - You can click on the crawler name to access the details tab.
    DescriptionDescription entered during creation
    StatusStatus of the crawler
    ScheduleScheduled time for crawler execution
    Last run statusStatus of the most recent crawler run
    Last run dateDate and time when the last crawler run occurred
    [More] icon- Modify: Update the crawler's description or schedule.
    - Run: Manually trigger the crawler.
    - Delete: Remove the crawler.
    * If the crawler is in the CREATING/ALTERING/DELETING/RUNNING status, modification, execution, and deletion are not allowed.

View crawler details​

You can check detailed information about a crawler.

  1. Go to the KakaoCloud Console > Analytics > Data Catalog.
  2. Click the Crawler menu and select the crawler to check.
  3. View the detailed information for the selected crawler.

View crawler run history​

You can view the execution history of a crawler.

  1. Go to the KakaoCloud Console > Analytics > Data Catalog.

  2. Click the Crawler menu and select the crawler to view the run history.

  3. In the detailed view, click the Run history tab to check the execution history.

    info

    Only the run history within the last 90 days is displayed. Execution history older than 90 days is automatically deleted.

    FieldDescription
    Start timeTime the crawling started
    End timeTime the crawling ended
    Execution timeDuration of the crawler run
    StatusStatus of the crawler run
    - Succeeded: Crawling finished successfully.
    - Running: Crawling is in progress.
    - Failed: Crawling failed.

Delete crawler​

You can delete crawlers that are no longer in use.

caution

Deleted crawlers and their run histories cannot be recovered. If a catalog is deleted, the associated crawler will also be removed.

  1. Go to the KakaoCloud Console > Analytics > Data Catalog.
  2. Click the Crawler menu, then click the [More] icon for the crawler to delete, and select Delete.
  3. In the popup, enter the crawler name exactly as shown and click [Delete].