Create and manage table

Create table

A table is metadata dependent on a database. To create a table, a catalog and database must be created first. The database must be in Active or Altering status to create a table in it. Here’s how to create a table in the Data Catalog service.

1. Go to the KakaoCloud Console > Analytics > Data Catalog.

2. Click the [Create table] button in the Table menu.

3. Enter the following information and click [Create].

   Field	Description
   Database	Name of the database to which the table will be added - When selecting a database, network/subnet information can be checked
   Table name	Name of the table - Duplicate MySQL instance group names cannot be used within the same project
   Data storage path	Enter the path to the object storage where the table will be created (the table path will reference the database path) - S3 connection: If selected, the full path is set to s3a://. If not selected, it defaults to swifta - Bucket name: Enter the object storage bucket name - Directory: Enter the path to the directory under the bucket
   Data type	Specify the data type of the table - Supported types: Avro, JSON, Parquet, ORC, CSV
   Pub/Sub integration	Enable Pub/Sub integration - If enabled, select the Pub/Sub topic to use - For more details on creating Pub/Sub topics, refer to Create Pub/Sub topics
   Description (optional)	Additional description of the table
   Schema	Click [Add field] to add fields - Up to 50 fields can be added - At least one non-partition key field must exist to create a schema - Field name: Name of the field   ᄂ For avro type, field names cannot start with numbers - Data type: Field's data type - Partition key: Whether it is a partition key, only configurable when creating a table - Comment: Field comment, up to 130 characters, emojis are not allowed - [More] icon: Edit or delete fields

info

A schema must have at least one non-partition key field. For more details on how to create a schema, refer to Adding schema fields.

Manage table

Here’s how to manage tables in the Data Catalog service.

View table list

You can view the list of active tables in Data Catalog.

1. Go to the KakaoCloud Console > Analytics > Data Catalog.

2. Click the Table menu to view the list of tables.

   Field	Description
   Table name	Name of the table entered during creation - Click the table name to go to the detail tab
   Associated database	Name of the database to which the table is connected
   Status	Status of the table
   Data storage path	Path where the table data is stored, as entered during creation
   Data type	Data type of the table
   Last modified	Date and time of the last table modification
   [More] icon	- View full schema: View the full schema of the table - Edit table: Edit the table name and description - Delete table: Delete the table * Only single deletion is allowed for tables in INACTIVE status, and table editing is not available

View table details

You can view detailed information about a table.

1. Go to the KakaoCloud Console > Analytics > Data Catalog.
2. In the Table menu, select the table to check from the list.
3. View the detailed information for the selected table.

Edit table

You can edit the information of a created table.

Notice

The scope of modifiable fields varies depending on the table type.

1. Go to the KakaoCloud Console > Analytics > Data Catalog.

2. In the Table menu, click the [More] icon next to the table to edit, then select Edit.

3. In the popup, check the areas that can be edited.

   Field	Description
   Table name	Change the table name - Table names cannot be changed for MySQL-type tables
   Pub/Sub integration	Choose whether to enable or disable Pub/Sub integration - Pub/Sub integration is not available for MySQL-type tables - If the Pub/Sub topic used for integration is deleted, the integration remains active, but a new topic must be selected for proper functionality
   Description	Edit the table description

4. After making the necessary changes, click [Save].

Delete table

You can delete tables that are no longer in use.

caution

Deleted tables and data cannot be recovered.

1. Go to the KakaoCloud Console > Analytics > Data Catalog.
2. In the Table menu, click the [More] icon next to the table to delete, then select Delete.
3. In the popup, enter the name of the table to be deleted and click [Delete].

Pub/Sub integration

This section explains how to integrate a Pub/Sub topic with a Data Catalog table.

Integrate with Pub/Sub topic

info

To integrate a Pub/Sub topic with a table, the Pub/Sub topic must already be created.
For more details on creating a topic, refer to Create Pub/Sub topics.

Here’s how to integrate a Pub/Sub topic with a Data Catalog table.

1. When creating a table, enable the Pub/Sub integration option.

2. Select the desired topic from the list of available topics.

3. If the table has already been created, enable Pub/Sub integration through the Edit option.

4. Select the desired topic from the list of available topics.

Manage Pub/Sub events

When the table information is changed, an event is published to the specified Pub/Sub topic.
You can check the published events in the Pub/Sub Manage Subscriptions menu.

The types and categories of generated events are as follows.

Type	Description	Categories
Table events	Generated by creating, modifying, or deleting tables in Data Catalog	- CreateTableEvent - AlterTableEvent - DropTableEvent
Partition events	Generated when partition information of a table is changed through Hadoop Eco integration	- AddPartitionEvent - AlterPartitionEvent - DropPartitionEvent

Manage table data

This section explains how to manage table data in the Data Catalog service in KakaoCloud Console. You can manage Serde parameters, table properties, schema, and partitions.

info

For tables created through a crawler (MySQL-type), only schema data is displayed.

View Serde parameters

Here’s how to view Serde parameters.

1. Go to the KakaoCloud Console > Analytics > Data Catalog.

2. In the Table menu, select the table whose Serde parameters to view from the list.

3. Click the Serde Parameters tab.

   Field	Description
   key	Key of the Serde parameter
   Value	Value of the Serde parameter
   [More] icon	- Edit Serde parameter * As of January 2023, deleting parameters is not supported

Add Serde parameters

Here’s how to add new Serde parameters.

1. Go to the KakaoCloud Console > Analytics > Data Catalog.

2. In the Table menu, select the table to which to add Serde parameters.

3. In the Serde Parameters tab, click the [Add Serde Parameter] button.

4. In the popup, enter the parameters and click [Add].

   Field	Description
   key	Key of the Serde parameter
   Value	Value of the Serde parameter - Emojis are not allowed

Edit Serde parameters

Here’s how to edit Serde parameters.

1. Go to the KakaoCloud Console > Analytics > Data Catalog.
2. Click the Table tab, then select the table whose Serde parameters to edit.
3. In the Serde Parameters tab, click the [More] icon and select Edit Serde Parameter.
4. In the popup, modify the value and click [Save].

View table properties

You can view table properties from the list of tables in Data Catalog.

1. Go to the KakaoCloud Console > Analytics > Data Catalog.

2. In the Table menu, select the table whose properties to view.

3. In the Table Properties tab, check the table properties.

   Field	Description
   key	Key of the table property
   Value	Value of the table property
   [More] icon	- Edit table property - Delete table property

Add table properties

Here’s how to add new table properties.

1. Go to the KakaoCloud Console > Analytics > Data Catalog.

2. In the Table menu, select the table to which to add properties.

3. In the Table properties tab, click the [Add table property] button.

4. In the popup, enter the property and click [Add].

   Field	Description
   key	Key of the table property
   Value	Value of the table property - Emojis are not allowed

Edit table properties

Here’s how to edit table properties.

1. Go to the KakaoCloud Console > Analytics > Data Catalog.
2. In the Table menu, select the table to which to add properties.
3. In the Table Properties tab, click the [More] icon next to the property to edit, and select Edit table property.
4. In the popup, modify the property and click [Save].

Delete table properties

You can delete table properties from the detailed view of a table in Data Catalog.

caution

Deleted data cannot be recovered.

1. Go to the KakaoCloud Console > Analytics > Data Catalog.
2. In the Table menu, select the table whose properties to delete.
3. In the Table Properties tab, click the [More] icon next to the property to delete, and select Delete table property.
4. In the popup, click [Delete].

View schema fields

You can view schema fields from the detailed view of a table in Data Catalog.

1. Go to the KakaoCloud Console > Analytics > Data Catalog.

2. In the Table menu, select the table whose schema fields to view.

3. In the Schema tab, view the schema fields.

   Field	Description
   No.	Column number for sorting partition fields - The initial column number starts at 1 - Column numbers can be set between 1 and the last number plus 1 * Not entered for partition keys
   Field name	Name of the schema field
   Data type	Data type of the field If the schema is viewable, click to check - Supported types: array, bigint, binary, boolean, char, date, decimal, double, float, int, map, smallint, string, struct, timestamp, tinyint, union, varchar
   Partition key	Whether the field is designated as a partition key during table creation
   Comment	Description of the schema
   [More] icon	- Modify filed: Opens the field edit popup - Delete field: Opens the field delete popup

Add schema fields

You can add schema fields in the detailed view of a table in Data Catalog. You can add up to 50 fields.

info

For tables with data types JSON, CSV, or MySQL, fields cannot be added, edited, or deleted after table creation.

1. Go to the KakaoCloud Console > Analytics > Data Catalog.

2. In the Table menu, select the table whose schema fields to add.

3. In the Schema tab, click the [Add field] button.

4. In the popup, enter the required information and click [Add].

   Field	Description
   Column number	Number for sorting partition fields - The initial column number starts at 1 - Column numbers can be set between 1 and the last number plus 1 * Not entered for partition keys
   Field name	Enter the field name
   Data type	Data type of the field - Supported types: array, bigint, binary, boolean, char, date, decimal, double, float, int, map, smallint, string, struct, timestamp, tinyint, union, varchar
   Comment (optional)	Enter a description for the field

Modify schema fields

You can edit schema fields in the detailed view of a table in Data Catalog.

1. Go to the KakaoCloud Console > Analytics > Data Catalog.
2. In the Table menu, select the table whose schema fields to edit.
3. In the Schema tab, click the [Modify] icon next to the schema field to edit, and select Modify filed.
4. In the popup, modify the information and click [Save].

Delete schema fields

You can delete schema fields from the detailed view of a table in Data Catalog.

caution

Deleted data cannot be recovered.

1. Go to the KakaoCloud Console > Analytics > Data Catalog.
2. In the Table menu, select the table whose schema fields to delete.
3. In the Schema tab, click the [More] icon next to the schema field to delete, and select Delete field.
4. In the popup, click [Delete].

View partitions

You can view partitions in the detailed view of a table in Data Catalog.

1. Go to the KakaoCloud Console > Analytics > Data Catalog.

2. In the Table menu, select the table whose partitions to view.

3. In the Partitions tab, view the partition information.

   Field	Description
   id	Partition key information created by the user
   Files	Linked files for the partition
   Properties	Schema of the partition