Troubleshoot Advanced Managed Search

This document describes common Advanced Managed Search issues and how to resolve them.

• 1. Cluster remains in Provisioning for a long time
  • Symptom
  • Diagnosis
  • Resolution
• 2. Cluster remains in Scaling Up with a detailed status of Relocating Nodes for a long time
  • Symptom
  • Background: Normal behavior
  • 2-1. Relocation takes a long time because the cluster contains a large amount of data
  • 2-2. Data relocation stops during scale-up

---

info

The provisioningStatus and detailedStatus values in this document are based on API responses. They differ from the names displayed in the console UI as follows:

• Spaces added: ScalingUp → Scaling Up, RelocatingNodes → Relocating Nodes
• Spaces added and name expanded: ProvisioningErr → Provisioning Error

For a complete mapping of status values to display names, see Cluster lifecycle and status.

1. Cluster remains in Provisioning for a long time

Symptom

The cluster provisioningStatus does not transition from Provisioning to Initializing or Running.

Diagnosis

Check cluster status

provisioningStatus: Provisioning

Check node status

Check the status of each OpenSearch data node and cluster manager node in the cluster.

Status	Description
Provisioning	OpenStack resources are being created (normal waiting state)
Initializing	OpenSearch is being installed and configured (normal progress)
ProvisioningErr	Provisioning failed (cannot be recovered automatically)

Resolution

If any node has a status of ProvisioningErr, the operation is not retried automatically. Contact the Service Desk.

Example cause of ProvisioningErr

• An error occurred while node resources were being created.

info

When the status is ProvisioningErr, the cluster remains in Provisioning. Unlike scale-up and scale-out operations, an automatic rollback is not performed.

---

2. Cluster remains in Scaling Up with a detailed status of Relocating Nodes for a long time

Symptom

provisioningStatus: ScalingUp
detailedStatus: RelocatingNodes

Background: Normal behavior

During a scale-up operation, data is moved from existing nodes to new nodes. The operation can take several hours or longer depending on the amount of data stored in the cluster.
A prolonged RelocatingNodes status does not necessarily indicate an error. Use the following cases to determine whether the operation is progressing normally or requires assistance.

2-1. Relocation takes a long time because the cluster contains a large amount of data

Criteria

• The cluster contains many indexes or a large amount of data.
• Some time has passed since the scale-up operation began, but data is still moving slowly.

Check whether the operation is progressing normally

If the amount of data on the existing nodes is gradually decreasing as it moves to the new nodes, the operation is likely progressing normally. You can use the following APIs in Dev Tools to check.

Examples using the OpenSearch API

Check the overall relocation status:

GET /_cluster/health?pretty

• A relocating_shards value greater than 0 indicates that shards are being relocated.
• If initializing_shards is also displayed, shards might be initializing on the new nodes.

Check the allocation and relocation status of each shard:

GET /_cat/shards?v

• If shards have a state of RELOCATING or INITIALIZING, relocation is in progress.
• If the number of shards on the existing nodes decreases over time, the operation is progressing normally.

Check active recoveries:

GET /_cat/recovery?v&active_only=true

• If recovery entries are displayed and values such as bytes_percent continue to change, the operation is likely progressing, even if slowly.

Increase relocation speed (optional)

If relocation is progressing but is too slow, you can temporarily adjust the following cluster settings in Dev Tools.

PUT /_cluster/settings
{
  "transient": {
    "cluster.routing.allocation.node_concurrent_recoveries": "4",
    "cluster.routing.allocation.cluster_concurrent_rebalance": "4",
    "indices.recovery.max_bytes_per_sec": "200mb"
  }
}

Setting	Description
cluster.routing.allocation.node_concurrent_recoveries	Number of concurrent recoveries per node (default: 2)
cluster.routing.allocation.cluster_concurrent_rebalance	Number of concurrent rebalances across the cluster (default: 2)
indices.recovery.max_bytes_per_sec	Maximum recovery transfer rate (default: 40mb)

After applying the settings, use GET /_cat/recovery?v&active_only=true to check whether the transfer speed has changed.

We recommend resetting the settings after the scale-up operation is complete.

PUT /_cluster/settings
{
  "transient": {
    "cluster.routing.allocation.node_concurrent_recoveries": null,
    "cluster.routing.allocation.cluster_concurrent_rebalance": null,
    "indices.recovery.max_bytes_per_sec": null
  }
}

caution

• Administrator privileges are required to change these settings, and you might need to adjust the values based on cluster load.
• During a scale-up operation, do not change other cluster settings such as allocation exclusions. Doing so can cause data loss.
• These settings alone might not resolve the issue described in case 2-2 (relocation stalled). If relocation has stopped, contact the Service Desk.

Resolution

• If the operation is progressing normally, wait for it to complete.
• If it takes significantly longer than expected or you need help determining whether it is progressing, contact the Service Desk.

2-2. Data relocation stops during scale-up

Criteria

• The RelocatingNodes status persists for a long time, but no data is being moved.
• Data appears to remain on existing nodes after the scale-up operation, and the status does not advance to the next stage.

Examples using the OpenSearch API

Check whether any shards are being relocated:

GET /_cluster/health?pretty

• If both relocating_shards and initializing_shards remain at 0 for a long time, relocation might not be progressing.

Check whether shards remain on the existing nodes:

GET /_cat/shards?v

• If shards remain assigned to existing (legacy) node names and few or no shards have the RELOCATING state, relocation might have stalled.
• If a primary shard (p) for the same index and shard remains on an existing node while its replica (r) is on a new node, relocation might be blocked.

Check whether any recoveries are active:

GET /_cat/recovery?v&active_only=true

• If the result is empty or progress does not change for a long time, relocation might have stopped.

Check why a specific shard cannot be relocated. Replace the index and shard values with actual values.

GET /_cluster/allocation/explain
{
  "index": "my-index",
  "shard": 0,
  "primary": true
}

• The explanation field describes why the shard cannot be relocated.

Cause 1: Replica configuration does not match the number of data nodes

If the cluster has one data node and an index has number_of_replicas set to 1 or higher, the replica shard can never be assigned. The cluster remains in the yellow state, and relocation might not advance to the next stage.

Resolution

• Set the index's number_of_replicas value to match the number of data nodes. This can allow relocation to resume.

PUT /<index-name>/_settings
{
  "index": {
    "number_of_replicas": 0
  }
}

Cause 2: Other data allocation mismatch (cause cannot be identified)

If cause 1 does not apply and relocation has stalled, a data allocation mismatch between the existing and new nodes might be delaying or blocking relocation.

Resolution

• Contact the Service Desk.
• Do not change cluster settings or manually relocate data. Doing so can cause data loss.