Synchronize users and keys across Azure Cloud HSM nodes

This article explains how to identify and resolve synchronization issues when users or keys are missing from one or more nodes in your Azure Cloud HSM cluster.

Overview

In some cases, users or keys might not be replicated to all nodes in your Azure Cloud HSM cluster. This can occur during service events such as self-healing or upgrades, or if creation fails on one or more nodes. If you're experiencing intermittent authentication failures or cryptographic operation errors, you might have users or keys that need to be synchronized.

Prerequisites

• Access to a VM with the Azure Cloud HSM SDK installed
• The azcloudhsm_mgmt_util tool
• Cryptography officer (CO) credentials for your Azure Cloud HSM deployment

User synchronization

All users in Azure Cloud HSM are fully managed by the customer. The service doesn't perform backend user synchronization if user creation fails. If user creation fails on one or more nodes, you must manually synchronize the user to the missing nodes.

Identify missing users

1. Start the management utility:

   ./azcloudhsm_mgmt_util ./azcloudhsm_resource.cfg
   

2. Run the listUsers command to display the User ID, User Type, and Username under each node (server 0, 1, 2):

   listUsers
   

3. Observe the number of users found for each node and corresponding usernames. Compare the lists across all three servers to identify any missing users.

Synchronize missing users

1. Sign in as a cryptography officer (CO):

   loginHSM CO admin <adminPassword>
   

   Verify that you successfully signed in to all three nodes:

   loginHSM success on server 0
   loginHSM success on server 1
   loginHSM success on server 2
   

   Note

   If sign-in fails on any node, the sync operation might fail. Ensure successful sign-in to all nodes before proceeding.

2. Identify the source node that has the user. In this example, server 0 has a user that's not available on server 1 and server 2:

   server 0
   

3. Run the syncUser command for each server where the user is missing. Replace <UserID> with the actual User ID:

   syncUser <UserID> 1
   syncUser <UserID> 2
   

   Note

   If syncUser is executed against a node where the user already exists, the error message "user already created, unable to insert object" appears. If the user doesn't exist, the operation succeeds.

Validate user synchronization

1. Exit the current server context:

   exit
   

2. Run listUsers to confirm that all User IDs, User Types, and Usernames are now equal and available under each node (server 0, 1, 2):

   listUsers
   

Key synchronization

When you create keys, it's your responsibility to ensure keys are present on all nodes. Although Azure Cloud HSM supports service-side key synchronization and restore operations, you must verify that keys are available on any missing nodes before use.

Identify missing keys

1. Start the management utility:

   ./azcloudhsm_mgmt_util ./azcloudhsm_resource.cfg
   

2. Sign in as a cryptography officer (CO):

   loginHSM CO admin <adminPassword>
   

   Verify that you successfully signed in to all three nodes:

   loginHSM success on server 0
   loginHSM success on server 1
   loginHSM success on server 2
   

3. Run the findAllKeys command to display the number of keys and key handle IDs under each node:

   findAllKeys 0 0
   

4. Observe the number of keys found for each node and corresponding key handle IDs. Compare the results across all three servers to identify any missing keys.

Synchronize missing keys

1. Identify the source node that has the key. In this example, server 0 has a key handle that's not available on server 1 and server 2:

   server 0
   

2. Run the syncKey command for each server where the key is missing. Replace <KeyHandle> with the actual key handle ID:

   syncKey <KeyHandle> 1
   syncKey <KeyHandle> 2
   

   For example, to synchronize key handle 262150 to servers 1 and 2:

   syncKey 262150 1
   syncKey 262150 2
   

Validate key synchronization

1. Exit the current server context:

   exit
   

2. Run findAllKeys 0 0 to confirm that all key handles and the number of keys found are now equal and available under each node (server 0, 1, 2):

   findAllKeys 0 0
   

Best practices

To prevent synchronization issues and potential data loss:

• Verify after creation: After creating any user or key, immediately verify that it exists on all three nodes.
• Maintain regular backups: Use the backup and restore functionality to protect against node failures.
• Monitor for discrepancies: Periodically run listUsers and findAllKeys 0 0 to check for inconsistencies across nodes.
• Act quickly on failures: If you notice a user or key creation failure, synchronize it to the missing nodes before a node failure occurs.

Related content

• Troubleshoot Azure Cloud HSM
• User management in Azure Cloud HSM
• Key management in Azure Cloud HSM
• Backup and restore in Azure Cloud HSM