Wednesday 14 July 2021

Live-Migrating a Hyper-V VM with TPM on Server Core

As part of our testing environment, we set up a Windows 11 Insider build in a VM and as per requirement, we enabled the virtual Trusted Platform Module (TPM) for that VM. I soon noticed that Cluster Aware Updating (CUA) stopped working on the failover cluster that hosted the VM.


More specifically, it turned out that the node that hosted the VM with the vTPM, could not be drained because the VM role could not be live-migrated to the other node.


Live migration of 'Virtual Machine Insider11' failed.

Virtual machine migration operation for 'Insider11' failed at migration destination 'HYPER-V25'. (Virtual machine ID E567C1C9-B323-4AED-B055-F9DCF98D0853)

The version of the device 'Microsoft Virtual TPM Device' of the virtual machine 'Insider11' is not compatible with device on physical computer 'HYPER-V25'. (Virtual machine ID E567C1C9-B323-4AED-B055-F9DCF98D0853)

The key protector for the virtual machine '' could not be unwrapped. HostGuardianService returned: One or more arguments are invalid (0x80070057) . Details are included in the HostGuardianService-Client event log. (Virtual machine ID )

The vTPM prevented live migration of the virtual machine. The solution lies in exporting the required certificates from the node's certificate store and importing them on the other node.

The issue I ran into with that solution was that on the free Hyper-V Server product, the certificate management console does not exist.


The solution is to do it in PowerShell. The certificates that need to be exported are in the local machine's certificate store in a folder called Shielded VM Local Certificates.

PS C:\> dir "cert:\LocalMachine\Shielded VM Local Certificates"


   PSParentPath: Microsoft.PowerShell.Security\Certificate::LocalMachine\Shielded VM Local Certificates

Thumbprint                                Subject
----------                                -------
A068364B6618C532067D93B3752ABEA4C86CF50D  CN=Shielded VM Encryption Certificate (UntrustedGuardian) (Hyper-V24)
883480C7627A4D63EC3E56E4F9A82A9F1EB1C4EB  CN=Shielded VM Signing Certificate (UntrustedGuardian) (Hyper-V24)


PS C:\>

I stored the certificates in a variable

PS C:\> $cert1 = Get-ChildItem -Path "cert:\LocalMachine\Shielded VM Local Certificates\A068364B6618C532067D93B3752ABEA4C86CF50D"
PS C:\> $cert2 = Get-ChildItem -Path "cert:\LocalMachine\Shielded VM Local Certificates\883480C7627A4D63EC3E56E4F9A82A9F1EB1C4EB"
PS C:\>

Next, I chose a password for the .pfx file,

PS C:\> $mypwd = ConvertTo-SecureString -String "1234" -Force -AsPlainText
PS C:\> 

 exported the first certificate and repeated the process for the second certificate.

PS C:\> $cert1 | Export-PfxCertificate -FilePath C:\cert1.pfx -Password $mypwd

    Directory: C:\

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----        7/14/2021   4:37 PM           2599 cert1.pfx

PS C:\>

Instead of using a password, I could protect the file by using -ProtectTo instead and set a User or group that is allowed to access the private key but I will not cover this possibility here.

Of course I could do this in one step by piping the output from Get-ChildItem into Export-PfxCertificate.

PS C:\> Get-ChildItem -Path "cert:\LocalMachine\Shielded VM Local Certificates\A068364B6618C532067D93B3752ABEA4C86CF50D" | Export-PfxCertificate -FilePath C:\cert1.pfx -Password $mypwd

    Directory: C:\

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----        7/14/2021   4:43 PM           2599 cert1.pfx

PS C:\>

The certificates were now exported as .pfx files on my c: drive and were moved to the other node.

PS C:\> dir c:\ *.pfx

    Directory: C:\

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----        7/14/2021   4:37 PM           2599 cert1.pfx
-a----        7/14/2021   4:37 PM           2599 cert2.pfx

PS C:\> 

On the other node, that is, on all nodes the VM potentially needs to be live-migrated to, I imported the certificate. This also needed to be done in PS as the certificate MMC was missing. The folder Shielded VM Local Certificates was also missing.

PS C:\> dir "cert:\LocalMachine\Shielded VM Local Certificates"
dir : Cannot find path '\LocalMachine\Shielded VM Local Certificates' because it does not exist.
At line:1 char:1
+ dir "cert:\LocalMachine\Shielded VM Local Certificates"
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (\LocalMachine\S...al Certificates:String) [Get-ChildItem], ItemNotFound
   Exception
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetChildItemCommand
PS C:\>

The folder did not exist, presumably because there never were any VMs with vTPM on that node. I needed to create the folder first.

PS C:\> mkdir "cert:\LocalMachine\Shielded VM Local Certificates"

Name : Shielded VM Local Certificates


PS C:\>

I was able to confirm that the folder existed and was empty.

PS C:\> dir "cert:\LocalMachine\Shielded VM Local Certificates" 
PS C:\> 

It was time to import the certificates from the other node. I needed the password that I had set before.

PS C:\> $mypwd = ConvertTo-SecureString -String "1234" -Force -AsPlainText
PS C:\> 

The import once again needed to be done for both certificates.

PS C:\> Import-PfxCertificate -FilePath D:\temp\cert1.pfx -CertStoreLocation "cert:\LocalMachine\Shielded VM Local Certificates" -Password $mypwd -Exportable


   PSParentPath: Microsoft.PowerShell.Security\Certificate::LocalMachine\Shielded VM Local Certificates

Thumbprint                                Subject
----------                                -------
A068364B6618C532067D93B3752ABEA4C86CF50D  CN=Shielded VM Encryption Certificate (UntrustedGuardian) (Hyper-V24)


PS C:\>

Note, that I chose to make the private key exportable by using the argument "-Exportable". This is not a requirement however.

Both certificates were imported. Note that the subject name contains the hostname of the node that hosts the VM with TPM

PS C:\> dir "cert:\LocalMachine\Shielded VM Local Certificates"

   PSParentPath: Microsoft.PowerShell.Security\Certificate::LocalMachine\Shielded VM Local Certificates
Thumbprint                                Subject
----------                                -------
A068364B6618C532067D93B3752ABEA4C86CF50D  CN=Shielded VM Encryption Certificate (UntrustedGuardian) (Hyper-V24)
883480C7627A4D63EC3E56E4F9A82A9F1EB1C4EB  CN=Shielded VM Signing Certificate (UntrustedGuardian) (Hyper-V24)

PS C:\> 

 That was it. The VM could now be live-migrated to the other node.

Sources:
Can I create a new folder/directory under Windows Certificates and import all my self signed CA certificates in it - Stack Overflow
How to manage certificate private keys on server 2016 Core : sysadmin (reddit.com)
Export-PfxCertificate (pki) | Microsoft Docs
Import-PfxCertificate (pki) | Microsoft Docs