Hi all
I’ve been experiencing a strange issue with SCVMM and after countless attempts at fixing it, I'm coming here asking for help.
I’ve recently deployed an install of SCVMM current branch 1807 in our environment (currently upgraded to 2019) and I’ve been attempting to configure a logical switch on a new temporary hypervisor based on Server 2016. Having
encountered loads of SCVMM service crashes, I found that a host agent update was pending so I went ahead and installed it for that host which indeed allowed me to configure the logical switch and the virtual network adapter for management. Thinking all is
well, I kicked off our migration script to continue migrating from our old ESXi environment however as soon as the V2V operation in SCVMM starts, the host agent service crashes with the following error:
Faulting application name: vmmAgent.exe, version: 4.1.3403.0, time stamp: 0x5b33b02b
Faulting module name: ntdll.dll, version: 10.0.14393.2608, time stamp: 0x5bd133d4
Exception code: 0xc0000374
Fault offset: 0x00000000000f7b43
Faulting process id: 0x16cc
Faulting application start time: 0x01d4bed3ed6bffc4
Faulting application path: C:\Program Files\Microsoft System Center\Virtual Machine Manager\bin\vmmAgent.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 3ebd0a7c-46e8-4223-9249-0f8fad13103f
Faulting package full name:
Faulting package-relative application ID:
This didn’t happen before and I’ve been able to migrate a lot of VMs before, it then suddenly became an issue affecting all hypervisors. I initially thought it was affecting the new hypervisor but then realised that it is the same
case in relation to the old hypervisor as well. In fact, it happens at any migration attempt. I’ve not been able to find much about this issue online, the error code is very specific. I’ve seen hints relating to bits and the bits ssl port but none of those
tweaks help here.
If nobody can shed any light at this, I believe the only way will be to raise a support case.
Thanks for any ideas
The SCVMM cmdlet returns (I've hidden some environment specific names):
13-02-2019 14:08:26 vmname (3) Microsoft.VirtualManager.Utils.CarmineException: An internal error has occurred
trying to contact the 'hv09.domain.local' server: : .
WinRM: URL: [http://hv09.domain.local:5985], Verb: [INVOKE], Method: [CreateHttpPostClientJob], Resource: [http://sc
hemas.microsoft.com/wbem/wsman/1/wmi/root/scvmm/HttpPostDeploymentJob]
Check that WS-Management service is installed and running on server 'hv09.domain.local'. For more information use th
e command "winrm helpmsg hresult". If 'hv09.domain.local' is a host/library/update server or a PXE server role then
ensure that VMM agent is installed and running. Refer to http://support.microsoft.com/kb/2742275 for more details.
at Microsoft.SystemCenter.VirtualMachineManager.TaskHelper.ThrowOnFailure()
at Microsoft.SystemCenter.VirtualMachineManager.Cmdlets.TaskCmdlet`1.WaitForResult(Task task)
at Microsoft.SystemCenter.VirtualMachineManager.Cmdlets.TaskCmdletBase`1.Process()
at Microsoft.SystemCenter.VirtualMachineManager.Cmdlets.CmdletBase.ProcessRecord()
*** Carmine error was: HostAgentFail (2912); HR: 0x800706BE
*** xx-esxi-10.domain.local ** 443 ** /folder/vmname /vmname.vmdk?dsName=xx-esxi-10:localdisk1:10kS
AS ** WinRM: URL: [http://hv09.domain.local:5985], Verb: [INVOKE], Method: [CreateHttpPostClientJob], Resource: [htt
p://schemas.microsoft.com/wbem/wsman/1/wmi/root/scvmm/HttpPostDeploymentJob] ** hv09.domain.local ** ** **
[s#1] DnsHelper.GetDomainName(dnsName: (string) "mgmt-scvmm.Domain.local", parseOption: DnsParseOption.RequireFQDN)
in s:\btvsts\6723\private\product\common\Utils\DnsHelper_pp.cs:line 219
Full call stack from when the exception was thrown: at Microsoft.SystemCenter.VirtualMachineManager.Cmdlets.TaskCmdlet
Base`1.Process()
at Microsoft.SystemCenter.VirtualMachineManager.Cmdlets.CmdletBase.ProcessRecord()
at System.Management.Automation.CommandProcessor.ProcessRecord()
at System.Management.Automation.CommandProcessorBase.DoExecute()
at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input)
at System.Management.Automation.PipelineOps.InvokePipeline(Object input, Boolean ignoreInput, CommandParameterInterna
l[][] pipeElements, CommandBaseAst[] pipeElementAsts, CommandRedirection[][] commandRedirections, FunctionContext funcCo
ntext)
at System.Management.Automation.Interpreter.ActionCallInstruction`6.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.Interpreter.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.LightLambda.RunVoid1[T0](T0 arg0)
at System.Management.Automation.PSScriptCmdlet.RunClause(Action`1 clause, Object dollarUnderbar, Object inputToProces
s)
at System.Management.Automation.PSScriptCmdlet.DoEndProcessing()
at System.Management.Automation.CommandProcessorBase.Complete()
at System.Management.Automation.CommandProcessorBase.DoComplete()
at System.Management.Automation.Internal.PipelineProcessor.DoCompleteCore(CommandProcessorBase commandRequestingUpstr
eamCommandsToStop)
at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input)
at System.Management.Automation.PipelineOps.InvokePipeline(Object input, Boolean ignoreInput, CommandParameterInterna
l[][] pipeElements, CommandBaseAst[] pipeElementAsts, CommandRedirection[][] commandRedirections, FunctionContext funcCo
ntext)
at System.Management.Automation.Interpreter.ActionCallInstruction`6.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.Interpreter.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.LightLambda.RunVoid1[T0](T0 arg0)
at System.Management.Automation.DlrScriptCommandProcessor.RunClause(Action`1 clause, Object dollarUnderbar, Object in
putToProcess)
at System.Management.Automation.DlrScriptCommandProcessor.Complete()
at System.Management.Automation.CommandProcessorBase.DoComplete()
at System.Management.Automation.Internal.PipelineProcessor.DoCompleteCore(CommandProcessorBase commandRequestingUpstr
eamCommandsToStop)
at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input)
at System.Management.Automation.Runspaces.LocalPipeline.InvokeHelper()
at System.Management.Automation.Runspaces.LocalPipeline.InvokeThreadProc()
at System.Management.Automation.Runspaces.LocalPipeline.StartPipelineExecution()
at System.Management.Automation.Runspaces.PipelineBase.CoreInvoke(IEnumerable input, Boolean syncCall)
at System.Management.Automation.PowerShell.Worker.ConstructPipelineAndDoWork(Runspace rs, Boolean performSyncInvoke)
at System.Management.Automation.PowerShell.Worker.RunspaceAvailableCallback(IAsyncResult asyncResult)
at System.Management.Automation.Runspaces.AsyncResult.SetAsCompleted(Exception exception)
at System.Management.Automation.Runspaces.Internal.RunspacePoolInternal.ServicePendingRequests(Object useCallingThrea
dState)
at System.Management.Automation.Runspaces.Internal.RunspacePoolInternal.EnqueueCheckAndStartRequestServicingThread(Ge
tRunspaceAsyncResult requestToEnqueue, Boolean useCallingThread)
at System.Management.Automation.Runspaces.Internal.RunspacePoolInternal.BeginGetRunspace(AsyncCallback callback, Obje
ct state)
at System.Management.Automation.PowerShell.CoreInvokeAsync[TInput,TOutput](PSDataCollection`1 input, PSDataCollection
`1 output, PSInvocationSettings settings, AsyncCallback callback, Object state, PSDataCollection`1 asyncResultOutput)
at System.Management.Automation.PowerShell.BeginInvoke[TInput,TOutput](PSDataCollection`1 input, PSDataCollection`1 o
utput, PSInvocationSettings settings, AsyncCallback callback, Object state)
at System.Management.Automation.ServerRunspacePoolDriver.HandleCreateAndInvokePowerShell(Object sender, RemoteDataEve
ntArgs`1 eventArgs)
at System.Management.Automation.ExtensionMethods.SafeInvoke[T](EventHandler`1 eventHandler, Object sender, T eventArg
s)
at System.Management.Automation.Remoting.ServerRemoteSessionDSHandlerStateMachine.DoMessageReceived(Object sender, Re
moteSessionStateMachineEventArgs fsmEventArg)
at System.Management.Automation.ExtensionMethods.SafeInvoke[T](EventHandler`1 eventHandler, Object sender, T eventArg
s)
at System.Management.Automation.Remoting.BaseTransportManager.OnDataAvailableCallback(RemoteDataObject`1 remoteObject
)
at System.Management.Automation.Remoting.ReceiveDataCollection.ProcessRawData(Byte[] data, OnDataAvailableCallback ca
llback)
at System.Management.Automation.Remoting.PriorityReceiveDataCollection.ProcessRawData(Byte[] data, DataPriorityType p
riorityType, OnDataAvailableCallback callback)
at System.Management.Automation.Remoting.BaseTransportManager.ProcessRawData(Byte[] data, String stream)
at System.Management.Automation.Remoting.Server.OutOfProcessServerTransportManager.ProcessRawData(Byte[] data, String
stream)
at System.Management.Automation.Remoting.Server.OutOfProcessMediatorBase.OnDataPacketReceived(Byte[] rawData, String
stream, Guid psGuid)
at System.Management.Automation.Remoting.OutOfProcessUtils.ProcessElement(XmlReader xmlReader, DataProcessingDelegate
s callbacks)
at System.Management.Automation.Remoting.OutOfProcessUtils.ProcessData(String data, DataProcessingDelegates callbacks
)
at System.Management.Automation.Remoting.Server.OutOfProcessMediatorBase.ProcessingThreadStart(Object state)
at System.Management.Automation.Utils.WorkItemCallback(Object callBackArgs)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object
state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, B
oolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()