Lost ALL VM from services

[Amdusis]

Hello,

In our MIQ infrastructure VMs was lost from services but allive on vCenter. We have no idea why it happened.

In logs we see "MiqAeEvent.build_evm_event" for all orphand VMs in the same time.

WARN -- evm: MIQ(MiqAlert.evaluate_alerts) [vm_create] Target: ManageIQ::Providers::Vmware::InfraManager::Vm Name: [*****], Id: [1000000036541] Result: No enabled alerts are assigned to target! Nothing to do.

INFO -- evm: MIQ(MiqAlert.evaluate_alerts) [vm_create] Target: ManageIQ::Providers::Vmware::InfraManager::Vm Name: [*****], Id: [1000000036541]

INFO -- policy: MIQ(policy-enforce_policy): Event: [vm_create], To: [*******]

INFO -- automation: MiqAeEvent.build_evm_event >> event=<"vm_create"> inputs=<{:vm=>#<ManageIQ::Providers::Vmware::InfraManager::Vm id: 1000000036541, vendor: "vmware", format: nil, version: nil, name: "*******", description: "Owner: nEmail: ...", location: "bf5c1469-8632-994e-e6e8-88e9a4147000/", config_xml: nil, autostart: nil, host_id: 1000000000269, last_sync_on: nil, created_on: "2025-11-18 08:24:11.245767000 +0000", updated_on: "2025-11-18 08:24:11.245767000 +0000", storage_id: 1000000000001, guid: "82189a1a-53d4-451c-afe5-5a796d4f4880", ems_id: 1000000000003, last_scan_on: nil, last_scan_attempt_on: nil, uid_ems: "4205a578-5133-dfd9-a844-8aa672cbf927", retires_on: nil, retired: nil, boot_time: "2025-11-16 02:46:53.998340000 +0000", tools_status: "toolsOk", standby_action: "checkpoint", power_state: "on", state_changed_on: "2025-11-18 08:24:11.240019000 +0000", previous_state: nil, connection_state: "connected", last_perf_capture_on: nil, registered: nil, busy: nil, smart: nil, memory_reserve: 0, memory_reserve_expand: false, memory_limit: -1, memory_shares: 40960, memory_shares_level: "normal", cpu_reserve: 0, cpu_reserve_expand: false, cpu_limit: -1, cpu_shares: 4000, cpu_shares_level: "normal", cpu_affinity: nil, ems_created_on: nil, template: false, evm_owner_id: nil, miq_group_id: 1000000000001, linked_clone: nil, fault_tolerance: nil, type: "ManageIQ::Providers::Vmware::InfraManager::Vm", ems_ref: "vm-2657135", ems_cluster_id: 1000000000003, retirement_warn: nil, retirement_last_warn: nil, vnc_port: nil, flavor_id: nil, availability_zone_id: nil, cloud: false, retirement_state: nil, cloud_network_id: nil, cloud_subnet_id: nil, cloud_tenant_id: nil, raw_power_state: "poweredOn", publicly_available: nil, orchestration_stack_id: nil, retirement_requester: nil, tenant_id: 1000000000001, resource_group_id: nil, deprecated: nil, storage_profile_id: nil, cpu_hot_add_enabled: false, cpu_hot_remove_enabled: false, memory_hot_add_enabled: false, memory_hot_add_limit: 4096, memory_hot_add_increment: 0, hostname: nil, ems_ref_type: "VirtualMachine", restart_needed: nil, ancestry: nil, placement_group_id: nil>, :host=>#<ManageIQ::Providers::Vmware::InfraManager::HostEsx id: 1000000000269, name: "*******", hostname: "********", ipaddress: "******", vmm_vendor: "vmware", vmm_version: "8.0.3", vmm_product: "ESXi", vmm_buildnumber: "24859861", created_on: "2024-10-18 09:25:07.371837000 +0000", updated_on: "2025-09-13 11:17:50.823450000 +0000", guid: "34d8b08b-1935-47d3-ba9c-9931123069d6", ems_id: 1000000000003, user_assigned_os: nil, power_state: "on", smart: 1, settings: {}, last_perf_capture_on: nil, uid_ems: "********", connection_state: "connected", ssh_permit_root_login: nil, admin_disabled: true, service_tag: "*****", asset_tag: nil, ipmi_address: nil, mac_address: nil, type: "ManageIQ::Providers::Vmware::InfraManager::HostEsx", failover: nil, ems_ref: "host-1803363", hyperthreading: true, ems_cluster_id: 1000000000003, next_available_vnc_port: nil, hypervisor_hostname: nil, availability_zone_id: nil, maintenance: false, maintenance_reason: nil, physical_server_id: nil, ems_ref_type: "HostSystem">, "MiqEvent::miq_event"=>1000102667621, :miq_event_id=>1000102667621, "EventStream::event_stream"=>1000102667621, :event_stream_id=>1000102667621}>

Seems like manageiq recreated VMs.

Can you help with this problem? Very critical for us.

Thnaks!

[Fryguy]

That's creating the new VMs, but do you see it removing the old VMs?

[Amdusis]

@agrare
No VMs in archive.

from rails:
No objects in vmdb

irb(main):005:0> vm = $evm.vmdb('vm').find('1000000036554')
(irb):5:in <main>': undefined methodvmdb' for nil (NoMethodError)
        from <internal:kernel>:187:in loop'
        from /opt/manageiq/manageiq-gemset/gems/railties-7.0.8.7/lib/rails/commands/console/console_command.rb:74:instart'
        from /opt/manageiq/manageiq-gemset/gems/railties-7.0.8.7/lib/rails/commands/console/console_command.rb:19:in start'
        from /opt/manageiq/manageiq-gemset/gems/railties-7.0.8.7/lib/rails/commands/console/console_command.rb:106:inperform'
        from /opt/manageiq/manageiq-gemset/gems/thor-1.3.2/lib/thor/command.rb:28:in run'
        from /opt/manageiq/manageiq-gemset/gems/thor-1.3.2/lib/thor/invocation.rb:127:ininvoke_command'
        from /opt/manageiq/manageiq-gemset/gems/thor-1.3.2/lib/thor.rb:538:in dispatch'
        from /opt/manageiq/manageiq-gemset/gems/railties-7.0.8.7/lib/rails/command/base.rb:87:inperform'
        from /opt/manageiq/manageiq-gemset/gems/railties-7.0.8.7/lib/rails/command.rb:48:in invoke'
        from /opt/manageiq/manageiq-gemset/gems/railties-7.0.8.7/lib/rails/commands.rb:18:in<main>'
        from /usr/share/ruby/bundled_gems.rb:69:in require'
        from /usr/share/ruby/bundled_gems.rb:69:inblock (2 levels) in replace_require'
        from /opt/manageiq/manageiq-gemset/gems/bootsnap-1.18.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in require'
        from bin/rails:4:in<main>'

In miq:

irb(main):003:0> vm = Vm.find_by(name: "****redis08")
=>

#<ManageIQ::Providers::Vmware::InfraManager::Vm:0x00007fc22d04a000
...
irb(main):004:0> vm
=>

#<ManageIQ::Providers::Vmware::InfraManager::Vm:0x00007fc22d04a000
 id: 1000000036554,
 vendor: "vmware",
 format: nil,
 version: nil,
 name: "redis08",
 description: "\n\nMIQ GUID=06ddb69d-fa3b-4a12-99eb-a348cc711485",
 location: "db851f69-aa44-bdd5-2cfe-1402ece3bf38/redis08.vmx",
 config_xml: nil,
 autostart: nil,
 host_id: 1000000000038,
 last_sync_on: nil,
 created_on: Tue, 18 Nov 2025 08:33:49.572438000 UTC +00:00,
 updated_on: Sun, 23 Nov 2025 00:19:04.363323000 UTC +00:00,
 storage_id: 1000000000752,
 guid: "97953454-6ac7-4db7-854b-2d294f88ae0f",
 ems_id: 1000000000003,
 last_scan_on: nil,
 last_scan_attempt_on: nil,
 uid_ems: "4205928a-5b51-255b-1d87-4d4cd6813d99",
 retires_on: nil,
 retired: nil,
 boot_time: Sun, 23 Nov 2025 00:18:43.000000000 UTC +00:00,
 tools_status: "toolsOk",
 standby_action: "checkpoint",
 power_state: "on",
 state_changed_on: Tue, 18 Nov 2025 08:33:49.566513000 UTC +00:00,
 previous_state: nil,
 connection_state: "connected",
 last_perf_capture_on: nil,
 registered: nil,
 busy: nil,
 smart: nil,
 memory_reserve: 0,
 memory_reserve_expand: false,
 memory_limit: -1,
 memory_shares: 40960,
 memory_shares_level: "normal",
 cpu_reserve: 0,
 cpu_reserve_expand: false,
 cpu_limit: -1,
 cpu_shares: 4000,
 cpu_shares_level: "normal",
 cpu_affinity: nil,
 ems_created_on: nil,
 template: false,
 evm_owner_id: nil,
 miq_group_id: 1000000000001,
 linked_clone: nil,
 fault_tolerance: nil,
 type: "ManageIQ::Providers::Vmware::InfraManager::Vm",
 ems_ref: "vm-2657128",
 ems_cluster_id: 1000000000226,
 retirement_warn: nil,
 retirement_last_warn: nil,
 vnc_port: nil,
 flavor_id: nil,
 availability_zone_id: nil,
 cloud: false,
 retirement_state: nil,
 cloud_network_id: nil,
 cloud_subnet_id: nil,
 cloud_tenant_id: nil,
 raw_power_state: "poweredOn",
 publicly_available: nil,
 orchestration_stack_id: nil,
 retirement_requester: nil,
 tenant_id: 1000000000001,
 resource_group_id: nil,
 deprecated: nil,
 storage_profile_id: nil,
 cpu_hot_add_enabled: false,
 cpu_hot_remove_enabled: false,
 memory_hot_add_enabled: false,
 memory_hot_add_limit: 4096,
 memory_hot_add_increment: 0,
 hostname: nil,
 ems_ref_type: "VirtualMachine",
 restart_needed: nil,
 ancestry: nil,
 placement_group_id: nil>

created_on: Tue, 18 Nov 2025 08:33:49.572438000 UTC +00:00, - time when all vms lost

[agrare]

Don't use find_by it will return only a single record, try using Vm.where(:name => "....")

[Amdusis]

@agrare
Just one record

irb(main):003:0> Vm.where(:name => "redis08")
=>
[#<ManageIQ::Providers::Vmware::InfraManager::Vm:0x00007f55f4dc9438
  id: 1000000036554,
  vendor: "vmware",
  format: nil,
  version: nil,
  name: "redis08",
  description: "\nMIQ GUID=06ddb69d-fa3b-4a12-99eb-a348cc711485",
  location: "db851f69-aa44-bdd5-2cfe-1402ece3bf38/redis08.vmx",
  config_xml: nil,
  autostart: nil,
  host_id: 1000000000038,
  last_sync_on: nil,
  created_on: Tue, 18 Nov 2025 08:33:49.572438000 UTC +00:00,
  updated_on: Sun, 23 Nov 2025 00:19:04.363323000 UTC +00:00,
  storage_id: 1000000000752,
  guid: "97953454-6ac7-4db7-854b-2d294f88ae0f",
  ems_id: 1000000000003,
  last_scan_on: nil,
  last_scan_attempt_on: nil,
  uid_ems: "4205928a-5b51-255b-1d87-4d4cd6813d99",
  retires_on: nil,
  retired: nil,
  boot_time: Sun, 23 Nov 2025 00:18:43.000000000 UTC +00:00,
  tools_status: "toolsOk",
  standby_action: "checkpoint",
  power_state: "on",
  state_changed_on: Tue, 18 Nov 2025 08:33:49.566513000 UTC +00:00,
  previous_state: nil,
  connection_state: "connected",
  last_perf_capture_on: nil,
  registered: nil,
  busy: nil,
  smart: nil,
  memory_reserve: 0,
  memory_reserve_expand: false,
  memory_limit: -1,
  memory_shares: 40960,
  memory_shares_level: "normal",
  cpu_reserve: 0,
  cpu_reserve_expand: false,
  cpu_limit: -1,
  cpu_shares: 4000,
  cpu_shares_level: "normal",
  cpu_affinity: nil,
  ems_created_on: nil,
  template: false,
  evm_owner_id: nil,
  miq_group_id: 1000000000001,
  linked_clone: nil,
  fault_tolerance: nil,
  type: "ManageIQ::Providers::Vmware::InfraManager::Vm",
  ems_ref: "vm-2657128",
  ems_cluster_id: 1000000000226,
  retirement_warn: nil,
  retirement_last_warn: nil,
  vnc_port: nil,
  flavor_id: nil,
  availability_zone_id: nil,
  cloud: false,
  retirement_state: nil,
  cloud_network_id: nil,
  cloud_subnet_id: nil,
  cloud_tenant_id: nil,
  raw_power_state: "poweredOn",
  publicly_available: nil,
  orchestration_stack_id: nil,
  retirement_requester: nil,
  tenant_id: 1000000000001,
  resource_group_id: nil,
  deprecated: nil,
  storage_profile_id: nil,
  cpu_hot_add_enabled: false,
  cpu_hot_remove_enabled: false,
  memory_hot_add_enabled: false,
  memory_hot_add_limit: 4096,
  memory_hot_add_increment: 0,
  hostname: nil,
  ems_ref_type: "VirtualMachine",
  restart_needed: nil,
  ancestry: nil,
  placement_group_id: nil>]

[agrare]

Okay thanks, I'd be surprised if an error with the Refresher caused this because we don't actually delete VMs from the database we only archive them. Not ruling anything fully out though, I'll have to see the logs from the error.

[Amdusis]

Thanks!
If you need any additional logs or information we are redy to help)

[nikobo]

Same issue since November 14. vCenter sync crash and vcenter inventory on manageiq is reseted.
The strange think is that we have a another manageiq instance targeting the same vcenter with the same issue..

[nikobo]

Hello @agrare, have you find the patch to debug more ?

[agrare]

Hey sorry I was off for the holidays I'll get on this patch this week

[Amdusis]

Hello!

Any updates?)

[agrare]

Hey @Amdusis sorry for the delay.

I've created a script to record the XML interactions with your vCenter using the VCR gem.

Please download [vmware_refresh.rb] and place it in the root of a manageiq development environment. It cannot be a production appliance because we have to have the vcr and webmock gems installed and you don't want to go breaking your production gemset.

$ rails r vmware_refresh_vcr.rb -- --ems-id=EMS_ID

Where EMS_ID is the id of your VMware system, you can find the ID in the URL:

This will start up a VMware refresh worker. You'll want to tail the logs to see when the refresh finishes and then continue to receive updates. Once the updates stop coming in ctrl-c the script. This will instruct the worker to exit and write out the XML interactions to a file ./spec/vcr_cassettes/vmware-refresh.yml.

The username/password will be in this file, so before sending it make sure you change at least the <password> attribute:

- request:
    method: post
    uri: https://vc.example.com/sdk
    body:
      encoding: UTF-8
      string: <env:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:env="http://schemas.xmlsoap.org/soap/envelope/"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><env:Body><Login xmlns="urn:vim25"><_this
        type="SessionManager">SessionManager</_this><userName>USERNAME</userName><password>PASSWORD</password></Login></env:Body></env:Envelope>

You can also change the hostname/uri to something else if you'd like but there will be many more instances of that (sed is your friend here).

Once you've done this you can email the results to myself mailto:adam@grare.com or create a secret gist and send me that link.

Once I have that I'll be able to run the exact refresh that you are seeing on my system and investigate the results.

Look forward to hearing from you

[Amdusis]

@agrare
Big thanks!

@nikobo
FYI

[Amdusis]

@Fryguy @agrare
Good day friends!)
The problem is reproduced
https://github.com/orgs/ManageIQ/discussions/23661#discussion-9163596
See the same logs: create event and VM lost from service
INFO -- automation: MiqAeEvent.build_evm_event >> event=<"vm_create"> inputs=<{:vm=>#<ManageIQ::Providers::Vmware::InfraManager::Vm id: 1000000043416, vendor: "vmware", format: nil, version: nil, name: "*****", description: "Owner: ******\nEmail: ...", location: "******", config_xml: nil, autostart: nil, host_id: 1000000000348, last_sync_on: nil, created_on: "2026-04-14 13:38:08.002425000 +0000", updated_on: "2026-04-14 13:38:08.002425000 +0000", storage_id: 1000000000111, guid: "9a8663b8-fd61-4782-bea5-9d02d8242892", ems_id: 1000000000004, last_scan_on: nil, last_scan_attempt_on: nil, uid_ems: "4239cb82-1d8b-0248-19e4-c0b2bb7d5c36", retires_on: nil, retired: nil, boot_time: "2026-04-03 23:30:01.099804000 +0000", tools_status: "toolsOk", standby_action: "checkpoint", power_state: "on", state_changed_on: "2026-04-14 13:38:07.996437000 +0000", previous_state: nil, connection_state: "connected", last_perf_capture_on: nil, registered: nil, busy: nil, smart: nil, memory_reserve: 0, memory_reserve_expand: false, memory_limit: -1, memory_shares: 40960, memory_shares_level: "normal", cpu_reserve: 0, cpu_reserve_expand: false, cpu_limit: -1, cpu_shares: 2000, cpu_shares_level: "normal", cpu_affinity: nil, ems_created_on: nil, template: false, evm_owner_id: nil, miq_group_id: 1000000000001, linked_clone: nil, fault_tolerance: nil, type: "ManageIQ::Providers::Vmware::InfraManager::Vm", ems_ref: "vm-10754534", ems_cluster_id: 1000000000013, retirement_warn: nil, retirement_last_warn: nil, vnc_port: nil, flavor_id: nil, availability_zone_id: nil, cloud: false, retirement_state: nil, cloud_network_id: nil, cloud_subnet_id: nil, cloud_tenant_id: nil, raw_power_state: "poweredOn", publicly_available: nil, orchestration_stack_id: nil, retirement_requester: nil, tenant_id: 1000000000001, resource_group_id: nil, deprecated: nil, storage_profile_id: nil, cpu_hot_add_enabled: false, cpu_hot_remove_enabled: false, memory_hot_add_enabled: false, memory_hot_add_limit: 8192, memory_hot_add_increment: 0, hostname: nil, ems_ref_type: "VirtualMachine", restart_needed: nil, ancestry: nil, placement_group_id: nil>, :host=>#<ManageIQ::Providers::Vmware::InfraManager::HostEsx id: 1000000000348, name: "*****", hostname: "****", ipaddress: "****", vmm_vendor: "vmware", vmm_version: "8.0.3", vmm_product: "ESXi", vmm_buildnumber: "25067014", created_on: "2025-02-21 08:10:34.330231000 +0000", updated_on: "2026-03-21 17:58:31.986844000 +0000", guid: "68751634-d0b1-481f-940d-e1ecb7409a51", ems_id: 1000000000004, user_assigned_os: nil, power_state: "on", smart: 1, settings: {}, last_perf_capture_on: nil, uid_ems: "*****", connection_state: "connected", ssh_permit_root_login: nil, admin_disabled: true, service_tag: "*****", asset_tag: "To be filled by O.E.M.", ipmi_address: nil, mac_address: nil, type: "ManageIQ::Providers::Vmware::InfraManager::HostEsx", failover: nil, ems_ref: "host-8930287", hyperthreading: true, ems_cluster_id: 1000000000013, next_available_vnc_port: nil, hypervisor_hostname: nil, availability_zone_id: nil, maintenance: false, maintenance_reason: nil, physical_server_id: nil, ems_ref_type: "HostSystem">, "MiqEvent::miq_event"=>1000104101024, :miq_event_id=>1000104101024, "EventStream::event_stream"=>1000104101024, :event_stream_id=>1000104101024}>

No VM in archive

In VMDB only one record:

irb(main):014:0> Vm.where(:name => "vm_name").count
=> 1

VM was deployed on vCenter in

$vm.CreateDate
Jun 21 2026 г. 13:02:15

ManageIQ discovered VM immediately after problem:
Discovered Tue, 14 Apr 2026 16:38:08 +0300

Was no work on the virtualization infrastructure. Totaly don't understand why its happened.

Miracles)

[agrare]

Hey @Amdusis were you able to run the script I built above https://github.com/orgs/ManageIQ/discussions/23661#discussioncomment-15563544 ?

Based on prior investigation this seems like a bug with VMware's WaitForUpdatesEx API call returning "complete" too early, but I need to see the exact API requests/responses to be sure.

[agrare]

@Amdusis can you also show the full record for the query Vm.where(:name => "vm_name")? I'm specifically curious about the created_on/updated_on dates.

[Amdusis]

@agrare Hello!
Trying run script but recive error

vmdb]# rails r vmware_refresh_vcr.rb -- --ems-id=1000000000003
/usr/share/ruby/bundled_gems.rb:69:in require': cannot load such file -- vcr (LoadError)
        from /usr/share/ruby/bundled_gems.rb:69:inblock (2 levels) in replace_require'
        from /opt/manageiq/manageiq-gemset/gems/bootsnap-1.18.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:27:in require'
        from /opt/manageiq/manageiq-gemset/gems/zeitwerk-2.6.18/lib/zeitwerk/kernel.rb:34:inrequire'
        from /var/www/miq/vmdb/vmware_refresh_vcr.rb:6:in <main>'
        from /opt/manageiq/manageiq-gemset/gems/railties-7.0.8.7/lib/rails/commands/runner/runner_command.rb:43:inload'
        from /opt/manageiq/manageiq-gemset/gems/railties-7.0.8.7/lib/rails/commands/runner/runner_command.rb:43:in perform'
        from /opt/manageiq/manageiq-gemset/gems/thor-1.3.2/lib/thor/command.rb:28:inrun'
        from /opt/manageiq/manageiq-gemset/gems/thor-1.3.2/lib/thor/invocation.rb:127:in invoke_command'
        from /opt/manageiq/manageiq-gemset/gems/thor-1.3.2/lib/thor.rb:538:indispatch'
        from /opt/manageiq/manageiq-gemset/gems/railties-7.0.8.7/lib/rails/command/base.rb:87:in perform'
        from /opt/manageiq/manageiq-gemset/gems/railties-7.0.8.7/lib/rails/command.rb:48:ininvoke'
        from /opt/manageiq/manageiq-gemset/gems/railties-7.0.8.7/lib/rails/commands.rb:18:in <main>'
        from /usr/share/ruby/bundled_gems.rb:69:inrequire'
        from /usr/share/ruby/bundled_gems.rb:69:in block (2 levels) in replace_require'
        from /opt/manageiq/manageiq-gemset/gems/bootsnap-1.18.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:inrequire'
        from bin/rails:4:in `<main>'

vmdb]# gem list | grep -E "vcr|optimist|webmock"
optimist (3.1.0)
vcr (6.4.0)
webmock (3.26.2)

can you also show the full record for the query Vm.where(:name => "vm_name")? I'm specifically curious about the created_on/updated_on dates.

irb(main):018:0> Vm.where(:name => "vm_name")
=>
[#<ManageIQ::Providers::Vmware::InfraManager::Vm:0x00007f0731340d20
  id: 1000000043416,
  vendor: "vmware",
  format: nil,
  version: nil,
  name: "vm_name",
  description: "vm_name",
  location: "vm_name.vmx",
  config_xml: nil,
  autostart: nil,
  host_id: 1000000000348,
  last_sync_on: nil,
  created_on: Tue, 14 Apr 2026 13:38:08.002425000 UTC +00:00,
  updated_on: Thu, 16 Apr 2026 08:48:26.254921000 UTC +00:00,
  storage_id: 1000000000111,
  guid: "9a8663b8-fd61-4782-bea5-9d02d8242892",
  ems_id: 1000000000004,
  last_scan_on: nil,
  last_scan_attempt_on: nil,
  uid_ems: "4239cb82-1d8b-0248-19e4-c0b2bb7d5c36",
  retires_on: nil,
  retired: nil,
  boot_time: Fri, 03 Apr 2026 23:30:01.099804000 UTC +00:00,
  tools_status: "toolsOk",
  standby_action: "checkpoint",
  power_state: "on",
  state_changed_on: Tue, 14 Apr 2026 13:38:07.996437000 UTC +00:00,
  previous_state: nil,
  connection_state: "connected",
  last_perf_capture_on: nil,
  registered: nil,
  busy: nil,
  smart: nil,
  memory_reserve: 0,
  memory_reserve_expand: false,
  memory_limit: -1,
  memory_shares: 40960,
  memory_shares_level: "normal",
  cpu_reserve: 0,
  cpu_reserve_expand: false,
  cpu_limit: -1,
  cpu_shares: 2000,
  cpu_shares_level: "normal",
  cpu_affinity: nil,
  ems_created_on: nil,
  template: false,
  evm_owner_id: nil,
  miq_group_id: 1000000000351,
  linked_clone: nil,
  fault_tolerance: nil,
  type: "ManageIQ::Providers::Vmware::InfraManager::Vm",
  ems_ref: "vm-10754534",
  ems_cluster_id: 1000000000013,
  retirement_warn: nil,
  retirement_last_warn: nil,
  vnc_port: nil,
  flavor_id: nil,
  availability_zone_id: nil,
  cloud: false,
  retirement_state: nil,
  cloud_network_id: nil,
  cloud_subnet_id: nil,
  cloud_tenant_id: nil,
  raw_power_state: "poweredOn",
  publicly_available: nil,
  orchestration_stack_id: nil,
  retirement_requester: nil,
  tenant_id: 1000000000189,
  resource_group_id: nil,
  deprecated: nil,
  storage_profile_id: nil,
  cpu_hot_add_enabled: false,
  cpu_hot_remove_enabled: false,
  memory_hot_add_enabled: false,
  memory_hot_add_limit: 8192,
  memory_hot_add_increment: 0,
  hostname: nil,
  ems_ref_type: "VirtualMachine",
  restart_needed: nil,
  ancestry: nil,
  placement_group_id: nil>]

[agrare]

Hey @Amdusis it looks like you're trying to run it from an appliance, per my comment above and place it in the root of a manageiq development environment not a production appliance.

Looking at the VM record, the VM was actually created recently, created_on: Tue, 14 Apr 2026 13:38:08.002425000 UTC +00:00 not just archived & reconnected. We don't delete VMs or Hosts as part of a typical ems refresh so this would seem to indicate something else is going on.