IPMI trick for undercloud via VirtualBMC

Notes from the author: Probably you were spending some time looking up this answer like I did before you. If you are interesting in collaborate, let me know, you can DM me via twitter, linkedin or instagram. You can have an space in this blog. Appreciate if you can at least follow me via my social accounts. Thanks.

Red Hat have been pushing its automated installation of OpenStack with Director based on TripleO concept over the last years. Many Telcos are adopting this concept for the flexibility and agility that automation brings. This post called “IPMI trick for undercloud via VirtualBMC” is support material for your lab and use KVM instances simulating bare metals that can be set up via ironic and its IPMI (IP management interface) .

TripleO is an OpenStack project that aims to utilize OpenStack itself as the foundations for deploying OpenStack. To clarify, TripleO advocates the use of native OpenStack components, and their respective API’s to configure, deploy, and manage OpenStack environments itself. Undercloud is the OpenStack component that install the actual OpenStack called Overcloud. Containers are also a key component of this architecture.

VirtualBMC installation

I am working with RHEL 8.2. You have, at least on my knowledge, two ways to install vbmc. IMPORTANT: I did it after installing the undercloud (tripleO).

VirtualBMC installation via pip3

vbmc is a python app , then you can use directly pip3 and adding the required packages. .

# those commands must run in the undercloud server (RHEL 8.2 in my case)
# add the repo
sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
# enable it
sudo ARCH=$( /bin/arch ) subscription-manager repos --enable "codeready-builder-for-rhel-8-${ARCH}-rpms"
# add the required packages
sudo yum -y install libvirt libvirt-devel gcc
# install virtualbmc (check dependencies in advanced)
sudo pip3 install virtualbmc

In this case, you will vbmc and vbmcd in a known path in ‘/usr/local/bin’

You have to enable the service and for this you will have to create the system file.

# I am assuming you're logged as root
cat > /etc/systemd/system/vbmcd.service <<EOF
[Install]
WantedBy = multi-user.target
[Service]
BlockIOAccounting = True
CPUAccounting = True
ExecReload = /bin/kill -HUP $MAINPID
ExecStart = /usr/local/bin/vbmcd --foreground
Group = root
MemoryAccounting = True
PrivateDevices = False
PrivateNetwork = False
PrivateTmp = False
PrivateUsers = False
Restart = on-failure
RestartSec = 2
Slice = vbmc.slice
TasksAccounting = True
TimeoutSec = 120
Type = simple
User = root
[Unit]
After = libvirtd.service
After = syslog.target
After = network.target
Description = vbmc service
EOF
systemctl daemon-reload
systemctl enable vbmcd.service
systemctl start vbmcd.service
systemctl status vbmcd.service

it’s the service is in ‘active’ status, then you’re done with the installation

VirtualBMC installation via virtualenv

Or you can go with the one I used . It’s the same, however, the files we’ll be in a unknown path, then you will have to add it to your .bashrc file

# Assuming you have already install undercloud tripleO
virtualenv --system-site-packages /opt/vbmc
/opt/vbmc/bin/pip install virtualbmc
cat > /etc/systemd/system/vbmcd.service <<EOF
[Install]
WantedBy = multi-user.target
[Service]
BlockIOAccounting = True
CPUAccounting = True
ExecReload = /bin/kill -HUP $MAINPID
ExecStart = /opt/vbmc/bin/vbmcd --foreground
Group = root
MemoryAccounting = True
PrivateDevices = False
PrivateNetwork = False
PrivateTmp = False
PrivateUsers = False
Restart = on-failure
RestartSec = 2
Slice = vbmc.slice
TasksAccounting = True
TimeoutSec = 120
Type = simple
User = root
[Unit]
After = libvirtd.service
After = syslog.target
After = network.target
Description = vbmc service
EOF
systemctl daemon-reload
systemctl enable vbmcd.service
systemctl start vbmcd.service
systemctl status vbmcd.service

You should see the service as ‘active’ like this

[stack@undercloud ~]$ systemctl status vbmcd.service
● vbmcd.service - vbmc service
   Loaded: loaded (/etc/systemd/system/vbmcd.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2020-11-09 12:19:24 EST; 3 days ago
 Main PID: 32504 (vbmcd)
    Tasks: 5 (limit: 152850)
   Memory: 43.8M
      CPU: 4min 52.281s
   CGroup: /vbmc.slice/vbmcd.service
           ├─32504 /opt/vbmc/bin/python3.6 /opt/vbmc/bin/vbmcd --foreground
           └─34558 /opt/vbmc/bin/python3.6 /opt/vbmc/bin/vbmcd --foreground

Configuring vbmc and testing ipmi ports

Having the vbmcd service up, then we can start creating the first KVM instance and test it. My KVM host is at 10.5.0.101 (Centos 7.6, and yes, it’s a different distro and version), then I’ve generated the certificates in the undercloud server to access this KVM hosts directly via ssh and without password

After that I’ve tried to access virsh remotely via “virsh –connect” like this

[stack@undercloud ~]$ virsh --connect=qemu+ssh://root@10.5.0.101/system
Welcome to virsh, the virtualization interactive terminal.
Type:  'help' for help with commands
       'quit' to quit
virsh # list --all
 Id   Name                   State
---------------------------------------
 -    overcloud-controller   shut off

You can see the instance ‘overcloud-controller’ is off. keep it like this. You can check its configuration in our repo in github.

Let’s create the access to ipmi via port 6001

vbmc add overcloud-controller --port 6001 --username admin --password nuagenet --libvirt-uri qemu+ssh://root@10.5.0.101/system

Don’t forget to start it up.

/opt/vbmc/bin/vbmc start overcloud-controller

After that, you can test your ipmi port using ipmitool like this:

[stack@undercloud ~]$ ipmitool -I lanplus -U admin -P nuagenet -H 127.0.0.1 -p 6001 power status
Chassis Power is off

Importing baremetal servers

Let’s create the JSON file to import this pseudo bare metal server:

[stack@undercloud ~]$ cat overcloud-compute1.json
{
  "nodes": [
    {
      "arch": "x86_64",
      "disk": "200",
      "memory": "24576",
      "name": "overcloud-compute1",
      "pm_user": "admin",
      "pm_addr": "127.0.0.1",
      "pm_password": "nuagenet",
      "pm_port": "6002",
      "pm_type": "pxe_ipmitool",
      "mac": [
        "52:54:00:5e:68:b1"
      ],
      "cpu": "12"
    }
  ]
}

And then, just need to execute the import process. I am doing also the introspection.

openstack overcloud node import --introspect --provide overcloud-compute1.json

And we’re done.

Thanks for reading “IPMI trick for undercloud via VirtualBMC”. See ya!

Troubleshooting

No kernel image provided

If you get this message “RuntimeError: No kernel image provided and none of [‘x86_64/agent.kernel’, ‘agent.kernel’] found in /var/lib/ironic/httpboot” after executing command: “openstack overcloud node import –introspect –provide overcloud-compute1.json”

(undercloud) [stack@undercloud ~]$ openstack overcloud node import --introspect --provide overcloud-compute1.json
Exception occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_node.py", line 420, in take_action
    http_boot=parsed_args.http_boot)
  File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 1755, in update_nodes_deploy_data
    http_boot))
RuntimeError: No kernel image provided and none of ['x86_64/agent.kernel', 'agent.kernel'] found in /var/lib/ironic/httpboot
No kernel image provided and none of ['x86_64/agent.kernel', 'agent.kernel'] found in /var/lib/ironic/httpboot
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.5.20.12', 34288), raddr=('10.5.20.12', 13000)>

Means you have missed to upload the image for your undercloud. Check this section: 4.8.1. Single CPU architecture overclouds.

In summary, you need to upload your images, for the CPU architecture you will have introspected.

[stack@director ~]$ source ~/stackrc
(undercloud) [stack@director ~]$ sudo dnf install rhosp-director-images rhosp-director-images-ipa
(undercloud) [stack@director ~]$ cd ~/images
(undercloud) [stack@director images]$ for i in /usr/share/rhosp-director-images/overcloud-full-latest-16.1.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-16.1.tar; do tar -xvf $i; done
(undercloud) [stack@director images]$ openstack overcloud image upload --image-path /home/stack/images/

Failed to bind DHCP server socket

My introspection was processing but was getting forever stuck trying to finish

(undercloud) [stack@undercloud ~]$ openstack overcloud node import --introspect --provide overcloud-compute1.json
Waiting for messages on queue 'tripleo' with no timeout.
 
 
1 node(s) successfully moved to the "manageable" state.
Successfully registered node UUID d954af44-a12e-4b8c-b377-146d82cbc0ad
Waiting for introspection to finish...
Waiting for messages on queue 'tripleo' with no timeout.
Introspection of node attempt failed:d954af44-a12e-4b8c-b377-146d82cbc0ad.
Retrying 1 nodes that failed introspection. Attempt 1 of 3
Introspection of node attempt failed:d954af44-a12e-4b8c-b377-146d82cbc0ad.
Retrying 1 nodes that failed introspection. Attempt 2 of 3
Introspection of node attempt failed:d954af44-a12e-4b8c-b377-146d82cbc0ad.
Retrying 1 nodes that failed introspection. Attempt 3 of 3

I did a tcpdump (“sudo tcpdump -i br-ctlplane”) at br-ctlplane (the bridge used by undercloud) and I could see the following

10:41:11.363487 IP undercloud.vim.nokialab.net > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 103, authtype none, intvl 1s, length 20
10:41:11.363584 IP undercloud.vim.nokialab.net > 224.0.0.18: VRRPv2, Advertisement, vrid 52, prio 103, authtype none, intvl 1s, length 20
10:41:12.364052 IP undercloud.vim.nokialab.net > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 103, authtype none, intvl 1s, length 20
10:41:12.364186 IP undercloud.vim.nokialab.net > 224.0.0.18: VRRPv2, Advertisement, vrid 52, prio 103, authtype none, intvl 1s, length 20
10:41:12.676965 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:5e:68:b1 (oui Unknown), length 396

I means the VM was sending the DHCP request and didn’t get answer from the undercloud DHCP service.

Then, I’ve realized the service ‘tripleo_ironic_inspector_dnsmasq.service’ was down. And I couldn’t make it work because some issue with t he container ‘ironic_inspector_dnsmasq’ with the log message /var/log/ironic-inspector/dnsmasq.log (I checked that inside the container) “failed to bind DHCP server socket: Address already in use”

Finally, I’ve realized I got a libvirt network using dnsmasq and blocking the port. After I remove it everything worked as a charm.

Thanks for reading: “IPMI trick for undercloud via VirtualBMC”

IPMI trick for undercloud via VirtualBMC: Information Source

How to deploy TripleO Overcloud (Controller , Compute) on CentOS 7 VMs