Rebuilding my home lab: Part 3

In part 1 of this series we looked at the hardware configuration for the new lab, and installed the base OS via kickstart from Satellite. In part 2 we installed and configured Gluster to provide shared storage between the hosts. In this part we will look at installing Red Hat Virtualisation and performing the initial configuration.

Red Hat Virtualisation can be installed in a number of ways. For my usage, the ideal way would be to make use of Red Hat Hyperconverged Infrastructure (RHHI), which uses an installation method that provides a wizard to install and configure both Gluster and RHV. As I mentioned in part 2, this unfortunately wasn’t possible for me due to the architecture and version mismatch on my Raspberry Pi Gluster arbiter node.

Another method allows for RHV to be installed using pre-existing resources, which we now have. The RHV Manager (RHV-M) is the first component that needs to be installed, and there are two methods to do this as well. The first is to have the manager on a dedicated host, the second is to have the manager ‘self-hosted’. Self Hosted means that the RHV-M server is hosted as a guest on a RHV hypervisor, in a similar way that VMWare operate their vCenter Appliance (VCA). I am going to be using the Self-Hosted option, using the RHV 4.1 Self Hosted Engine guide.


Before installing any RHV components, there are some prerequisite checks to perform to ensure everything is in order.


When we are using Gluster as our Storage Domains for RHV, the installation manual recommends assigning the volumes to the ‘virt’ volume group.  This applies optimisations to the volume for virtualisation. Unfortunately, due to the version mismatch between our main Gluster nodes (v3.8.4) and our Arbiter node (v3.8.8) this command fails.

[root@baremetal1 ~]# gluster volume set rhvm-dom group virt
volume set: failed: Staging failed on storage3. Error: option: network.remote-dio op-version mismatch

When we have a look at the ‘virt’ profile, we can see that it is actually just a grouping of volume settings to apply.

[root@baremetal1 ~]# cat /var/lib/glusterd/groups/virt 

Lets apply what we can individually to the volumes – I am also applying the other volume configuration requirements from the RHV installation guide here too.

The two parameters from the ‘virt’ group that won’t set due to the version mismatch are  network.remote-dio  and  cluster.server-quorum-type

[root@baremetal1 ~]# for vol in rhvm-dom data-dom; do
  gluster volume set ${vol} performance.quick-read off;
  gluster volume set ${vol} off;
  gluster volume set ${vol} off;
  gluster volume set ${vol} performance.low-prio-threads 32;
  gluster volume set ${vol} cluster.eager-lock enable;
  gluster volume set ${vol} cluster.quorum-type auto;
  gluster volume set ${vol} full;
  gluster volume set ${vol} cluster.locking-scheme granular;
  gluster volume set ${vol} cluster.shd-max-threads 8;
  gluster volume set ${vol} cluster.shd-wait-qlength 10000;
  gluster volume set ${vol} features.shard on;
  gluster volume set ${vol} user.cifs off;
  gluster volume set ${vol} cluster.quorum-type auto;
  gluster volume set ${vol} 10;
  gluster volume set ${vol} auth.allow \*;
  gluster volume set ${vol} storage.owner-uid 36;
  gluster volume set ${vol} storage.owner-gid 36;
  gluster volume set ${vol} server.allow-insecure on;

Great! We have them all applied now except for the two that complain about the versions.  Since these are RECOMMENDATIONS, I’m not going to worry too much about the two options that failed to set (yet).

We also need to enable root ssh using keys between our nodes. First create an ssh keypair as the root user on all three hosts. This will allow the RHV Manager to query the gluster status.

[root@baremetal1 ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/

[root@baremetal2 ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/

root@glusterpi:~# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/

Next, copy the public key from each host to the other two  (I had to enable root ssh on the Raspberry pi in /etc/ssh/sshd_config). We are not worried about the RPi connecting to the RHEL hosts, as the RHV Manager won’t be able to use it for querying anyway.

[root@baremetal1 ~]# ssh-copy-id
[root@baremetal1 ~]# ssh-copy-id

[root@baremetal2 ~]# ssh-copy-id
[root@baremetal2 ~]# ssh-copy-id


We need to ensure that we have a DNS entry for our RHV Manager. This needs to resolve during the install – failure to resolve will halt the installation during the wizard.  For my lab, I am have defined  as my manager (My baremetal hosts and RHV-M are considered CORE infrastructure, the VMs will be in my LAB domain). Testing that forward and reverse resolution works correctly:

[root@baremetal1 ~]# host has address

[root@baremetal1 ~]# host domain name pointer

Installing the Self Hosted Engine

Now we are ready to install our RHV-M Self-Hosted engine. First we need to enable the repositories. Again, since my RHEL hosts are subscribed to my Satellite, I have the required repository and subscription available for use by them:

[root@baremetal1 ~]# subscription-manager repos --enable=rhel-7-server-rhv-4-mgmt-agent-rpms

[root@baremetal2 ~]# subscription-manager repos --enable=rhel-7-server-rhv-4-mgmt-agent-rpms

Next we need to install the necessary packages on both nodes. I originally elected to use the installation method that used the cockpit installer, so I installed the ovirt dashboard, and enable the cockpit service. Even though I am not using this for the installation, it may be useful later. I am installing the vdsm-gluster package here as well – this is important!  Without this package, the RHV Manager won’t have visibility of our gluster volumes.

[root@baremetal1 ~]# yum -y install screen cockpit cockpit-ovirt-dashboard vdsm-gluster
[root@baremetal1 ~]# systemctl start cockpit

[root@baremetal2 ~]# yum -y install screen cockpit cockpit-ovirt-dashboard vdsm-gluster
[root@baremetal2 ~]# systemctl start cockpit

The reason I am not using the Cockpit based installation is that we cannot specify backup Gluster hosts for the engine storage volume (rhvm-dom) – this leaves us with a single point of failure in our cluster as if the host we define as the storage mount goes down, the whole cluster goes down. Turns out that using the good ol’ CLI based installer allows us to pull in a customisation file so that we can define redundancy on our storage.

On the baremetal hosts (our RHEL hosts that will be the hypervisors) we need to ensure that the network ports required are open in the firewall.  Failure to open these will result in the installer hanging checking the VDSM status of the manager VM after it has been installed. The firewalld service files required are delivered by the various RPM packages pulled in by the command above, so we can’t enable the firewall services until the package is installed.

[root@baremetal1 ~]# firewall-cmd --permanent \
    --add-service ovirt-vmconsole \
    --add-service ovirt-imageio \
    --add-service vdsm \
    --add-service libvirt-tls \
    --add-service cockpit
[root@baremetal1 ~]# firewall-cmd --reload

[root@baremetal2 ~]# firewall-cmd --permanent \
    --add-service ovirt-vmconsole \
    --add-service ovirt-imageio \
    --add-service vdsm \
    --add-service libvirt-tls \
    --add-service cockpit 
[root@baremetal2 ~]# firewall-cmd --reload

Now, lets make our storage configuration file for our override. What this is defining is that our primary storage point for the Hosted Engine is storage1:/rhvm-dom (our Gluster brick), but we have backup servers at storage2 and storage3. If storage1 becomes unavailable, the others will be used to mount the volume, keeping our cluster alive.

[root@baremetal1 ~]# cat << EOF > /root/storage.conf

Now we can start the deployment of our self-hosted engine. We run this in a ‘screen’ session so that if our connection drops out we can reconnect – the deploy takes some time to complete.

[root@baremetal1 ~]# screen
[root@baremetal1 ~]# hosted-engine --deploy --config-append=/root/storage.conf

We need to complete a series of questions from the installation wizard:

  • Confirm that we really do want to install this host as a RHV hypervisor
  • Confirm that we want to download the rhvm-appliance RPM
  • Define storage type (glusterfs, iscsi, fc, nfs3, nfs4) – We want glusterfs (The gluster volume path is defined in our answer file addon above)
  • We DO want the installer to configure our Gluster environment – Although we have already done this manually, this adds the hooks into the RHV Manager.
  • We do NOT want the installer to configure iptables for us. We are not installing a pure RHV+RHGS or RHHI configuration – we don’t want the installer to screw with our setup.
  • We need to confirm a pingable gateway. In this case the gateway on my core network was auto selected – this is fine.
  • Define which NIC will be the oVirt management network. I want my eno1 (core network, untagged) to be used for management.
  • We are asked to confirm the OVA image to install for the engine. The wizard downloaded this earlier.
  • We need to confirm the console type for the manager VM. I chose Spice instead of the default VNC.
  • The installer want to use cloud-init to configure the VM – that’s fine by me.
  • We are prompted to supply or generate a cloud init – select Generate.
  • Enter the FQDN of manager VM (This MUST already resolve in DNS)
  • Enter Engine VM Domain name – should default to the domain of the FQDN above
  • Do we want to automatically execute the engine-setup – Yes
  • Automatically restart as a monitored service after engine setup – Yes
  • Enter/Confirm root password for appliance (for console/ssh)
  • Enter root SSH public key (optional)
  • Enable root SSH for the appliance – Yes (Only because I didn’t define a key)
  • Size of VM disk (Default 58Gb – remember our Gluster volume is 100Gb)
  • Size of appliance memory (Default 16Gb – that’s fine)
  • CPU type (Select default – mine was Haswell-noTSX)
  • Number of vCPUs (Default is 4 – we can reduce this later)
  • MAC Address (Accept the generated default)
  • Network setup – DHCP or Static. I chose Static so I know where this VM is.
  • Define the IP address – make sure you use the one assigned to the hostname in DNS!
  • Confirm the DNS addresses
  • Update /etc/hosts – I chose YES here, my DNS should be reliable, but you never know….
  • Enter/Confirm Engine admin password (for the WebUI)
  • Define the SMTP Server address for notifications
  • Define the source Email address for notifications
  • Define the recipient Email address for notifications

After completing the wizard, the installation of the RHV Manager will continue – it will take some time.  Hopefully the installation will complete, and you will see a successfully deployed message.

[ INFO  ] Waiting for the host to become operational in the engine. This may take several minutes...
[ INFO  ] The VDSM Host is now operational
[ INFO  ] Saving hosted-engine configuration on the shared storage domain
[ INFO  ] Shutting down the engine VM
[ INFO  ] Enabling and starting HA services
[ INFO  ] Stage: Clean up
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20180415144208.conf'
[ INFO  ] Generating answer file '/etc/ovirt-hosted-engine/answers.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ INFO  ] Hosted Engine successfully deployed

And, browsing to the RHV manager FQDN, we see the landing page, so success there too! (It may take a few moments for the VM to start up once the installation has completed)

Logging into our RHV-M Administration Portal (as the admin user) we can see our baremetal1 host listed as part of the default cluster. Note that the RHV-M Hosted Engine VM itself is not yet seen within the cluster – this will be configured a little later.

We can also see our two Gluster volumes, although we only see one brick (but its status is UP) and the status details are incomplete.

In the next part of this series we will continue with the configuration of our RHV environment.

Leave a Reply

Your email address will not be published. Required fields are marked *