A few notes on Oracle VM

This is an old post I kept as draft 8 months before I published it...

I've been playing around with Oracle Virtual Manager (OVM) version 2.2 lately. This is collection of a few notes, not all directly related to OVM. The project started with building the server from parts I hoped would play along (they do). The CPU is an Intel i7-930, 12GB of RAM and two Western Digital 1.5TB disks (WD15EARS-00Z5B1). Current version of OVM (2.2) uses update 3 of OEL 5. The kernel lags behind a bit and support for certain wirless card is not included. The AR5008 Wireless chip from Atheros (a Dlink card) was not supported; in order to avoid more cables in the house I created a bridge between the server and another PC nearby.

The installation of OVM went smooth, but the disks are a disgrace with a horrible I/O performance until you do the partitioning right (each partition should start on a sector divisible by 4096 bytes). Meaning I had to choose a custom layout of the partitions and file systems during installation.

The OVM manager was installed in a guest based on a OVM template downloaded from Oracle.

Configuration of OVM Manager fails because host names cannot be resolved:

The server resides on a private network (192.168.x.x) without any DNS; if you get error messages indicating connection problems to the OVM server then make sure the hostname is defined in /etc/hosts, also make sure the hostname is not resolved to localhost (127.0.0.1); which was the case on a fresh installation of OVM Server. This should be done both on the OVM Server and in the guest hosting OVM Manager. (If the initial configuration of OVM Manager fails, connect with ssh and fix /etc/hosts before restarting).

Connecting to the Manager database from afar:
The listener binds to standard port 1521 on localhost. In case you want to connect to the database over SQL*Net you need to change listener.ora and replace localhost with a public IP address/name of the Manager's host. Then you need to add a line to /etc/sysconfig/iptables allowing remote access to the listener:

-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 1521 -j ACCEPT

Configuring yum:
I added the following lines to the file /etc/yum.repos.d/public-yum-el5.repo

[el5_u3_base]
name=Enterprise Linux $releasever U3 - $basearch - base
baseurl=http://public-yum.oracle.com/repo/EnterpriseLinux/EL5/3/base/$basearch/
gpgkey=http://public-yum.oracle.com/RPM-GPG-KEY-oracle-el5
gpgcheck=1
enabled=1

[ovm22_2.2.1_base]
name=OracleVM 2 - 2.2.1 - $basearch - base
baseurl=http://public-yum.oracle.com/repo/OracleVM/OVM2/2.2.1/base/$basearch/
gpgkey=http://public-yum.oracle.com/RPM-GPG-KEY-oracle-el5
gpgcheck=1
enabled=1

Then gcc and others can be installed with

yum install gcc

Tuning network
After the disk performance was improved I started to get annoyed about the slow refresh of VNC. The following command reported an increasing queue length when I interacted with a GUI through VNC:

netstat -o -t -c

This Guide was helpful.

Using physical devices for storage
I could not find any way to add a physical device to a guest in the Manager. From the documentation though it is clear how one goes about to do this. Disadvantage is that the added physical device does not show up in Manager after. Anyway, the performance gain outweighs that. I had one disk that I gave entirely to a volume group and then created a logical device with pvcreate, vgcreate and lvcreate.

To add the logical volume as a physical volume to the guest:

xm block-attach 24 phy:/dev/vg00/oradb1_asm1 xvdc 'w'

24 is the id of the guest as reported by xm list, xvdc is the device name at the guest (/dev/xvdc), and 'w' means read/write. At the guest this disk device needs to partitioned with fdisk or similar as with any other physical device.

To make the change permanent the following line (in bold) was added to the disk configuration in vm.cfg:

disk = ['file:/var/ovs/mount/3FEB9655D88E4177B8571AFCF81C5B8A/running_pool/50_oel2/System.img,xvda,w',
'phy:/dev/vg00/oel2_1,xvdb,w',
'file:/var/ovs/mount/3FEB9655D88E4177B8571AFCF81C5B8A/sharedDisk/software.img,xvdc,w!',
]

Monitoring with sar -d showed that the service time substantially lower (1/10) with the physical device, even when the physical device had almost max utilization on writing.

VNC and maximum resolution
Any changes made to the screen resolution was quietly ignored and the connection to the vnc server that is started from dom0 was unreliable. After a while I gave up and started a new vnc server from inside the guest.


Update: Using OVM for my own lab was not optimal, I installed Fedora 15 which among other things supports the wireless card without any hassle. I plan to install Virtual Box on top of it for my lab needs.

Permission denied when adding ASM disk

This is a simple one, a reminder till next time I forget this, since stuff encountered after a Google-search led me astray...

Just in case you try to add disks to an existing ASM disk group using syntax like this:

alter diskgroup data
add disk '/dev/xvdd1'
add disk '/dev/xvde1';

You may get error messages like these:

ORA-15032: not all alterations performed
ORA-15031: disk specification '/dev/xvde1' matches no disks
ORA-15025: could not open disk '/dev/xvde1'
ORA-15056: additional error message
Linux-x86_64 Error: 13: Permission denied
Additional information: 42
Additional information: -1614995373
Additional information: 1730318408
ORA-15031: disk specification '/dev/xvdd1' matches no disks
ORA-15025: could not open disk '/dev/xvdd1'
ORA-15056: additional error message
Linux-x86_64 Error: 13: Permission denied
Additional information: 42
Additional information: 87328232
Additional information: 8192

Solution: Use the ASM disk names as given when creating them with oracleasm:

alter diskgroup data
add disk 'ORCL:VOL3'
add disk 'ORCL:VOL4';

Here VOL3 and VOL4 are the names given when creating the ASM-disk with oracleasm:

oracleasm createdisk VOL3 /dev/xvdd1
oracleasm createdisk VOL4 /dev/xvde1

The initialization parameter asm_diskstring is empty as is default.

The error messages from OS (Linux in this case) may lead you to think that there is something wrong with the owner or attributes of the devices you've just created, even they have the same permissions as the devices previously allocated to ASM... Then when you against reason set the owner and group to be like 'oracle:dba' the error message from Linux does not show up, but you will still get error messages like this:

ORA-15032: not all alterations performed
ORA-15031: disk specification '/dev/xvde1' matches no disks
ORA-15014: location '/dev/xvde1' is not in the discovery set
ORA-15031: disk specification '/dev/xvdd1' matches no disks
ORA-15014: location '/dev/xvdd1' is not in the discovery set

A note on MOS with id 603210.1 is one example of this confusion, it contains this command indicating that it is OK to use the physical device name (which it may be in certain environments for what I know):

create diskgroup DATADG external redundancy disk '/dev/sda14'; 

I think this is one of several reasons why certain un*x guys used to old and mature volume managers and file systems do not easily fall in love with ASM... Once one has passed these initial obstacles I find ASM OK to work with, even for non-RAC databases. It usually shortens the path from getting a LUN allocated on the SAN or whatever until more free space is available to the tablespace in the database.

[This testing was done on a Oracle VM guest, created from an OVM template downloaded from Oracle. It comes with OEL 5 64-bit and version 10.2.0.4 of the database. Oracle calls this OS for JeOS (Just enough OS), but for some reason it does not come with the 'man' command, so for me it is JneOS.]