My tips and impression on RHCSA and RHCE exams

Posted by root on June 23, 2011 Comments Off

Last friday I took two exams – EX200 and EX300.  I didn`t take any courses as I was preparing by myself at home. I must say that it was more challenging that 4 years earlier when I had taken my previous RHCE exam. First of all RHEL6 gained new features compared to RHEL4. Second thing is that they added pretty much more tasks and objectives in exam requirements. You can find them in prep guides available here (for RHCSA) and here (for RHCE).

And this is my first tip – use these guides and follow all of objectives. For example if there`s a task about iSCSI initiator then learn and make some exercises. You have to follow all of these objectives. It`s especially important to people like me who didn`t take any course before, because it`s harder to make yourself some excercises. I was using two virtual machines for this purpose – one of them was a server with some services (nfs, ldap, iscsi target, etc..) and one was a client.

Second thing – don`t forget about SELinux. All of services must be SELinux enabled. Do not turn this off during your study. If you`re not familiar with SELinux then read this documentation – it`s really good and helpful. I recommend reading it at least twice and check all of described commands/tools.

Documentation. I think that Red Hat guys did a really good job creating a bunch of nice docs available on their site.  I recommend reading all of them as they describe most of exam objectives with nice examples. Unfortunately it`s not available during the exam and you must also know how to use manual pages (e.g. man httpd_selinux – describes options on selinux configuration of httpd service) and use it. Also config examples in /usr/shared/doc/* are pretty useful. That`s a very nice thing in Red Hat exams – when you forgot some detail you can always use built-in documentation. They don`t require you to remember all of options although you must know how particular services work. So my tip is this – learn how to find info on something using documentation available in you system rather than just ask google like you (probably) always do :-)

And last but not least tip – get yourself a good sleep night before exam. I didn`t take any notes nor my laptop – try to relax and think positive. If you learn  and practice then everything will go smoothly.

Of course I have passed both exams and got both RHCSA and RHCE certificates :-) In my personal opinion I consider RHCSA to be more challenging than RHCE – just look at exam objectives. RHCE adds SElinux and some basic configuration of network services while RHCSA covers a lot of other things. I recommend taking two exams on one day – it`s not that scary as you think.

Good luck to all of you preparing and studying! I`m starting preparation for RHCA exams. I`ll try to write some tips on them too. Stay tuned.

RHEL6 and ldap server with ssl

Posted by root on June 13, 2011 Comments Off

During my preparation for RHCSA+RHCE exams (I have to recertificate as I passed my previous RHCE on RHEL4) I encountered some serious problems with ssl and openldap (slapd) server.

I created self-signed certificate:

openssl req -new -x509 -nodes -out /etc/pki/tls/certs/slapd.crt -keyout /etc/pki/tls/certs/slapd.key -days 365

and added following lines to /etc/openldap/slapd.d/cn=config/olcDatabase={1}bdb.ldif

olcTLSCACertificateFile: /etc/pki/tls/certs/slapd.crt
olcTLSCertificateKeyFile: /etc/pki/tls/certs/slapd.key

Everything seemed to work fine, but I couldn`t query ldap using ldaps uri – I got the following error message:

ldapsearch -x  -H ldaps://localhost -b 'dc=example,dc=com'
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)

After short and intensive investigation, reading dozens of howto pages I founded the cause – it turned out that in RHEL6 you can`t use self-signed certificates. Well, maybe it`s not that they forbid you to do this, but it`s not as simple as it was in previous versions. Now you have to explicitly add your certificate so that is no longer unknown. There`s even more – you have to configure your Common Name field of the certificate so it matches exactly your server`s name.

So eventually to solve this issue I had to confgure ldap tools/clients and add path to server`s certificate to /etc/openldap/ldap.conf:

TLS_CACERT /etc/pki/tls/certs/slapd.crt

So nothing is the same now – RHEL6 brings new features, new tools and new problems :-) It`s time to learn and adapt.

Problem with grub – can`t find stage1

Posted by root on April 7, 2011 Comments Off

If you were moving system from one machine to another (e.g. physical to virtual machine) and you stuck on installing grub to mbr:

root (hd0,0)
setup (hd0)
 Checking if "/boot/grub/stage1" exists... no
 Checking if "/grub/stage1" exists... no

then I have a solution for you. If you checked dozens of pages (like I did) that say there`s a problem with your grub files and you checked them hundred times that everything is fine (files were there, you can still install grub on the source system using exactly the same set of files) then do not loose hope :-) You might want to check if your grub is able to read your filesystem. Maybe you created fs using newer tools. I was moving Fedora Core 3 using sysrescuecd. I created partition and filesystem using tools available on that system. It seems that probably there were minor changes between ext3 fs created on FC3 and sysrecurecd and therefore after chrooting to FC3 environment I couldn`t install grub.

The solution is quite simple – do it outside chroot environment!

Once you have mounted your root partition with /boot install mbr using grub from your livecd environment. In my case I had it mounted on /mnt/custom so I had to enter the following command to install grub on /dev/sda:

grub-install --root-directory=/mnt/custom /dev/sda

I didn`t find a way to do this using grub command line, but using grub-install I was able to boot my system and that was what I needed.

List installed dpkg and rpm packages with their size

Posted by root on November 3, 2010 Comments Off

Some time ago I found a small script dpkg-du used to list dpkg packages and size of files that belongs to them. Unfortunately I couldn`t find it now when I needed it, but few minutes with manual brought me to this trivial oneliner:

dpkg-query -W -f='${Installed-Size} ${Package}\n'

And here`s a rpm version:

rpm -qa --queryformat '%{SIZE} %{NAME}\n'

Permission denied when mounting nfs share

Posted by root on June 10, 2010 Comments Off

I had a problem with nfs share today. I couldn`t mount it on remote server. It failed with the following error:


mount: xxx:/yyy failed, reason given by server: Permission denied

I checked the configuration dozens of times – it was good, and the nfs server granted access to the share. The following info appeared in logs:


.. authenticated mount request from 1.1.1.1 for /yyy

So everything looked ok, but it didn`t work. The problem was a pseudo filesystem nfsd which should be mounted in /proc/fs/nfsd, but for some reason it wasn`t. I`ve added the following entry to /etc/fstab


none /proc/fs/nfsd nfsd defaults 0 0


and mounted it:


mount /proc/fs/nfsd

After that I was able to mount the nfs share. It took me some time to figure it out and I think that there should be more specific error messages, because the one with permission denied is quite confusing.

Loading mysql database from compressed dump

Posted by root on April 9, 2010 Comments Off

Today I had to load database on mysql. There`s nothing new or exciting about it, but I encountered few problems. First one is that dump was quite big (few gigabytes) and it was compressed with gzip. (G)Unzipping it simply to a file would take some time, waste space on disk and it wasn`t the right way ;-) So here how I managed to decompress it and load it at the same time:

gunzip -c mydbdump.sql.gz|mysql -umyuser -pmypass mydb

But then I found another problem – in my dump there was hardcoded database name. I wasn`t recovering that database, but just wanted to load it to a diffrent one. Names weren`t the same so it failed to load. I looked at the begining of that file using head command, as opening so huge file in vi(m) would probably kill the server. I found that there was two sql commands that creates database itself and use it (sql use command). So with a little help of sed I managed to modify my command so it looked like this:

gunzip -c mydbdump.sql.gz| sed -e '1,30s/old_dbname/mydb/'| mysql -umyuser -pmypass mydb

I limited sed`s search&replace to first 30 lines, because database name was at the start of the dump file and I didn`t want to mess with the rest of the file :-)

Advantages of software raid

Posted by root on April 2, 2010 Comments Off

Why using software raid when there are plenty hardware raid controllers with dedicated processors for checksum calculations, dedicated memory, cache and bbu? Well there is one major advantage of software raid solutions – hardware independence.
Recently we had a simultaneously crash of two disks in raid5 on 3ware 9550-sxu controller. That`s a real bad luck :-> Of course we recovered data from backups, but before that we tried to rebuild array. We discovered that one dik had crashed, but the other had just few bad sectors. As an experiment we cloned that disk without these few sectors and tried to rebuild the array. Unfortunatelly controller didn`t recognize that disk as a member of the array. It turned out that 3ware controller appends serial numbers of all members of array at the end of disks with a checksum (we tried to replace serial number of cloned disk with no luck).
So how would it look on software raid? Much easier. Linux for example writes only array identificator (UUID) so disk cloning is possible. Furthermore disks can be transfered to another hardware. With 3ware controller in case of its failure we would have to find another 3ware controller (probably the same model).
So a lesson has been taught – don`t rely on hardware raid, use raid6 when possible and keep your backups safe because they may save your ass someday :-)

HOWTO Massive virtual hosting on tomcat

Posted by root on January 7, 2010 Comments Off

I found out today that tomcat virtual hosts are pretty lame. I thought that these all java based tomcat`s fancy plugins are much more powerful than old, simpleapache httpd. Well they aren`t. I wasn`t able to configure massive virtual hosting for servlets. I wanted to run diffrent web application based on domain name that comes in URL. For example myapp1.example.com would run application myapp1, myapp2.example.com would run myapp2, etc. And there could be hundreds of them so I didn`t want to add hundreds of entries in server.xml config file. I`m a lazy person ;-)

In order to configure virtual hosting based on above assumptions I used apache http server as a proxy to tomcat.
First you need to add wildcard records to your domain (example.com) so that records of all its subdomains can be resolved to IP address of your server. Following record should be added to your bind server zone config:

*   IN   A   1.2.3.4

where 1.2.3.4 is IP address of your server.

Next you need to configure httpd server. Please make sure that you have ajp proxy module installed on your server, as connections to tomcat are based on AJP protocol. On CentOS/RHEL 5 this module is included in standard httpd package (see /etc/httpd/conf.d/proxy_ajp.conf).
Now you need to create configuration for your virual hosts. I created a new file /etc/httpd/conf.d/tomcat-vhosting.conf:

UseCanonicalName Off
RewriteEngine On

# vhost map using perl script
RewriteMap vhost prg:/usr/local/bin/apache-getvhost.pl

# do no rewrite restricted names
RewriteCond %{SERVER_NAME} !^docs\.
RewriteCond %{SERVER_NAME} !^examples\.
RewriteCond %{SERVER_NAME} !^host-manager\.
RewriteCond %{SERVER_NAME} !^ROOT\.

# rewrite it
RewriteRule ^/(.*)$ ajp://localhost:8009/${vhost:%{SERVER_NAME}}/$1 [P]

Quite simple and cool, isn`t it? :-) I`m sure that you probably expected VirtualHost directives, but all you need is a powerfull rewriting feature of apache. This configuration allows to access myapp application located in tomcat`s webapps directory via http://myapp.example.com.
I used custom rewrite map which is a simple perl script. All it does is extract subdomain from server name based on URL.

Put the following in /usr/local/bin/apache-getvhost.pl

#!/usr/bin/perl

$| = 1;

while (<STDIN>) {
  if (/(.*?)\.example\.com/)      {
      print $1."\n";
  } else {
      print $_."\n";
  }
}

and make it executable

chmod +x /usr/local/bin/apache-getvhost.pl

Now all you need to do is provide some applications to tomcat.

Booting Xen paravirtualized vm in rescue mode

Posted by root on December 22, 2009 Comments Off

This is my first post in english. I am sorry if there are some mistakes – it is not my native language :-)

Recently I had to rename volume group in one of my virtual machines. It is based on CentOS (both dom0 and domU) and it was paravirtualized. I couldn`t do it on running system because root partition was on logical volume. I had to boot the machine in rescue mode. I had no idea how to boot from CD/DVD iso image – AFAIK it is impossible on pv guests. I had to boot vm directly from kernel and initrd used in installation (both are xen aware). So I copied  them to /tmp/xen from /images/xen/ on CD/DVD installation disc.

Then I had to comment out the following line from vm`s config file to bypass pygrub bootloader:

#bootloader = "/usr/bin/pygrub"

Now I needed to tell my vm to use kernel and initrd I had previously copied so I added the following lines:

kernel = "/tmp/xen/vmlinuz"
ramdisk = "/tmp/xen/initrd.img"
extra = "rescue method=http://192.168.0.2/install/centos/"

The last line passes extra arguments to kernel. There is a rescue keyword and a method which tells anaconda installer to get install files from my http server.

After that I was able to boot vm in rescue and rename my LVM virtual group.

Klastrowy LVM i Xen

Posted by root on November 23, 2009 Comments Off

Bardzo intensywnie wykorzystuję wirtualizację jaką oferuje CentOS, czyli opensource`ową wersję Xena. W połączeniu z LVM można bardzo efektywnie zarządzać maszynami m.in. poprzez snapshoty. Dodatkowo zastosowanie LVM jako backendu storage`owego owocuje wysoką wydajnością.

Xen posiada również możliwość migracji maszyn. Funkcja ta znana jako live migration jest odpowiednikiem vmotion dla produktów VMware. Można ją wykorzystać jedynie w przypadku, gdy wszystkie serwery dom0 mają dostęp do współdzielonego storage`u. Najłatwiej jest wykorzystać zasób NFS podmontowany na wszystkich serwerach, a maszyny wirtualne instalować na plikach (obrazy dysków) tam umieszczonych. Oczywiście tracimy na wydajności, a poza tym NFS nie wydaje się dobrym pomysłem dla większych instalacji.
I tu z pomocą przychodzi klastrowa odmiana LVM (CLVM). W RHEL/CentOS demon zarządzający klastrową częścią LVM (clvmd) komunikuje się z menadżerem klastra RHCS (Red Hat Cluster Suite). Mi osobiście to niezbyt pasuje. Nie tylko chyba ja uważam, że twór RHCS nie jest jeszcze stabilny ani też łatwy w konfiguracji i zarządzaniu/utrzymaniu. Ale to już chyba temat na odrębny wpis. Ku czemu innemu zmierzam. Mianowicie bardzo zainteresował mnie ten wpis. Okazuje się, że jest możliwe zmuszenie CLVM do działania bez skonfigurowanego RHCS! Należy zamiast tego użyć openais, który i tak wchodzi w skład RHCS. Jako odrębna część jest jednak o wiele przyjemniejsza w zarządzaniu i konfiguracji. Zachęcony wspomnianym wpisem przystąpiłem do pracy. Udało mi się przerobić paczkę lvm2-cluster, tak aby linkowała się z openais. Wpis ten byłby chyba pierwszym howto w sieci opisującym ten proces dla systemów RHEL/CentOS. Niestety w trakcie działania pojawiają się wycieki w pamięci w demonie aisexec. Przy każdorazowym wysłaniu wiadomości do aisexec przez demon clvmd ten pierwszy rezerwuje około 6MB pamięci i nie zwalnia jej poprawnie. Kilkunastogodzinne próby nie przyniosły rozwiązania. Wielka szkoda. Byłaby to świetna wiadomość i dalsze pole do eksperymentowania. Liczę, że uda mi się jeszcze powrócić do tematu, a o rezultatach postaram się niezwłocznie poinformować.