Testing MPICH2 cluster on Ubuntu 12.04

All have been already explained many times, and it's not difficult to find information. But there were some parts not very clear, at the end for me it doesn't show how to test if the cluster is doing it's job on different hosts. The point is you can run Mpich on only one host after all, but this is not very useful. So first let set the MPICH. My sources of information were:

http://wiki.mcs.anl.gov/mpich2/index.php/Using_the_Hydra_Process_Manager
http://www.petur.eu/blog/?p=59
http://webappl.blogspot.ca/2011/05/setting-up-mpich2-cluster-with-ubuntu.html

Well, how exactly I set my cluster? The steps are below, detailed information on all of them can be found on the sources I published, here is just the list:

1. Install MPICH2 - sudo aptitude install mpich2
2. Change /etc/hosts file, include hosts information for the cluster hosts. Don't forget to remove the 127.0.1.1 address for your hostname.
3. Enable ssh-key login for some (cluster) user.
4. Create mpich host file. I used mpd.hosts, but this is not the default name, and I don't know what it is (or if there is a default name at all). I just executed mpiexec with -f parameter.
5. Set .mpd.conf with the password.

That's all to have functional cluster. But how to prove the cluster is populated around the hosts? I used three tools to demonstrate this.

1. mpi_hello.c There are many variations of this simple program, just choose one of them.
2. John the ripper as described in petur.eu blog
3. tcpdump

Now the results. Compile mpi_hello.c and John. On one of the hosts run tcpdump host other_host. On other execute mpi_hello like this:

cluster@d:~$ mpiexec -f mpd.hosts -n 4 ./mpi_hello
Hello from processor 2 of 4
Hello from processor 0 of 4
Hello from processor 3 of 4
Hello from processor 1 of 4

Well, that will work even if there is one host, and give the same result. But the real proof is the tcpdump output on the other host:

tcpdump host host_2
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
16:54:34.439518 ARP, Request who-has x.lan tell d.lan, length 46
16:54:34.439588 ARP, Reply x.lan is-at 08:00:27:dc:9d:cc (oui Unknown), length 28
16:54:34.440101 IP d.lan.33199 > x.lan.ssh: Flags [S], seq 922409158, win 14600, options [mss 1460,sackOK,TS val 2205427 ecr 0,nop,wscale 3], length 0
16:54:34.440157 IP x.lan.ssh > d.lan.33199: Flags [S.], seq 1120118912, ack 922409159, win 14480, options [mss 1460,sackOK,TS val 2210524 ecr 2205427,nop,wscale 3], length 0
16:54:34.441520 IP d.lan.33199 > x.lan.ssh: Flags [.], ack 1, win 1825, options [nop,nop,TS val 2205428 ecr 2210524], length 0
16:54:34.456768 IP x.lan.ssh > d.lan.33199: Flags [P.], seq 1:40, ack 1, win 1810, options [nop,nop,TS val 2210528 ecr 2205428], length 39
16:54:34.457608 IP d.lan.33199 > x.lan.ssh: Flags [.], ack 40, win 1825, options [nop,nop,TS val 2205432 ecr 2210528], length 0
16:54:34.458207 IP d.lan.33199 > x.lan.ssh: Flags [P.], seq 1:40, ack 40, win 1825, options [nop,nop,TS val 2205432 ecr 2210528], length 39
16:54:34.458381 IP x.lan.ssh > d.lan.33199: Flags [.], ack 40, win 1810, options [nop,nop,TS val 2210529 ecr 2205432], length 0
16:54:34.460435 IP x.lan.ssh > d.lan.33199: Flags [P.], seq 40:1024, ack 40, win 1810, options [nop,nop,TS val 2210529 ecr 2205432], length 984
16:54:34.462420 IP d.lan.33199 > x.lan.ssh: Flags [P.], seq 40:1312, ack 1024, win 2071, options [nop,nop,TS val 2205433 ecr 2210529], length 1272
16:54:34.502874 IP x.lan.ssh > d.lan.33199: Flags [.], ack 1312, win 2172, options [nop,nop,TS val 2210540 ecr 2205433], length 0
--- cut ---

No need to continue, clearly the first host is talking to the second in the moment of mpi_hello execution. Let's do one more test with John. First execute standalone test, then MPI-enabled one.

cluster@d:~/src/john-1.7.2-bp17-mpi8$ run/john -format=DES -test
Benchmarking: Traditional DES [128/128 BS SSE2]... DONE
Many salts:     1112K c/s real, 1112K c/s virtual

Only one salt:  1011K c/s real, 1013K c/s virtual

cluster@d:~/src/john-1.7.2-bp17-mpi8$ mpiexec -f ~/mpd.hosts -n 4 run/john -format=DES -test
Benchmarking: Traditional DES [128/128 BS SSE2]... DONE
Many salts:     2804K c/s real, 2806K c/s virtual

Only one salt:  3790K c/s real, 3806K c/s virtual

Definitely a different results. First one is run on one core only. In my config second one is run on two hosts, two cores each. The result is not x4, but more than x2, which proves the case. Run tcpdump if you want also, or best start top on second host and watch how john kicks in like this:

 top - 17:11:50 up  2:49,  1 user,  load average: 0.10, 0.06, 0.05
Tasks:  93 total,   3 running,  90 sleeping,   0 stopped,   0 zombie
Cpu(s): 99.5%us,  0.4%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:    507796k total,   462940k used,    44856k free,    52220k buffers
Swap:  1046524k total,      272k used,  1046252k free,   330820k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                      
 6106 cluster   20   0 13248 1692 1420 R   96  0.3   0:03.05 john                                                                                          
 6105 cluster   20   0 13248 1720 1448 R   96  0.3   0:03.15 john                                                                                          
  988 lightdm   20   0 99028  11m 9492 S    1  2.3   0:35.98 lightdm-gtk-gre                                                                               
 5959 root      20   0  2832 1232  984 R    1  0.2   0:00.10 top

This is it, now you can use your cluster to do some more work.

Compile Wireshark in Ubuntu 12.04

Time to move on. 10.04 was a good LTS, now let's do the same for 12.04. The procedure is relatively the same, only few things have changed.

Download source from http://www.wireshark.org/download.html

Now the version is 1.10.2. The rest of the material is about 1.8.3, but the compilation works the same way.
Note the file image/wsicon32.xpm is missing by some reason. Before running compilation create it using  image/wsicon32.png and imagemagic like this:

$ convert image/wsicon32.png image/wsicon32.xpm

I'm also covering this issue later on the process.

Since 1.8.3 there were some additional features added, as example using c-ares, GNU ADNS and SMI MIB libraries. It also fixes http://www.wireshark.org/security/wnpa-sec-2012-26.html, http://www.wireshark.org/security/wnpa-sec-2012-27.html and http://www.wireshark.org/security/wnpa-sec-2012-29.html.

$ tar -jxf wireshark-1.8.3.tar.bz2

# read INSTALL and check dependencies
$ pkg-config glib-2.0 --modversion

$ gtk-config --version

# install dependencies
$ sudo aptitude install --add-user-tag wir libgtk2.0-dev
$ pkg-config glib-2.0 --modversion # confirm that it's working now

$ pkg-config gtk+-2.0 --modversion

# Good, let's install the rest of dependencies.
$ sudo aptitude install --add-user-tag wir libgtk2.0-dev libpcap0.8-dev bison flex libssl-dev libgnutls-dev libpcre3-dev libadns1-dev libc-ares-dev libsmi2-dev # I have libpcap0.8 installed already, you may need to install it as well.

./configure --with-ssl --with-gnutls --with-c-ares --with-libsmi

# two possible ways from here - traditional make, or Debian/Ubuntu making .deb pakages
## Use method #1 for 10.04, this way the compilation completes nowadays in this old Ubuntu version, making debian packages fail because of libraries dependencies. For making debian-package in 12.04 use #2

## 1
# this one was not tested for missing 
$ make -j4; make install # if make is followed by make debian-package the whole compilation runs from the beginning. Skip this step and go to #2 instead; using -j4 to better use my 4 core CPU

## 2
# add dependencies
$ sudo aptitude install --add-user-tag wir dpatch libtool automake1.9 autoconf autotools-dev libc-ares-dev docbook-xsl libpcre3-dev libcap-dev libgnutls-dev portaudio19-dev libkrb5-dev liblua5.1-0-dev libsmi2-dev libgeoip-dev xsltproc # adding dependent packages for .deb building, xsltproc is additional this time

# ...and make



$ if [  -f image/wsicon32.xpm ]; then convert image/wsicon32.png image/wsicon32.xpm; fi # fix missing image/wsicon32.xpm if not done already

$ make -j4 debian-package

# install (packages are one directory up)
$ sudo dpkg -i wireshark wireshark-common tshark

#now clean packages installed to resolve dependencies. This time it takes aptitude 4 steps to propose the solution we want:

$ sudo aptitude purge '?user-tag(wir)'


 The following packages will be REMOVED:   autoconf{p} ...
...
...
...  zlib1g-dev{p}
0 packages upgraded, 0 newly installed, 114 to remove and 0 not upgraded.
Need to get 0 B of archives. After unpacking 113 MB will be freed.
The following packages have unmet dependencies:
 libportaudio2 : Depends: libjack-jackd2-0 (>= 1.9.5~dfsg-14) but it is not going to be installed. or
                          libjack-0.116 which is a virtual package.
 audacious-plugins : Depends: libjack-jackd2-0 (>= 1.9.5~dfsg-14) but it is not going to be installed. or
                              libjack-0.116 which is a virtual package.
 gstreamer0.10-plugins-good : Depends: libjack-jackd2-0 (>= 1.9.5~dfsg-14) but it is not going to be installed. or
                                       libjack-0.116 which is a virtual package.
 mplayer2 : Depends: libjack-jackd2-0 (>= 1.9.5~dfsg-14) but it is not going to be installed. or
                     libjack-0.116 which is a virtual package.
 libfluidsynth1 : Depends: libjack-jackd2-0 (>= 1.9.5~dfsg-14) but it is not going to be installed. or
                           libjack-0.116 which is a virtual package.
 wireshark-common : Depends: libc-ares2 (>= 1.7.0) but it is not going to be installed.
                    Depends: liblua5.1-0 but it is not going to be installed.
                    Depends: libsmi2ldbl but it is not going to be installed.
The following actions will resolve these dependencies:

      Remove the following packages:
1)      audacious
2)      audacious-plugins
3)      gecko-mediaplayer
4)      gnome-mplayer
5)      gstreamer0.10-plugins-good
6)      guvcview
7)      libfarstream-0.1-0
8)      libfluidsynth1
9)      libportaudio2
10)     libpurple0
11)     mplayer2
12)     pidgin
13)     pidgin-microblog
14)     tshark
 15)     wireshark
16)     wireshark-common

      Leave the following dependencies unresolved:
17)     lubuntu-desktop recommends audacious
18)     lubuntu-desktop recommends audacious-plugins
19)     lubuntu-desktop recommends gecko-mediaplayer
20)     lubuntu-desktop recommends gnome-mplayer
21)     lubuntu-desktop recommends guvcview
22)     lubuntu-desktop recommends pidgin
23)     lubuntu-desktop recommends pidgin-microblog
24)     pidgin recommends gstreamer0.10-plugins-good
25)     audacious-plugins-data recommends audacious-plugins


Accept this solution? [Y/n/q/?] n
The following actions will resolve these dependencies:

     Remove the following packages:
1)     tshark
2)     wireshark
3)     wireshark-common

     Install the following packages:
4)     libjack-jackd2-0 [1.9.8~dfsg.1-1ubuntu1 (now, precise)]



Accept this solution? [Y/n/q/?] n
The following actions will resolve these dependencies:

     Remove the following packages:
1)     tshark
2)     wireshark
3)     wireshark-common

     Keep the following packages at their current version:
4)     libjack0 [1:0.121.0+svn4538-3ubuntu1 (now, precise)]



Accept this solution? [Y/n/q/?] n
The following actions will resolve these dependencies:

     Install the following packages:
1)     libjack-jackd2-0 [1.9.8~dfsg.1-1ubuntu1 (now, precise)]

      Keep the following packages at their current version:
2)     libc-ares2 [1.7.5-1 (now, precise)]
3)     liblua5.1-0 [5.1.4-12ubuntu1 (now, precise)]
4)     libsmi2ldbl [0.4.8+dfsg2-4build1 (now, precise)]



Accept this solution? [Y/n/q/?]y

# final notes:  as this was an experiment, removing wireshark actually removes all the dependencies left from previous step:

$ sudo aptitude purge wireshark wireshark-common tshark
The following packages will be REMOVED: 
  libc-ares2{u} liblua5.1-0{u} libsmi2ldbl{u} tshark{p} wireshark{p} wireshark-common{p}
0 packages upgraded, 0 newly installed, 6 to remove and 0 not upgraded.
Need to get 0 B of archives. After unpacking 53.7 MB will be freed.

# final-final notes: doing new installation on the same machine fail because of dependencies:

 $ sudo dpkg -i wireshark_1.8.3_i386.deb wireshark-common_1.8.3_i386.deb tshark_1.8.3_i386.deb
Selecting previously unselected package wireshark.
(Reading database ... 104217 files and directories currently installed.)
Unpacking wireshark (from wireshark_1.8.3_i386.deb) ...
Selecting previously unselected package wireshark-common.
Unpacking wireshark-common (from wireshark-common_1.8.3_i386.deb) ...
Selecting previously unselected package tshark.
Unpacking tshark (from tshark_1.8.3_i386.deb) ...
dpkg: dependency problems prevent configuration of wireshark-common:
 wireshark-common depends on libc-ares2 (>= 1.7.0); however:
  Package libc-ares2 is not installed.
 wireshark-common depends on liblua5.1-0; however:
  Package liblua5.1-0 is not installed.
 wireshark-common depends on libsmi2ldbl; however:
  Package libsmi2ldbl is not installed.
dpkg: error processing wireshark-common (--install):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of tshark:
 tshark depends on wireshark-common (= 1.8.3); however:
  Package wireshark-common is not configured yet.
dpkg: error processing tshark (--install):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of wireshark:
 wireshark depends on wireshark-common (= 1.8.3); however:
  Package wireshark-common is not configured yet.
dpkg: error processing wireshark (--install):
 dependency problems - leaving unconfigured
Processing triggers for desktop-file-utils ...
Processing triggers for man-db ...
Errors were encountered while processing:
 wireshark-common
 tshark
 wireshark

Hopefully we know how to fix:

$ sudo aptitude -f install
The following partially installed packages will be configured:
  tshark wireshark wireshark-common{b}
No packages will be installed, upgraded, or removed.
0 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B of archives. After unpacking 0 B will be used.
The following packages have unmet dependencies:
 wireshark-common : Depends: libc-ares2 (>= 1.7.0) but it is not going to be installed.
                    Depends: liblua5.1-0 but it is not going to be installed.
                    Depends: libsmi2ldbl but it is not going to be installed.
The following actions will resolve these dependencies:

     Remove the following packages:
1)     tshark
2)     wireshark
3)     wireshark-common



Accept this solution? [Y/n/q/?] n
 The following actions will resolve these dependencies:

     Install the following packages:
1)     libc-ares2 [1.7.5-1 (now, precise)]
2)     liblua5.1-0 [5.1.4-12ubuntu1 (now, precise)]
3)     libsmi2ldbl [0.4.8+dfsg2-4build1 (now, precise)]



Accept this solution? [Y/n/q/?] y

# and finally:

 Current status: 0 broken [-1].

aptitude fun!

We all know how to use aptitude user tags, don't we?

sudo aptitude install --add-user-tag tag_name package_name

sudo aptitude purge '?user-tag(tag_name)'

Well, that was simple. Let's do some more complicated stuff. Let's try to list all packages in our system that has user tag set:

aptitude show "?user-tag(d*)"

Got it? I didn't. You should add a letter (any letter) before your wildcard to get results. How and why it is working that way - no idea. The only thing I know is I run into it by accident. Of course it's a good idea to combine with grep:

aptitude show "?user-tag(d*)" | grep "User Tags"

Now it's more simple. We have a list of our user tags
(OK, OK, I know:

aptitude show "?user-tag(d*)" | grep "User Tags" | awk '{print $3}' | sort -u

)
But what now? Let's assume we want to remove some user tag from a package:

sudo aptitude remove-user-tag tag_name package_name

What if we want to remove user tag from all packages where it's set. Easy: just two commands:
1.

aptitude show "?user-tag(tag_name)" | grep ^Package | awk '{print $2}'

2. make a list of packages and run

sudo aptitude remove-user-tag tag_name [long list of packages here]

Is it really long list? If it really is you should experience a lot of hassle doing copy/paste into the aptitude command. But hopefully there is hope!

USER_TAG=tag_name; for i in $(aptitude show "?user-tag($USER_TAG)" | grep ^Package | awk '{print $2}'); do sudo aptitude remove-user-tag $USER_TAG $i; done

Not so bad, isn't it?