Tuesday, 10 April 2012

Jboss 7, HTTPS and EC browser support

While messing around with Jboss AS 7.1 at work I've made some discoveries.

1. JDK7 supports EC key algorithm
./jdk1.7.0_03/bin/keytool -v -genkey -alias https -keyalg EC -keystore /opt/jboss/https.keystore -keysize 409 -validity 730 -dname "CN=*.example.com, OU=TW, O=Home, L=OL, ST=WiM, C=PL" -storepass s0m3p15s -keypass S0m3p15s

max size is -keysize 571

2. Firefox (14.0a1) also supports EC .. partially - only keys generated with -keysize 256 and -keysize 384

3. Opera (12 alpha) doesn't support EC keys at all :(.


To use keys generated with keytool
keytool -v -genkey -alias https -keyalg DSA -keystore /opt/httpdsa.keystore -keysize 1024  -validity 730 -dname "CN=my.domain.com, OU=Lap, O=Home, L=City, ST=State, C=UK" -storepass s0m3Pa5s -keypass s0m3Pa5s

Subsytem must be set : native="false"

<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="false">
  <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
  <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true">
     <ssl name="ssl" key-alias="https" password="s0m3Pa5s" certificate-key-file="/opt/httpdsa.keystore" protocol="TLSv1" verify-client="false"/>
  </connector>
</subsystem>



To use keys generated with openssl
Subsytem must be set : native="true"

<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="true">
  <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
  <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true">
      <ssl certificate-key-file="/opt/https-rsa4key.pem" protocol="TLSv1" verify-client="false" certificate-file="/opt/https-rsacert.pem" keystore-type="PKCS12" truststore-type="PKCS12"/>
  </connector>
</subsystem>


Generating RSA keys with password

openssl genrsa -des3 -out https-rsa4key.pem 4096
openssl req -new -key https-rsa4key.pem  -out https.csr
openssl x509 -req -days 720 -in https.csr -signkey https-rsa4key.pem -out https-rsacert.pem

To test if browser can handle keys types/size - type password (set at runinng `openssl genrsa...`) when asked
openssl s_server -www -accept 443 -cert https-rsacert.pem -key https-rsa4key.pem

To use in Jboss AS7 standalone.xml (don't froget password="S0m3Pa5s"):
<ssl password="S0m3Pa5s" certificate-key-file="/opt/https-rsa4key.pem" protocol="TLSv1" verify-client="false" certificate-file="/opt/https-rsacert.pem" keystore-type="PKCS12" truststore-type="PKCS12"/>


I've managed to get Jboss AS7.1 to start with openssl DSA keys

openssl dsaparam -out dsaparam 1024
openssl gendsa -out https-dsa.pem dsaparam
openssl req -new -key https-dsa.pem -out https.csr
openssl x509 -req -days 720 -in https.csr -signkey https-dsa.pem -out https-dsacert.pem

To test if browser can handle keys types/size
openssl s_server -www -accept 443 -cert https-dsacert.pem -key https-dsa.pem

and in standalone.xml
<ssl certificate-key-file="/opt/https-dsa.pem" protocol="TLSv1" verify-client="false" certificate-file="/opt/https-dsacert.pem" keystore-type="PKCS12" truststore-type="PKCS12"/>

Sunday, 8 April 2012

Linux JDK 6 installation - the hard way.

To clarify few things :
- I'm not a big fan of java (or any other bloated framework, especially if major versions are not backwards compatible)
- I hate installers that do things that we could really live without, or without users consent (eyecandy, toolbars, auto updaters etc.)

Last week at work I wanted to get JDk6 for JBoss based project. Since there are no tar.gz/tgz versions for 6u31 (like for JDK7), I dl'ed the i586-bin installer. My office server is openSuse 12.1 x86_64. I tried running the installer in different combinations.
chmod 755 jdk-6u31-linux-i586.bin
./jdk-6u31-linux-i586.bin

sh jdk-6u31-linux-i586.bin

Outcome was mostly the same :
jdk-6u31-linux-i586.bin: line 113: ./install.sfx : Permission denied

Installer is mix of shell script - few hundred lines at beginning of the file, and a self extracting binary.
To determinate where the script ends and sfx starts run
less -N jdk-6u31-linux-i586.bin
or
less jdk-6u31-linux-i586.bin
and press = to see line numbers currently seen on screen.

exit 0
^?ELF^A^A^A^@^@^..............
jdk-6u31-linux-i586.bin lines 144-189/327000 byte 5779/85292206 0%

To get the sfx part run
tail -n +189 jdk-6u31-linux-i586.bin > install.sfx

To run (uncompress) the self extracting binary
chmod 755 install.sfx
./install.sfx

I thought that I got what I needed, but at start JBoss 6 spitted out weired error. So I tried to check if everything is fine with the JDK.

cd jdk1.6.0_31
./bin/java -version
Error occurred during initialization of VM
java/lang/NoClassDefFoundError: java/lang/Object

After some googlin' for a reason to what could be the cause of the problem, I found a forum post suggesting that rt.jar could be missing.
find ./ -name rt.*

returned :

./jre/lib/rt.pack

In bin folder I found unpack200
./bin/unpack200 --help

So I had the tool and means, next I had to find what needed to be unpacked
find ./ -name *.pack

returned :

./lib/tools.pack
./jre/lib/charsets.pack
./jre/lib/jsse.pack
./jre/lib/deploy.pack
./jre/lib/javaws.pack
./jre/lib/plugin.pack
./jre/lib/rt.pack
./jre/lib/ext/localedata.pack

So let's get to it
./bin/unpack200  ./lib/tools.pack  ./lib/tools.jar
.
.
./bin/unpack200  ./jre/lib/ext/localedata.pack  ./jre/lib/ext/localedata.jar

After that everything worked fine.
If any1 want's to take a look at the script part of the installer :
head -n 188 jdk-6u31-linux-i586.bin > install.sh

Monday, 2 April 2012

Manual PostgreSQL instalation on Windows

This post is for those unlucky ppl who for some dumb reason had to install Postgres on Windows, and had no luck with it. Most common problems I came across are :
1. installation finishes but database isn't initialized - error says that libintl-8.dll is missing
2. installation stops at the beginning - VisualC Redist Setup crashes - most common on Win7

Other reasons to install PG by hand is that even when using installer with command line options, you can't get the result you wanted, like database encoding, service user etc.

What will need is :
1. Postgres binaries
2. Ntrights.exe

Let's copy all files from postgres zip to c:\pgsql.
To create service-user - in cmd as admin
net user pgsql S0m3Pa5sW0rd /add

Now to properly configure service-user
ntrights.exe -u pgsql +r SeServiceLogonRight
ntrights.exe -u pgsql -r SeInteractiveLogonRight
wmic.exe USERACCOUNT WHERE "name='pgsql'" SET PasswordExpires=FALSE

Service-user needs full control over c:\pgsql
cacls "c:\pgsql" /T /E /G "pgsql":F

To properly initialize database we need to run cmd as service-user pgsql. Still as admin run
runas /user:pgsql cmd
typ password when asked S0m3Pa5sW0rd

Now in new cmd window
cd c:\pgsql
mkdir data
cd bin
initdb.exe -D ../data -E LATIN2 --locale="Czech, Czech Republic"
exit

Now back in admins cmd
cd c:\pgsql\bin
pg_ctl.exe register -N PG84 -U pgsql -P S0m3Pa5sW0rd -D "c:\pgsql\data" -w

On Windows Vista and newer you need to uncomment last line in c:\pgsql\data\pg_hba.conf, since those versions have ipv6 support turned on by default - to test if your system qualifies
ping ::1
echo %ERRORLEVEL%

if echo returns 0, change last line in pg_hba.conf like so (remove # at the beginning of the line)
host    all     all     ::1/128      trust

To start the server - run services.msc , find and start PG84.

Sunday, 1 April 2012

Quick MySQL 5.6 manual instalation on Linux

Lately I've been building most software I need at work. Manually building from source you can minimize dependencies to what you need, and better familiarize yourself with all needed configuration files, than when installing from package - rpm, deb etc.

I got my src from this link.
At work I've been familiarized with PostgreSQL, so my dependency list will be similar to those standard for compiling Postgres : zlib, readline and ssl. CMake is needed for configuring source.

cmake -DCMAKE_INSTALL_PREFIX=/opt/mysql -DWITH_SSL=yes -DWITH_ZLIB=yes -DDEFAULT_CHARSET=utf8 -DWITH_READLINE=yes
make && make install

Prefix is for keeping everything in one place, no trashing in /etc or /usr.

Server starting script - needs to be copied to proper init folder
cp /opt/mysql/support-files/mysql.server /etc/init.d/

Server configuration - need to peek one cnf file as a base for customizing
cp  /opt/mysql/support-files/my-medium.cnf  /opt/mysql/mysql.cnf

I've set ownership to postgres user & group since I already had those in my system. I was curious if the developers considered using other user than default mysql - fortunately they did.
chown -R -h postgres.postgres /opt/mysql

I've added mysql lib folder to ld cache - just in case, couldn't find anything in /opt/mysql/bin that's linked against it.

echo /opt/mysql/lib > /etc/ld.so.conf.d/my.conf
ldconfig

Time to set proper paths and user in /opt/mysql/my.cnf

[client]
socket = /opt/mysql/mysql.sock

[mysqld]
collation-server = utf8_general_ci
user   = postgres
socket = /opt/mysql/mysql.sock

Same goes for the starting script /etc/init.d/mysql.server
#basedir=
basedir=/opt/mysql

#datadir=
datadir=/opt/mysql/data


# lockdir='/var/lock/subsys'
lockdir='/var/lock'


#mysqld_pid_file_path=
mysqld_pid_file_path=/opt/mysql/mysql.pid

Now we need to create the default database
cd  /opt/mysql 
./scripts/mysql_install_db --datadir=/opt/mysql/data --user=postgres --defaults-file=/opt/mysql/my.cnf

Now all that's left is to start the new server
/etc/init.d/mysql.server start

..and set password for user root
./bin/mysqladmin -u root -S mysql.sock password 'new-password'

To simply connect to server
./bin/mysql -S mysql.sock -u root -p

If you need a user with remote access - for example with MySQL Workbench - connect as root
mysql> use mysql
mysql> create user 'admin' identified by 'admin-pass';
mysql> grant all on *.* to 'admin';
mysql> flush privileges;
mysql> \q

What's weird is this won't work for socket and localhost.
I had to create user 'admin'@'127.0.0.1' and 'admin'@'localhost'.

mysql> select host, user  from mysql.user where user='admin';
+-----------+-------+
| host      | user  |
+-----------+-------+
| %         | admin |
| 127.0.0.1 | admin |
| localhost | admin |
+-----------+-------+


To use socket pointed in my.cnf
./bin/mysql --defaults-file=/opt/mysql/my.cnf -u admin -h localhost -p

Monday, 26 March 2012

One liners for encoders

Multimedia capabilities of portable and non PC like devices is growing every day, but mostly limited to variations of MPEG standards variations. On the other hand on PCs there seems to appear a new format once a week.
Luckily there are a few good and free software for converting media formats.

My favourite A/V container is Matroska (.mkv) - can hold multiple .. everything (audio, video, subtitles tracks) :D.
Mkvtoolnix is a awesome toolkit for playing with A/V files - for example splitting & joining files, even from different formats.
GUI is easy to use, so here's some examples of shell tools usage
mkvextract  tracks       example.mkv  1:video.mkv  2:audio.aac 3:english.ass
mkvextract  attachments  example.mkv  1:cover.jpg  2:arial.ttf



For some reason I can't get my TV to play audio from DTS sources.
I used to use FLAC (lossless audio) encoder as a pass-thru for encoding audio to AAC. Here are batches for that - just put them somewhere %PATH% points :

1. Whatever (ffmpeg can read/decode) to FLAC
"C:\Program Files\mkvtoolnix\ffmpeg.exe" -i %1 -vn -sn -acodec flac %2

2. FLAC to HEv1 AAC
@ECHO OFF
cls
set tmp=%CD%
set temp=%CD%
"C:\Program Files\mkvtoolnix\flac.exe" -d -c %1 | "C:\Program Files\mkvtoolnix\neroAacEnc.exe" -he -q 1 -if - -of %2
@PAUSE

The set tmp=%CD% part is useful if you use ramdisk, without it NeroAacEnc uses default "c:\SomeWhere\Temp",
In both batches first parameter is input file, second points to where to write.

I've found a better way for doing that : direct decode whatever audio format to aac
"C:\Program Files\mkvtoolnix\ffmpeg.exe" -i %1 -vn -sn -acodec pcm_s16le -ar 48000 -f wav - | "C:\Program Files\mkvtoolnix\neroAacEnc.exe" -he -q 1 -if - -ignorelength -of %2

And there's also ffmpeg's built in AAC encoder
ffmpeg -i /media/funny.mp4 -vcodec copy -ab 512k -ac 6 -acodec aac -strict experimental /media/funny.mkv


There's a popular opinion around the Net that Nero AAC Encoder gives better results than free libfaac.
Just google for : nero ftp NeroAACCodec-1.5.1.zip - it has encoder version 1.5.4.0 from February 2010.

Help info (-help) says that HEv1/v2 is selected automatically, but I always got LC profile on my encodes, so I started forcing HEv1 (-he) - didn't notice quality difference, but size reduction is significant.

Here's a wine variant (or you can use Linux binaries and skip the "wine") :
wine flac.exe -d -c /media/example.flac | wine neroAacEnc.exe -he -q 1 -if - -of /media/example.aac


Other fun example : got lots of photos from a trip, take a lot of space, so what's the best way to compress them and save on storage space ?
2 pass slow encode from jpg to 1 FPS x264 - got 200MB movie from 900MB of photos, perfect for small SD phone card.
ffmpeg -f image2 -r 1 -i "%d.jpg" -vcodec libx264 -pass 1 -passlogfile /media/1pass -sameq -preset slow /media/temp.mkv

ffmpeg -f image2 -r 1 -i "%d.jpg" -vcodec libx264 -pass 2 -passlogfile /media/1pass -sameq -preset slow -vb 3400k /media/trip.mkv
MPEG (x264 is MPEG4) compression doesn't stores separate frames individually like MJPEG, but looks for similar blocks in neighboring frames and stores differences, plus other complicated size reduction methods of course.
Images have to be named like 1.jpg, 2.jpg .. 10.jpg, .. 999.jpg etc. While first pass is fairly fast, the second encoding can drop to 1-2 FPS with hi-res pics - but size/quality is awesome - or you can use -preset normal.
For bit rate (-vb) I use average file size from the pics list.


Observation : my Philips TV from 5000 series can only display JPEGs compressed without Progressive methode, also can only play mkv with one audio track.

Izismile has option to download their videos in MP4 or FLV format - if you need sources to experiment.

Sunday, 25 March 2012

Wonderful world of RamDisks

Current market situation :
- HDD disks prices are almost twice that before floods in Asia
- SSD disks still very price, an still short writing life span
- RAM cheaper than ever

I've been trying to reduce wear of my office and home HDDs for years now, mostly by disabling usage of swap and tweaking mount options (Linux).

Example of one of my /etc/fstab
/dev/sda1 /   ext4 defaults,noatime,nodiratime 0 1
tmpfs  /media   tmpfs nodev,nosuid,size=3g  0 0

"noatime" disables updating time stamp of accessing file - "access" means reading file, not only modification time like on Windows. Not sure if nodiratime is needed - I've read few different info on the subject.

When RAM got cheap I've upgraded my trusty 6 years old personal notebook from 1 to 3 GB. System caching of often used files is great on both Linux and Windows, but I haven't found a way to prioritize which files I would prefer to buffer.
And what if you need to work on a file you will change many times in short time frame ? From what I read SSD disk based on SLC can withstand about 100k writes (MLC about 1/10 of that), seems like much but for a coder working on big project - not so much.

Ramdisk is virtual disk that uses part of system memory. System with UPS or note/netbook with healthy battery recommended, since dynamic memory looses data when not powered.

To no surprise setting up ramdisks on Linux is easy and doesn't need installation of any special software.
To types I know of :
- tmpfs - will use swap when needed, partition size limited at mount
- ramfs - none of the above

Mounting from shell
mount -t ramfs  ramfs  /ram
mount -t tmpfs -o size=3g  tmpfs /media
Which gives more or less this result
#mount
ramfs on /ram type ramfs (rw,relatime)
tmpfs on /media type tmpfs (rw,relatime,size=3145728k)

#df
ramfs          ramfs        0     0     0    - /ram
tmpfs          tmpfs     3.0G     0  3.0G   0% /media

I recommend tmpfs since its size can be limited, I've setup a 2.5GB ramdisk on a 3GB system - when it got full X crashed and system got stuck - live and learn :].


As for Windows, I've tried few free solutions - here's what I sticked too.

1. VSuite Free Edition
- simple/effective GUI
- a bit slow at mounting

2. My favorite IMDisk
- fast
- free, source code available

There's a neat GUI in Control Panel, but I prefer using batches for un-/mounting.
This mounts a FAT16 partition to a: , of size = first passed parameter
imdisk -a -t vm -s %1M -p "/fs:fat /y" -m a:
I use fat because I don't need journaling - ramdisk would disappear after reboot any how.

Forces dismounting a: ramdisk
imdisk -D -m a:

I'm heavily using IMDisk on 32b XP, but I encounter a small problem - I can't get ramdisk bigger than 1734 MB.
Besides awesome speed (at least 10x than HDD on DDR2), you can unmount ramdisk even if files are open - or something is holding a handle to them.

Monday, 19 March 2012

Keeping eth0 alive script

For some reason my servers at work are cursed with loosing connection to network - you have to connect with some app that send packets to keep network alive - for example Putty - or you won't be able to connect to server, without login in directly and pingin' some1. After little googlin' few years back, I thought it was related to net card driver. I've compiled few driver versions from vendors site but nothing changed. On second machine that I got later I had the same problem, and so did few of my co-workers with their machines. Apparently this behaviour can also mean that the router is dropping entries in arp tables for my machines. I don't have access to the router, so I've made a small hack/fix:

put this in file /etc/dhclient-eth0.conf
send dhcp-lease-time 1100;

This makes dhcp client renew lease every ~500 sec.

Of course I wasn't happy with this approach :). It depended on dhcp server to correctly work, could assign different ip etc. After changing to my eth0 configuration to static ip adress I had to think of other hack/fix.

Let's call it /usr/local/bin/keepalive.sh

#!/bin/bash

while [ -d /root ]
do
    if test -e /proc/net/nf_conntrack
    then
        cnt=`cat /proc/net/nf_conntrack | grep -v "192\.168" | grep EST | wc -l`
        if [ "${cnt}" -lt "3" ]; then
            ping -c 1 10.1.1.13  >> /dev/null 2>&1
        fi
    fi
    if [ ! -f /var/run/sshd.init.pid ]
    then
        break
    fi
    sleep 2m
done

Some explanation :
/proc/net/nf_conntrack - file containing information about network connection
grep -v "192\.168" - filters lines NOT including string "192.168", because I have virtual machines running on this server, with virtual network 192.168.1.0
grep EST - filter lines including connection status == ESTablished
wc -l - counts how many filtered lines were found

Instead of the whole line cnt=... you can just use :
cnt=`grep EST /proc/net/nf_conntrack | wc -l`

or if your system doesn't have /proc/net/nf_conntrack
cnt=`ss -an | grep -v "192\.168" | grep "ESTAB" | wc -l`
or
cnt=`netstat -an | grep -v "127\.0" | grep -v "192\.168" | grep "ESTABLISHED" | wc -l`
although ss is faster.

But to run it from crontab at boot I had to make a middle-man script to run my main script in background - scripts put in background directly from cron seem to time-out eventual and force cron to nag about it thou sendmail or syslog.

Let's call it /usr/local/bin/runbg.sh - not much to see here
/usr/local/bin/keepalive.sh &

And for the final touch, in /var/spool/cron/tabs/root add line
@reboot /usr/local/bin/runbg.sh >> /tmp/keepalive.log 2>&1

Kernel 3.3 and Xen 4.1.2 on openSuse 12.1

For over a year I used VMware Server 2 to run few virtual machines at work. But after every kernel update I had to look for a patch to build modules needed by VMS2. Eventually I got fed up with this procedure and started looking for a better alternative. Got XEN to run on Mandriva 2010, OS of choice at the time. Xen and kernel come as a team so no need to recompiling any thing, and it's a bit faster then VMS2. Sadly Mandriva 2011 didn't come with Xen full virtualization kernel, so I tried openSuse 12.1, after not so good experience with Fedora 16.
OpenSuse 12.1 comes with kernel 3.1.0 and xen 4.1.2. After updating to kernel 3.1.9, me and other users of virtual machines on my office server, noticed that net connection tends to break, for example when some1 starts his machine.

To allow every user to start VMs manually, I've added this to /etc/sudoers
%users  ALL=(ALL)         NOPASSWD: /usr/local/bin/xmst.sh

.. and /usr/local/bin/xmst.sh looks like this

#!/bin/bash

export LD_LIBRARY_PATH=/xen/lib64:/xen/lib
export LD_RUN_PATH=/xen/lib64:/xen/lib
export PYTHONPATH=/xen/lib64/python2.7/site-packages

/xen/sbin/xm start $1

I've renamed original xen init scripts
cd /etc/init.d/
rename -v xen z-xen xen*

1. Building kernel
# as root
cd /usr/src
curl -O http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.3.tar.bz2
tar xjf linux-3.3.tar.bz2
cd linux-3.3

a) configure
You can start with
make  allyesconfig
and just uncheck (in make menuconfig) what you don't need.

I've used .config from rpm of kernel-xen-3.1.9 as a base of my configuration.

DO NOT USE LZMA/XZ COMPRESSION FOR KERNEL/INITRD - Xen only supports gzip compression.

make menuconfig
make
make modules_install
make install

I've also copied the bare kernel image, just in case ;].
cp /usr/src/linux-3.3/vmlinux /boot/vmlinux-3.3


Building Xen
env PREFIX=/xen  make dist
make install


Now there are a few files to change. In /etc/init.d : xen-watchdog, xencommons, xend, xendomains (this one I don't use)

Here's what to add after INIT INFO block:
export LD_LIBRARY_PATH=/xen/lib64:/xen/lib
export LD_RUN_PATH=/xen/lib64:/xen/lib
export PYTHONPATH=/xen/lib64/python2.7/site-packages
export PATH=/xen/bin/xen/sbin:$PATH

I've prepended PATH with my build bins because I don't want to use those installed from rpm.

And added something like this /boot/grub/menu.lst
title Xen - openSUSE 12.1 - 3.3.0
    root (hd0,0)
    kernel /boot/xen-4.1.2.gz vga=mode-0x31a dom0_mem=4488M noreboot
    module /boot/vmlinuz-3.3.0-xen root=/dev/sda1 showopts console=tty7 xencons=tty8
    module /boot/initrd-3.3.0-xen


I've copied vms config.sxp to /xen/vm and set path in /xen/etc/xen/xend-config.sxp
(xend-domains-path /xen/vm/)


You can see that process running my vm has opened vnc access.
ps aux | grep xen

/xen/lib/xen/bin/qemu-dm -d 1 -domain-name dom1 -videoram 4 -k pl-pl -vnc 0.0.0.0:2,password -parallel none -vcpus 1 -vcpu_avail 0x1L -boot c -localtime -serial pty -acpi -usb -usbdevice tablet -net nic,vlan=1,macaddr=00:16:3e:3c:48:65,model=rtl8139 -net tap,vlan=1,ifname=tap1.0,bridge=virbr0 -M xenfv


I've added paths to my xen libs in /etc/ld.so.conf.d/xen.conf like so
/xen/lib
/xen/lib64

After running ldconfig I could probably comment-out those exports in /etc/init.d/my-xen*, but I did it mostly so I could run xentop without typing freaking long paths like this
env LD_LIBRARY_PATH=/xen/lib64:/xen/lib  PYTHONPATH=/xen/lib64/python2.7/site-packages  /xen/sbin/xentop

Since openSuse 12.1 includes custom confs for ld first, my libs will be used before those installed from rpm - in /usr/lib64.

ldd xenconsoled xenstored xenwatchdogd
xenconsoled:
        linux-vdso.so.1 =>  (0x00007fff3c288000)
        libxenctrl.so.4.0 => /xen/lib64/libxenctrl.so.4.0 (0x00007f6a35c82000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f6a35a7e000)
        libxenstore.so.3.0 => /xen/lib64/libxenstore.so.3.0 (0x00007f6a35874000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00007f6a35671000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f6a35469000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f6a350d9000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6a34ebc000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f6a35ea5000)
xenstored:
        linux-vdso.so.1 =>  (0x00007fff64d85000)
        libxenctrl.so.4.0 => /xen/lib64/libxenctrl.so.4.0 (0x00007fe712f30000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fe712d2c000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fe71299c000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe71277f000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fe713153000)
xenwatchdogd:
        linux-vdso.so.1 =>  (0x00007fff6a149000)
        libxenctrl.so.4.0 => /xen/lib64/libxenctrl.so.4.0 (0x00007f5b01bb5000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f5b019b1000)
        libxenguest.so.4.0 => /xen/lib64/libxenguest.so.4.0 (0x00007f5b0178b000)
        libxenstore.so.3.0 => /xen/lib64/libxenstore.so.3.0 (0x00007f5b01581000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f5b011f1000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5b00fd4000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f5b01dd8000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f5b00dbc000)

After checking that everything works, I've added system wide path to my xen bins
-in /etc/profile.local
export PATH=/xen/bin:/xen/sbin:$PATH

Here's a example how to use xl - the replacement for xm
xl vcpu-pin dom1 0 0
xl vcpu-pin dom1 1 1
xl vcpu-pin dom2 0 2
xl vcpu-pin dom2 1 3
- pins physical cpu (cores) 0 & 1 to dom1s virtual cpu 0 & 1, and physical cpu 2 and 3 to dom2 virtual cpu 0 & 1.

Xen 4.2 is due this or next month, hope it will be as easy to install ;].

UPDATE

Here's .config from my xen kernel 3.3.
Weird thing is - I've tried same .config with kernel 3.3.1 when it came out - can't get the damned thing to work.
Kernel on it's on boot fine, but with xen - ends with blank screen and reboots (removed noreboot from grub) - kernel 3.3 & xen works fine, so I guess I'll take a crack at next kernel version (3.3.2 / 3.4).


UPDATE 2
First thing I did at work on monday morning was compile kernel 3.3.2 ... and Xen works again, woo hoo :D

Monday, 12 March 2012

Fun with TLSv1.1 and TLSv1.2

Over a half a year ago there was a big fuss over the BEAST attack method against SSLv3/TLSv1, since then ... basically nothing changed :D. As far as I know Opera is only browser supporting better/newer TLSv1.1 and TLSv1.2 - IE on Win7 and Win2008 Server supposedly also supports those protocols, will check at work and write when I can.

OpenSSL (CVS changes log) with version 1.0.1 will introduce support for improved TLS, GnuTLS had it for few years now.

Here are few "simple" steps to install and play around with the latest software without interfering in system.
I'm doing everything in /tmp dir since it's running on tmpfs (ramdisk). Zlib is used to present a fun way to use env and LD_LIBRARY_PATH, and maybe for those interested with really old systems.

1. Downloading sources
mkdir /tmp/src /tmp/opt
cd /tmp/src

wget http://zlib.net/zlib-1.2.6.tar.gz
aria2c http://ftp.gnu.org/gnu/gnutls/gnutls-3.0.15.tar.xz
curl -O ftp://ftp.openssl.org/snapshot/openssl-1.0.1-stable-SNAP-20120311.tar.gz

tar xzf zlib-1.2.6.tar.gz
tar xJf gnutls-3.0.15.tar.xz
tar xzf openssl-1.0.1-stable-SNAP-20120311.tar.gz
Hint : tar xjf some.tar.bz2 for bz2 compressed tars.

2. Compiling zlib
cd zlib-1.2.6
./configure --prefix=/tmp/opt/zlib --64
make && make install

3. Compiling openssl
cd ../openssl-1.0.1-stable-SNAP-20120311
./config --prefix=/tmp/opt/ssl --with-zlib-lib=/tmp/opt/zlib/lib --with-zlib-include=/tmp/opt/zlib/include zlib threads shared
env LD_LIBRARY_PATH=/tmp/opt/zlib/lib make && make install

4. Compiling gnutls
cd ../gnutls-3.0.15
./configure --prefix=/tmp/opt/tls --with-libz-prefix=/tmp/opt/zlib --disable-openssl-compatibility
env LD_LIBRARY_PATH=/tmp/opt/zlib/lib make && make install

Sure, you could create my.conf in /etc/ld.so.conf.d/ containing path /tmp/opt/zlib/lib and run ldconfig, but this way may cause a problem.
# ldd /tmp/opt/ssl/lib/libcrypto.so.1.0.0
 linux-vdso.so.1 =>  (0x00007ffff2da8000)
 libdl.so.2 => /lib/libdl.so.2 (0x00007f9a81116000)
 libz.so.1 => /usr/lib/libz.so.1 (0x00007f9a80f00000)
 libc.so.6 => /lib/libc.so.6 (0x00007f9a80b5e000)
 /lib/ld-linux-x86-64.so.2 (0x00007f9a81711000)

# env LD_LIBRARY_PATH=/tmp/opt/zlib/lib  ldd /tmp/opt/ssl/lib/libcrypto.so.1.0.0
 linux-vdso.so.1 =>  (0x00007fffe35ff000)
 libdl.so.2 => /lib/libdl.so.2 (0x00007f062222c000)
 libz.so.1 => /tmp/opt/zlib/lib/libz.so.1 (0x00007f0622016000)
 libc.so.6 => /lib/libc.so.6 (0x00007f0621c74000)
 /lib/ld-linux-x86-64.so.2 (0x00007f0622827000)

Depending on content /etc/ld.so.conf custom confs from /etc/ld.so.conf.d/ may be used prior or after paths with standard system libraries. This means either every binary linked against libz.so.1 will use system library or that build in /tmp/opt/zlib/lib (apps I just compiled would use /usr/lib/libz.so.1, or everything would use my libz) - and I didn't want that.

5. We will need keys/certs
export  LD_LIBRARY_PATH=/tmp/opt/zlib/lib
cd  /tmp/opt
ssl/bin/openssl req -new -nodes -out req.pem -keyout key.pem
ssl/bin/openssl  rsa -in key.pem -out new.key.pem
ssl/bin/openssl  x509 -in req.pem -out server.pem -req -signkey new.key.pem -days 3650

Remember that export variable=value expires after login out or exiting mc if you're using Ctrl+o.

6. Run gnutls as a simple http server
env LD_LIBRARY_PATH=/tmp/opt/zlib/lib  tls/bin/gnutls-serv -p 443 --http -g --x509certfile=server.pem --x509keyfile=key.pem
Ctrl+C to exit.

In Opera : Tools > Preferences > Advanced > Security > Security Protocols ... - leave only [Enable TLS 1.2] checked (1.1). You have to close and open Opera anew for the changes to take effect. Now open https://localhost/ - you have to [Approve] the Security Issue.

7. Run openssl as a simple http server
You can force TLS version by using -tls1_2, -tls1_1, -tls1 or disable like this -no_tls1_2 - you can have all {Security Protocols} in Opera enabled.
cd /tmp/opt
env LD_LIBRARY_PATH=/tmp/opt/zlib/lib  ssl/bin/openssl s_server -accept 443 -key new.key.pem  -tls1_2 -www
Ctrl+C to exit.

Well that was fun, right ;]. More fun with LD_LIBRARY_PATH soon.

Sunday, 4 March 2012

SSL pubkeys to keystore update script

From what I've been told for Jboss to use other services (SMS, Smtp) over SSL, their public keys have to be added to keystore.
So i whipped out this piece of code.

#!/bin/bash

CRDIR=/opt/https
KEYTL=/opt/java/bin/keytool
KYSTR=/opt/test.keystore
LOG=/opt/cron_cert.log
LST=(smtp.gmail.com:465  ssl.hqsms.com:443)

date >> ${LOG}
echo "--== List count : ${#LST[*]}" >> ${LOG}

if [ ! -d ${CRDIR} ];then
    echo "!!! ERROR: Cert dir missing: ${CRDIR}" | tee -a ${LOG}
    exit 1
fi

cd ${CRDIR}
rm -f ${CRDIR}/*.tmp

for item in ${LST[*]}
do
    if [ ! -f ${item} ];then echo "dummy" > ${item}
    fi
done

CERTS=(*)

if [ "${CERTS[0]}" = "*" ];then
    echo "??? WARNING : No certs in ${CRDIR}" | tee -a ${LOG}
    if [ ${#LST[*]} -eq 0 ];then
        echo "!!! ERROR : Certs list is empty" | tee -a ${LOG}
        exit 1
    fi
else
    echo "--== Count of certs in ${CRDIR} : ${#CERTS[*]}" >> ${LOG}
fi

if [ ! -f ${KYSTR} ];then
    ${KEYTL} -genkey -noprompt -dname "CN=s, OU=s, O=s, L=s, ST=s, C=s" -alias fooxyz -keystore ${KYSTR} -storepass Somepass -keypass Somepass
    ${KEYTL} -delete -alias fooxyz -keystore ${KYSTR} -storepass Somepass
fi

#------------------------------------------
for CRT in ${CERTS[*]}
do
    echo "--== CERT :: ${CRT}" >> ${LOG}
    echo "Q" | openssl s_client -connect ${CRT} 2>>${LOG} | grep -B 100 "END CERTIF" | grep -A 100 "BEGIN CERTIF" > ${CRT}.tmp
    # gnutls-cli  --print-cert  -p 443  some.ip

    if [ -s ${CRT}.tmp ];then
        diff -qs ${CRT} ${CRT}.tmp >> ${LOG}
        rtn=`echo $?`
        if [ ! ${rtn} == 0 ]; then
            echo "--== Updating ${CRT}" >> ${LOG}
            mv -f ${CRT}.tmp ${CRT}
            ${KEYTL} -delete -alias ${CRT} -keystore ${KYSTR} -keypass Somepass -storepass Somepass >> ${LOG} 2>&1
            ${KEYTL} -importcert -noprompt -alias ${CRT} -file ${CRDIR}/${CRT} -keystore ${KYSTR} -storepass Somepass >> ${LOG} 2>&1
    fi
    else
        echo "!!! ERROR : Couldn't get cert from ${CRT}" >> ${LOG}
    fi
    rm -f ${CRDIR}/*.tmp
done

Still got one problem with this script - when adding a new cert to keystore, keytool will report false error about removing old key before update.

HTTPS redirecting with socat

Long story short : got jboss app (port 8443) that goes off-line for few minutes at night - data migration. For that time I wanted to redirect users to apache (port 443) page with proper info. I've tried two netcat processes with redirecting stdin & stdout, but it was awfully slow. I've found socat as a feature richer netcat alternative.

#!/bin/bash

/etc/init.d/jboss stop

dflt=`cat /proc/sys/net/ipv4/tcp_fin_timeout`
dflr=`cat /proc/sys/net/ipv4/tcp_tw_reuse`

echo ${dflt}
echo ${dflr}

echo 5 > /proc/sys/net/ipv4/tcp_fin_timeout
echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse

for i in {1..9}
do
    sclog=/tmp/socat${i}.log
    #sleep 1
    (nohup /opt/socat/bin/socat TCP-LISTEN:8443,fork tcp4-connect:127.0.0.1:443 > ${sclog} 2>&1)&
    sleep 5
    cnta=`grep -Ei "adres|alrea|use" ${sclog} | wc -l`
    if [ ! -s ${sclog} ] && [ ${cnta} == 0 ];then
        break
    else
        killall /opt/socat/bin/socat
    fi
    rm -f ${sclog}
done

rm -f ${sclog}

echo ${dflt} > /proc/sys/net/ipv4/tcp_fin_timeout
echo ${dflr} > /proc/sys/net/ipv4/tcp_tw_reuse

If you got commercial ssl cert only in keystore format - here's how to convert it to apache comptible format
/opt/java/bin/keytool -importkeystore -srcstoretype JKS -srcstorepass SomePass -srckeystore https.keystore -deststoretype PKCS12 -deststorepass SomePass -destkeystore https.pk12.der
openssl pkcs12 -in https.pk12.der -nodes -out apache.pem

Send mail with attachment - python script

I've needed a shell script to daily send a log of some cron job. So after some googlin' and mixing some examples I got this :

#!/usr/bin/env python
# -*- coding: utf_8 -*-

import sys
from os import path
from smtplib import SMTP
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.MIMEBase import MIMEBase
from email import Encoders

fromad = "UserName@gmail.com"
toaddr = "AdminLogin@gmail.com"

msg = MIMEMultipart('mixed')
msg['Subject'] = sys.argv[1]
msg['From'] = fromad
msg['To'] = toaddr

text = "Some text that says details are in attached log or something"
body = MIMEText(text, 'plain')
msg.attach(body)

part = MIMEBase('application', "octet-stream")
fp = open(sys.argv[2], 'rb')
part.set_payload( fp.read() )
fp.close()
Encoders.encode_base64(part)
part.add_header('Content-Disposition', 'attachment; filename="%s"' % path.basename(sys.argv[2]))
msg.attach(part)

sndm = SMTP()
sndm.set_debuglevel(0)
sndm.connect('smtp.gmail.com', '587')
sndm.starttls()
#sndm.ehlo()
try :
    #sndm.esmtp_features['auth'] = 'LOGIN DIGEST-MD5 PLAIN'
    sndm.esmtp_features['auth'] = 'LOGIN'
    sndm.login('UserName', 'PassWord')
    sndm.sendmail(fromad, toaddr, msg.as_string())
    sndm.quit()
except Exception, e:
    print e

Usage : ./script.sh "Some email subject" /tmp/cron.log

Works great on openSuse 12.1, on Arch Linux 2011 - not so much - have to change 1st line to :
#!/usr/bin/python2

To use SSL instead of StartTLS switch to SMTP_SSL and change to proper port - for gmail SSL it's 465

from smtplib import SMTP_SSL
#from smtplib import SMTP
.
.
sndm = SMTP_SSL()
#sndm = SMTP()
.
.
sndm.connect('smtp.gmail.com', '465')
#sndm.connect('smtp.gmail.com', '587')
.
.
#sndm.starttls()

On Arch Linux 2011 line
#!/usr/bin/env python
runs script under Python 3.2, not 2.7.

Here's a working version for Python 3.2 on Arch Linux.
#!/usr/bin/env python
# -*- coding: utf-8 -*-

import sys
from os import path
from smtplib import SMTP_SSL
#from smtplib import SMTP
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.mime.base import MIMEBase
from email import encoders

fromad = "UserName@gmail.com"
toaddr = "AdminLogin@gmail.com"

msg = MIMEMultipart('mixed')
msg['Subject'] = sys.argv[1]
msg['From'] = fromad
msg['To'] = toaddr

text = "Some text that says details are in attached log or something"
body = MIMEText(text, 'plain', 'utf-8')
msg.attach(body)

part = MIMEBase('application', "octet-stream")
fp = open(sys.argv[2], 'rb')
part.set_payload( fp.read() )
fp.close()
encoders.encode_base64(part)
part.add_header('Content-Disposition', 'attachment; filename="%s"' % path.basename(sys.argv[2]))
msg.attach(part)

sndm = SMTP_SSL()
#sndm = SMTP()
sndm.set_debuglevel(1)
sndm.connect('smtp.gmail.com', '465')
#sndm.starttls()
#sndm.ehlo()
try :
    #sndm.esmtp_features['auth'] = 'LOGIN DIGEST-MD5 PLAIN'
    sndm.esmtp_features['auth'] = 'LOGIN'
    sndm.login('UserName', 'PassWord')
    sndm.sendmail(fromad, toaddr, msg.as_string())
    sndm.quit()
except Exception as e:
    print(e)

Biggest problem I found with Python 3.2 version is you can't use non-ascii letters for email subject - if you do script will stop with something like this
'ascii' codec can't decode byte 0xc4 in position 0: ordinal not in range(128)

To check which auth methods servers support you can either use
telnet plus.smtp.mail.yahoo.com 25
ehlo me
quit
or for SSL
openssl s_client -connect smtp.zoho.com:465
ehlo me
Q  -to quit connection