Friday, 21 August 2015

Sending e-mail via python script - version 2, simplified


Usage :  ./script.sh   "E-Mail Subject"   /var/log/dmesg.log


This version puts contents of /var/log/dmesg.log in to an email message directly


script.sh :
#!/usr/bin/env python

import sys
from os import path
from smtplib import SMTP
from email.mime.text import MIMEText
from email.MIMEBase import MIMEBase
from email import Encoders

me = 'author@server.com'
you = 'recipient@example.com'
cc = 'other@carbon.copy'
bcc = 'nsa@gov.us'

fp = open(sys.argv[2], 'rb')
msg = MIMEText(fp.read(), 'plain', _charset="UTF-8")
fp.close()

msg['Subject'] = sys.argv[1]
msg['From'] = me
msg['To'] = you
msg['Cc'] = cc
msg['Bcc'] = bcc

sndm = SMTP()
sndm.set_debuglevel(1)
sndm.connect('smtp.server.com', 25)
sndm.starttls()
try :
    sndm.esmtp_features['auth'] = 'DIGEST-MD5 LOGIN'
    sndm.login('author@server.com', 'Pas5_W0rd')
    sndm.sendmail(me, [you,cc,bcc], msg.as_string())
    sndm.quit()
except Exception, e:
    print e

Thursday, 20 August 2015

PostgreSQL - Fixing Sequences

Tested on version 8.4

-- DROP FUNCTION public.SeqFix();
CREATE OR REPLACE  Function public.SeqFix() Returns void AS $$
DECLARE LIST record; MaxIDValue INTEGER; CurrentValue iNTEGER;
BEGIN
    FOR LIST iN Select table_schema, table_name, column_name, split_part(column_default,'''' ,2) AS seqname FROM information_schema.columns
                        Where table_catalog=current_database() AND column_default iS NOT NULL AND Position('nextval' iN column_default) =1 order by 1,2,3
    LOOP
        EXECUTE 'SELECT MAX(' || LIST.column_name || ') FROM ' || LIST.table_schema || '.' || LIST.table_name
        iNTO MaxIDValue;
        EXECUTE 'SELECT COUNT(*) FROM information_schema.sequences WHERE sequence_catalog=current_database() AND sequence_schema='''||LIST.table_schema||'''
             AND sequence_name='''||split_part(LIST.seqname, '.',2)||''''
        INTO CurrentValue;
        IF CurrentValue = 0 THEN
            RAISE WARNING E'?? SEQ ::\t%\t :: does not exists  ??', LIST.seqname ;
        ELSE
            EXECUTE 'SELECT last_value FROM ' || LIST.seqname
            INTO CurrentValue;
            IF CurrentValue < MaxIDValue THEN
                RAISE WARNING E'!! SEQ :: \t% = %\t<\tMAX(%.%.% = %) ', LIST.seqname, CurrentValue, LIST.table_schema, LIST.table_name, LIST.column_name, MaxIDValue;
               -- PERFORM pg_catalog.setval(LIST.seqname, MaxIDValue+1, false);
            END IF;
        END IF;
    END loop;
END;
$$ LANGUAGE plpgsql;

SELECT public.SeqFix();

Tuesday, 10 April 2012

Jboss 7, HTTPS and EC browser support

While messing around with Jboss AS 7.1 at work I've made some discoveries.

1. JDK7 supports EC key algorithm
./jdk1.7.0_03/bin/keytool -v -genkey -alias https -keyalg EC -keystore /opt/jboss/https.keystore -keysize 409 -validity 730 -dname "CN=*.example.com, OU=TW, O=Home, L=OL, ST=WiM, C=PL" -storepass s0m3p15s -keypass S0m3p15s

max size is -keysize 571

2. Firefox (14.0a1) also supports EC .. partially - only keys generated with -keysize 256 and -keysize 384

3. Opera (12 alpha) doesn't support EC keys at all :(.


To use keys generated with keytool
keytool -v -genkey -alias https -keyalg DSA -keystore /opt/httpdsa.keystore -keysize 1024  -validity 730 -dname "CN=my.domain.com, OU=Lap, O=Home, L=City, ST=State, C=UK" -storepass s0m3Pa5s -keypass s0m3Pa5s

Subsytem must be set : native="false"

<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="false">
  <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
  <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true">
     <ssl name="ssl" key-alias="https" password="s0m3Pa5s" certificate-key-file="/opt/httpdsa.keystore" protocol="TLSv1" verify-client="false"/>
  </connector>
</subsystem>



To use keys generated with openssl
Subsytem must be set : native="true"

<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="true">
  <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
  <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true">
      <ssl certificate-key-file="/opt/https-rsa4key.pem" protocol="TLSv1" verify-client="false" certificate-file="/opt/https-rsacert.pem" keystore-type="PKCS12" truststore-type="PKCS12"/>
  </connector>
</subsystem>


Generating RSA keys with password

openssl genrsa -des3 -out https-rsa4key.pem 4096
openssl req -new -key https-rsa4key.pem  -out https.csr
openssl x509 -req -days 720 -in https.csr -signkey https-rsa4key.pem -out https-rsacert.pem

To test if browser can handle keys types/size - type password (set at runinng `openssl genrsa...`) when asked
openssl s_server -www -accept 443 -cert https-rsacert.pem -key https-rsa4key.pem

To use in Jboss AS7 standalone.xml (don't froget password="S0m3Pa5s"):
<ssl password="S0m3Pa5s" certificate-key-file="/opt/https-rsa4key.pem" protocol="TLSv1" verify-client="false" certificate-file="/opt/https-rsacert.pem" keystore-type="PKCS12" truststore-type="PKCS12"/>


I've managed to get Jboss AS7.1 to start with openssl DSA keys

openssl dsaparam -out dsaparam 1024
openssl gendsa -out https-dsa.pem dsaparam
openssl req -new -key https-dsa.pem -out https.csr
openssl x509 -req -days 720 -in https.csr -signkey https-dsa.pem -out https-dsacert.pem

To test if browser can handle keys types/size
openssl s_server -www -accept 443 -cert https-dsacert.pem -key https-dsa.pem

and in standalone.xml
<ssl certificate-key-file="/opt/https-dsa.pem" protocol="TLSv1" verify-client="false" certificate-file="/opt/https-dsacert.pem" keystore-type="PKCS12" truststore-type="PKCS12"/>

Sunday, 8 April 2012

Linux JDK 6 installation - the hard way.

To clarify few things :
- I'm not a big fan of java (or any other bloated framework, especially if major versions are not backwards compatible)
- I hate installers that do things that we could really live without, or without users consent (eyecandy, toolbars, auto updaters etc.)

Last week at work I wanted to get JDk6 for JBoss based project. Since there are no tar.gz/tgz versions for 6u31 (like for JDK7), I dl'ed the i586-bin installer. My office server is openSuse 12.1 x86_64. I tried running the installer in different combinations.
chmod 755 jdk-6u31-linux-i586.bin
./jdk-6u31-linux-i586.bin

sh jdk-6u31-linux-i586.bin

Outcome was mostly the same :
jdk-6u31-linux-i586.bin: line 113: ./install.sfx : Permission denied

Installer is mix of shell script - few hundred lines at beginning of the file, and a self extracting binary.
To determinate where the script ends and sfx starts run
less -N jdk-6u31-linux-i586.bin
or
less jdk-6u31-linux-i586.bin
and press = to see line numbers currently seen on screen.

exit 0
^?ELF^A^A^A^@^@^..............
jdk-6u31-linux-i586.bin lines 144-189/327000 byte 5779/85292206 0%

To get the sfx part run
tail -n +189 jdk-6u31-linux-i586.bin > install.sfx

To run (uncompress) the self extracting binary
chmod 755 install.sfx
./install.sfx

I thought that I got what I needed, but at start JBoss 6 spitted out weired error. So I tried to check if everything is fine with the JDK.

cd jdk1.6.0_31
./bin/java -version
Error occurred during initialization of VM
java/lang/NoClassDefFoundError: java/lang/Object

After some googlin' for a reason to what could be the cause of the problem, I found a forum post suggesting that rt.jar could be missing.
find ./ -name rt.*

returned :

./jre/lib/rt.pack

In bin folder I found unpack200
./bin/unpack200 --help

So I had the tool and means, next I had to find what needed to be unpacked
find ./ -name *.pack

returned :

./lib/tools.pack
./jre/lib/charsets.pack
./jre/lib/jsse.pack
./jre/lib/deploy.pack
./jre/lib/javaws.pack
./jre/lib/plugin.pack
./jre/lib/rt.pack
./jre/lib/ext/localedata.pack

So let's get to it
./bin/unpack200  ./lib/tools.pack  ./lib/tools.jar
.
.
./bin/unpack200  ./jre/lib/ext/localedata.pack  ./jre/lib/ext/localedata.jar

After that everything worked fine.
If any1 want's to take a look at the script part of the installer :
head -n 188 jdk-6u31-linux-i586.bin > install.sh

Monday, 2 April 2012

Manual PostgreSQL instalation on Windows

This post is for those unlucky ppl who for some dumb reason had to install Postgres on Windows, and had no luck with it. Most common problems I came across are :
1. installation finishes but database isn't initialized - error says that libintl-8.dll is missing
2. installation stops at the beginning - VisualC Redist Setup crashes - most common on Win7

Other reasons to install PG by hand is that even when using installer with command line options, you can't get the result you wanted, like database encoding, service user etc.

What will need is :
1. Postgres binaries
2. Ntrights.exe

Let's copy all files from postgres zip to c:\pgsql.
To create service-user - in cmd as admin
net user pgsql S0m3Pa5sW0rd /add

Now to properly configure service-user
ntrights.exe -u pgsql +r SeServiceLogonRight
ntrights.exe -u pgsql -r SeInteractiveLogonRight
wmic.exe USERACCOUNT WHERE "name='pgsql'" SET PasswordExpires=FALSE

Service-user needs full control over c:\pgsql
cacls "c:\pgsql" /T /E /G "pgsql":F

To properly initialize database we need to run cmd as service-user pgsql. Still as admin run
runas /user:pgsql cmd
typ password when asked S0m3Pa5sW0rd

Now in new cmd window
cd c:\pgsql
mkdir data
cd bin
initdb.exe -D ../data -E LATIN2 --locale="Czech, Czech Republic"
exit

Now back in admins cmd
cd c:\pgsql\bin
pg_ctl.exe register -N PG84 -U pgsql -P S0m3Pa5sW0rd -D "c:\pgsql\data" -w

On Windows Vista and newer you need to uncomment last line in c:\pgsql\data\pg_hba.conf, since those versions have ipv6 support turned on by default - to test if your system qualifies
ping ::1
echo %ERRORLEVEL%

if echo returns 0, change last line in pg_hba.conf like so (remove # at the beginning of the line)
host    all     all     ::1/128      trust

To start the server - run services.msc , find and start PG84.

Sunday, 1 April 2012

Quick MySQL 5.6 manual instalation on Linux

Lately I've been building most software I need at work. Manually building from source you can minimize dependencies to what you need, and better familiarize yourself with all needed configuration files, than when installing from package - rpm, deb etc.

I got my src from this link.
At work I've been familiarized with PostgreSQL, so my dependency list will be similar to those standard for compiling Postgres : zlib, readline and ssl. CMake is needed for configuring source.

cmake -DCMAKE_INSTALL_PREFIX=/opt/mysql -DWITH_SSL=yes -DWITH_ZLIB=yes -DDEFAULT_CHARSET=utf8 -DWITH_READLINE=yes
make && make install

Prefix is for keeping everything in one place, no trashing in /etc or /usr.

Server starting script - needs to be copied to proper init folder
cp /opt/mysql/support-files/mysql.server /etc/init.d/

Server configuration - need to peek one cnf file as a base for customizing
cp  /opt/mysql/support-files/my-medium.cnf  /opt/mysql/mysql.cnf

I've set ownership to postgres user & group since I already had those in my system. I was curious if the developers considered using other user than default mysql - fortunately they did.
chown -R -h postgres.postgres /opt/mysql

I've added mysql lib folder to ld cache - just in case, couldn't find anything in /opt/mysql/bin that's linked against it.

echo /opt/mysql/lib > /etc/ld.so.conf.d/my.conf
ldconfig

Time to set proper paths and user in /opt/mysql/my.cnf

[client]
socket = /opt/mysql/mysql.sock

[mysqld]
collation-server = utf8_general_ci
user   = postgres
socket = /opt/mysql/mysql.sock

Same goes for the starting script /etc/init.d/mysql.server
#basedir=
basedir=/opt/mysql

#datadir=
datadir=/opt/mysql/data


# lockdir='/var/lock/subsys'
lockdir='/var/lock'


#mysqld_pid_file_path=
mysqld_pid_file_path=/opt/mysql/mysql.pid

Now we need to create the default database
cd  /opt/mysql 
./scripts/mysql_install_db --datadir=/opt/mysql/data --user=postgres --defaults-file=/opt/mysql/my.cnf

Now all that's left is to start the new server
/etc/init.d/mysql.server start

..and set password for user root
./bin/mysqladmin -u root -S mysql.sock password 'new-password'

To simply connect to server
./bin/mysql -S mysql.sock -u root -p

If you need a user with remote access - for example with MySQL Workbench - connect as root
mysql> use mysql
mysql> create user 'admin' identified by 'admin-pass';
mysql> grant all on *.* to 'admin';
mysql> flush privileges;
mysql> \q

What's weird is this won't work for socket and localhost.
I had to create user 'admin'@'127.0.0.1' and 'admin'@'localhost'.

mysql> select host, user  from mysql.user where user='admin';
+-----------+-------+
| host      | user  |
+-----------+-------+
| %         | admin |
| 127.0.0.1 | admin |
| localhost | admin |
+-----------+-------+


To use socket pointed in my.cnf
./bin/mysql --defaults-file=/opt/mysql/my.cnf -u admin -h localhost -p

Monday, 26 March 2012

One liners for encoders

Multimedia capabilities of portable and non PC like devices is growing every day, but mostly limited to variations of MPEG standards variations. On the other hand on PCs there seems to appear a new format once a week.
Luckily there are a few good and free software for converting media formats.

My favourite A/V container is Matroska (.mkv) - can hold multiple .. everything (audio, video, subtitles tracks) :D.
Mkvtoolnix is a awesome toolkit for playing with A/V files - for example splitting & joining files, even from different formats.
GUI is easy to use, so here's some examples of shell tools usage
mkvextract  tracks       example.mkv  1:video.mkv  2:audio.aac 3:english.ass
mkvextract  attachments  example.mkv  1:cover.jpg  2:arial.ttf



For some reason I can't get my TV to play audio from DTS sources.
I used to use FLAC (lossless audio) encoder as a pass-thru for encoding audio to AAC. Here are batches for that - just put them somewhere %PATH% points :

1. Whatever (ffmpeg can read/decode) to FLAC
"C:\Program Files\mkvtoolnix\ffmpeg.exe" -i %1 -vn -sn -acodec flac %2

2. FLAC to HEv1 AAC
@ECHO OFF
cls
set tmp=%CD%
set temp=%CD%
"C:\Program Files\mkvtoolnix\flac.exe" -d -c %1 | "C:\Program Files\mkvtoolnix\neroAacEnc.exe" -he -q 1 -if - -of %2
@PAUSE

The set tmp=%CD% part is useful if you use ramdisk, without it NeroAacEnc uses default "c:\SomeWhere\Temp",
In both batches first parameter is input file, second points to where to write.

I've found a better way for doing that : direct decode whatever audio format to aac
"C:\Program Files\mkvtoolnix\ffmpeg.exe" -i %1 -vn -sn -acodec pcm_s16le -ar 48000 -f wav - | "C:\Program Files\mkvtoolnix\neroAacEnc.exe" -he -q 1 -if - -ignorelength -of %2

And there's also ffmpeg's built in AAC encoder
ffmpeg -i /media/funny.mp4 -vcodec copy -ab 512k -ac 6 -acodec aac -strict experimental /media/funny.mkv


There's a popular opinion around the Net that Nero AAC Encoder gives better results than free libfaac.
Just google for : nero ftp NeroAACCodec-1.5.1.zip - it has encoder version 1.5.4.0 from February 2010.

Help info (-help) says that HEv1/v2 is selected automatically, but I always got LC profile on my encodes, so I started forcing HEv1 (-he) - didn't notice quality difference, but size reduction is significant.

Here's a wine variant (or you can use Linux binaries and skip the "wine") :
wine flac.exe -d -c /media/example.flac | wine neroAacEnc.exe -he -q 1 -if - -of /media/example.aac


Other fun example : got lots of photos from a trip, take a lot of space, so what's the best way to compress them and save on storage space ?
2 pass slow encode from jpg to 1 FPS x264 - got 200MB movie from 900MB of photos, perfect for small SD phone card.
ffmpeg -f image2 -r 1 -i "%d.jpg" -vcodec libx264 -pass 1 -passlogfile /media/1pass -sameq -preset slow /media/temp.mkv

ffmpeg -f image2 -r 1 -i "%d.jpg" -vcodec libx264 -pass 2 -passlogfile /media/1pass -sameq -preset slow -vb 3400k /media/trip.mkv
MPEG (x264 is MPEG4) compression doesn't stores separate frames individually like MJPEG, but looks for similar blocks in neighboring frames and stores differences, plus other complicated size reduction methods of course.
Images have to be named like 1.jpg, 2.jpg .. 10.jpg, .. 999.jpg etc. While first pass is fairly fast, the second encoding can drop to 1-2 FPS with hi-res pics - but size/quality is awesome - or you can use -preset normal.
For bit rate (-vb) I use average file size from the pics list.


Observation : my Philips TV from 5000 series can only display JPEGs compressed without Progressive methode, also can only play mkv with one audio track.

Izismile has option to download their videos in MP4 or FLV format - if you need sources to experiment.