ImageMagick - How to shrink images in batch mode

find . -size +1000k -name \*.png awk '{print $1,$1}' | xargs -n2 convert -scale 1024x\>

Setup a Future Rescue with the 'at' Comand

With a leased server, you never want to lock yourself out of your box. If you do get locked out, KVM style console access can save you but it's not cheap. It usually costs $30 per use. And that's if your ISP even supports it.

There are 3 risky things that can get you locked out of your server.
1) service iptables restart
2) service network restart
3) changing etc files like /etc/sudoers ... and then rebooting

For me, iptables is the riskiest to edit because one bad rule can lock you out of ssh on port 22.

To protect yourself, use 'at' to revert the file change 5 minutes in the future:
echo 'mv ifcfg-eth0.save ifcfg-eth0' at now + 5 minute

*this will work even if you are kicked out (as long as the server is running).


--update-- here is another way to prevent getting locked out by iptables during testing:

*/10 * * * * iptables -P INPUT ACCEPT; iptables -P OUTPUT ACCEPT; iptables -P FORWARD ACCEPT; /sbin/iptables -F

This will flush all the rules every 10 minutes, just in case you lock yourself out. When you're happy with the results of your work, remove the line from your crontab, and you're in business.

Consolidate your web architecture -w- vmware

Good and Cheap Hardware
CPU: AMD64 Dual Core Processor 4400+
Clock: 2.2 GHz
Ram: 1 Gig

1) Select a CentOS 4.4 install (With 5 usable ip addresses)
2) Install vmware server (See last weeks post)
3) Download CentOS 5 virtual appliance and cp into /vm directory
http://www.vmware.com/vmtn/appliances/directory/820
4) Download MySQL Appliance and cp into /vm directory
5) Download memcached Appliance and cp into /vm directory

6) Edit /etc/sysconfig/network-scripts/ifcfg-eth0-range0 to free up 3 ip addresses
## START
#IPADDR_START=72.xxx.xxx.19
IPADDR_START=72.xxx.xxx.22
IPADDR_END=72.xxx.xxx.22
CLONENUM_START=0
##END

7) service network restart
New ifconfig wil show IP's up on eth0 and and one ethernet alias eth0:0

8) run vmware_config.pl
for networking, select vmnet0 to bridge on eth0
set vmware default appliance directory to /vm

9) From a pc start vmware client and connect to the remote server.
10) "Open existing virtual machine" and browse to each appliance to install.
11) Start up both vm appliances and setup networking.
ifconfig eth0 72.xxx.xxx.19
route add default gw [use the same gateway that is used on the host server]
make changes permanent

/etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=static
HWADDR=00:0c:22:32:s0:d4
IPADDR=72.xxx.xxx.19
NETMASK=255.255.55.248
ONBOOT=yes

# vi /etc/sysconfig/network
Append/modify configuration as follows:
NETWORKING=yes
HOSTNAME=www1.fubar.com
GATEWAY=72.xxx.xxx.xxx

Set the correct DNS server defined in /etc/resolv.conf file:
# vi /etc/resolv.conf
nameserver 72.yyy.yyy.yyy

service network restart
ping google.com

12) Assign DNS names to the new servers

13) Install vmware tools. This is not required if you will usually login to the server via ssh.
http://www.thoughtpolice.co.uk/vmware/howto/centos-5-vmware-tools-install.html

Install Vmware on 64 bit CENTOS 4.4

Here are detailed instructions on how to successfully install vmware on a 64 bit server running CENT0S 4.4

The server:
OS: CentOS 5 x86_64
Hardware: AMD 64 Athlon 4400 DC/1024MB/160GB SATA

# uname -a
Linux xx.xx.xx.xx 2.6.9-42.0.10.ELsmp #1 SMP Tue Feb 27 09:40:21 EST 2007 x86_64 x86_64 x86_64 GNU/Linux

# arch -a
x86_64

#cat /etc/redhat-release
CentOS release 5 (Final) *

[Thanks to my hosting provider LayeredTech. They Rock!]

First, download and install the latest VMware Server. Since there are no x64 rpm files I used the x86. I hope it will efficiently using this CPU.

sh-3.00# rpm -ivh VMware-server-1.0.3-44356.i386.rpm

sh-3.00# vmware-config.pl
The correct version of one or more libraries needed to run VMware Server may be
missing. This is the output of ldd /usr/bin/vmware:
linux-gate.so.1 => (0xffffe000)
libm.so.6 => /lib/tls/libm.so.6 (0xf7fd0000)
libdl.so.2 => /lib/libdl.so.2 (0xf7fcc000)
libpthread.so.0 => /lib/tls/libpthread.so.0 (0xf7fba000)
libX11.so.6 => not found
libXtst.so.6 => not found
libXext.so.6 => not found
libXt.so.6 => not found
libICE.so.6 => not found
libSM.so.6 => not found
libXrender.so.1 => not found
libz.so.1 => not found
libc.so.6 => /lib/tls/libc.so.6 (0x001e6000)
/lib/ld-linux.so.2 (0x001cd000)

This program cannot tell for sure, but you may need to upgrade libc5 to glibc
before you can run VMware Server.

Hit enter to continue.

This is the first hint of trouble. Next after a few more prompts you will see an error message about a missing X library file.

/usr/lib/vmware/bin/vmware-vmx: error while loading shared libraries: libX11.so. 6: cannot open shared object file: No such file or directory
Please enter your 20-character serial number.

This is because vmware server is compiled against X libraries on the x86 architecture. To me this is an odd dependency because most servers do not run Xwindows. Personally, I never use Xwindows on servers, but to get past the vmware-config.pl installation you must install the xorg-x11-libs for x86 platform (even if you are on a x64).

Here is how it went:
rpm -ivh xorg-x11-libs-6.8.2-1.EL.13.37.7.i386.rpm
error: Failed dependencies:
libGL.so.1 is needed by xorg-x11-libs-6.8.2-1.EL.13.37.7.i386
libexpat.so.0 is needed by xorg-x11-libs-6.8.2-1.EL.13.37.7.i386
libfontconfig.so.1 is needed by xorg-x11-libs-6.8.2-1.EL.13.37.7.i386
libfreetype.so.6 is needed by xorg-x11-libs-6.8.2-1.EL.13.37.7.i386
libz.so.1 is needed by xorg-x11-libs-6.8.2-1.EL.13.37.7.i386

OK, so first I need to download the libraries that are needed.

wget ftp://ftp.nluug.nl/pub/os/Linux/distr/smeserver/releases/testing/7.2/smeos/i386/CentOS/RPMS/xorg-x11-libs-6.8.2-1.EL.13.37.7.i386.rpm
wget ftp://ftp.nluug.nl/pub/os/Linux/distr/smeserver/releases/7.1/smeos/i386/CentOS/RPMS/zlib-1.2.1.2-1.2.i386.rpm
wget ftp://ftp.nluug.nl/pub/metalab/distributions/smeserver/releases/7.1/smeos/i386/CentOS/RPMS/expat-1.95.7-4.i386.rpm
wget ftp://ftp.nluug.nl/pub/ibiblio/distributions/smeserver/releases/7.1/smeos/i386/CentOS/RPMS/fontconfig-2.2.3-7.centos4.i386.rpm
wget ftp://ftp.nluug.nl/pub/os/Linux/distr/CentOS/4.4/updates/i386/RPMS/freetype-devel-2.1.9-5.el4.i386.rpm
wget ftp://ftp.nluug.nl/pub/sunsite/distributions/smeserver/releases/testing/7.2/smeos/i386/CentOS/RPMS/xorg-x11-Mesa-libGL-6.8.2-1.EL.13.37.7.i386.rpm
wget ftp://ftp.nluug.nl/pub/os/Linux/distr/CentOS/4.4/updates/x86_64/RPMS/freetype-2.1.9-5.el4.i386.rpm


rpm -ivh xorg-x11-libs-6.8.2-1.EL.13.37.7.i386.rpm xorg-x11-Mesa-libGL-6.8.2-1.EL.13.37.7.i386.rpm expat-1.95.7-4.i386.rpm fontconfig-2.2.3-7.centos4.i386.rpm freetype-devel-2.1.9-5.el4.i386.rpm zlib-1.2.1.2-1.2.i386.rpm freetype-2.1.9-5.el4.i386.rpm
Preparing... ########################################### [100%]
1:zlib ########################################### [ 14%]
2:freetype ########################################### [ 29%]
3:expat ########################################### [ 43%]
4:fontconfig ########################################### [ 57%]
5:freetype-devel ########################################### [ 71%]
6:xorg-x11-Mesa-libGL ########################################### [ 86%]
7:xorg-x11-libs ########################################### [100%]

OK, so now I got xorg-x11 library installed I can retry the vmware-config.pl script.

Please enter your 20-character serial number.

Type XXXXX-XXXXX-XXXXX-XXXXX or 'Enter' to cancel:

Starting VMware services:
Virtual machine monitor [ OK ]
Virtual ethernet [ OK ]
Bridged networking on /dev/vmnet0 [ OK ]

The configuration of VMware Server 1.0.3 build-44356 for Linux for this running
kernel completed successfully.

Success!!

Thanks to Karanbir Singh for his blog post on installing vmware on CENTOS 4.4.
vmware_server_on_x86_64_centos4_redhat_e

Easiest possible way to port forward

Say you have a mysql server that you do not want to make public. Port 3306 is blocked by your firewall but port 22 is open for ssh.

Just connect tp your mysql server from putty on port 22 and modify one setting.
Connection->SSH->Tunnels add "L3306 127.0.0.1:3306"

After you make this change you can Click on Session and save your changes.

Once you connect through putty, you will be able to access the remote mysql server by localhost:3306.

Warning: Don't forget that this port is local now! If you forget that port 3306 is forwarded, you may wonder why you have a mysql server running on your machine.
If you decide later to install mysql on your local machine you should delete the port forward from the putty config to avoid unintentional changes your remote server.

Awking Mysql

How do you perform awk type functions in Mysql?

Say you have a list of urls and you want to find the most common domain names hosting gif files.

http://www.fileden.com/123/foo.gif
http://h1.ripway.com/11.gif
http://www.hostmyfile.info/22/web.gif
http://www.fileden.com/24/fun.gif

If this list is in a file you can use standard linux commands to find the most common domains.

grep '\.mp3' my_pictures |awk -F'/' '{print $3}' |sort |uniq -c|sort -nr


If the list is in a mysql database you can use locate() and substring() to accomplish the same thing!

SELECT count(*) as count, substring(url,1,locate('/',url,9)) as f FROM `my_pictures` WHERE url IS NOT NULL and url !='' and filename like '%.mp3' group by f order by count desc

Happy Awking!

phpMyAdmin #1045 - Access denied for user 'root'@'localhost' (using password: NO)

When installing phpMyAdmin the most common error is not being able to connect to the database after installation.

#1045 - Access denied for user 'root'@'localhost' (using password: NO)

Here is the solution:
Make sure your config file in the phpMyAdmin directory is named "config.inc.php"!

In the installation instuctions it says to copy config.default.php into the phpMyAdmin directory and then customize it (usually the only change needed is to change the 'auth_type' is changed from 'config' to 'http'). Make sure to rename config.default.php to config.inc.php.

Get the Latidude and Longitude from an IP

ip2location shell script

for k in `cat ip_list`;
do
id=`echo $k|awk -F_ '{print $1}'`;
ip_num=`echo $k|awk -F_ '{print $2}'|awk -F. '{ printf "%10d\n",$1 * 16777216 + $2 * 65536 + $3 * 256 + $4 }'`;
echo id=$id;
echo ip_num=$ip_num;
echo "SELECT ip_latitude,ip_longitude,ip_city FROM ip2location WHERE (ip_from <= $ip_num) AND (ip_to >= $ip_num) LIMIT 1"|mysql -u doug crostel_store;
done


UPDATE my_space_users u,
ip2location i SET u.user_location = i.ip_city WHERE (

ip_from <=1091213045
) AND (
ip_to >=1091213045
) LIMIT 2

mysql> update my_space_users u, ip2location l set u.ip_country=l.country_short, u.ip_region=l.ip_region, u.ip_city=l.ip_city, u.ip_latitude=l.ip_latitude, u.ip_longitude=l.ip_longitude, u.ip_isp=l.ip_isp, u.ip_domain=l.ip_domain WHERE (l.ip_from <= u.reg_ip_num) AND (l.ip_to >= u.reg_ip_num);

Using Wikipedia as a natural language corpus

1) download the english yahoo abstracts from

http://en.wikipedia.org/wiki/Sort-Merge_Join

wiki.txt
Anarchism is derived from the Greek αναÏÏία ("without archons (ruler, chief, king)"). Anarchism as a political philosophy, is the belief that rulers, governments, and hierarchal social relationships are unnecessary and should be abolished, although there are differing interpretations of what this means.
|
Albedo is the measure of reflectivity of a surface or body. It is the ratio of electromagnetic radiation (EM radiation) reflected to the amount incident upon it.
Abu Dhabi (Arabic: أب٠ظب٠ʼAbÅ« áºaby) is the largest of the seven emirates that comprise the United Arab Emirates and was also the largest of the former Trucial States. Abu Dhabi is also a city of the same name within the Emirate that is the capital of the country, in north central UAE.


wiki.txt1
Anarchism
is
derived
from
the

wiki.txt2
is
derived
from
the
Greek

wiki.txt12
Anarchism is
is derived
derived from
from the
the Greek

wiki.txt.sorted
wiki.txt.sorted.1
wiki.txt.sorted.nonums
wiki.txt.sorted.nonums1
wiki.txt.sorted.nonums1.uniq
wiki.txt.sorted.nonums1.uniq.2words

export LC_ALL=C

cat
wordnet.index
wordnet.index.sorted
wordnet.index.sorted.uniq
wiki.txt.sorted.nonums1.uniq.2words.cols
wiki.txt.sorted.nonums1.uniq.2words.cols.LC_ALL
wordnet.index.sorted.uniq.LC_ALL
wiki-wordnet.join


for k in `cat wordnet.index.sorted.uniq.LC_ALL`; do look $k wiki.txt.sorted.nonums1.uniq.2words.cols.LC_ALL; done >/tar-backups/wiki-wordnet.join



3 word n-grams
grep -v '[^[:print:]]' wiki.txt1 >wiki1.txt
tail +2 wiki1.txt >wiki2.txt
tail +2 wiki2.txt >wiki3.txt

grep "[[:print:]]\+\t[[:print:]]\+\t[[:print:]]" wiki123.txt >wiki123.txt3



find the most used 3 word phrases in a text file
sort rewrite123.txt3 >rewrite123.txt3
grep ".\+\t.\+\t.\+" rewrite123.txt >rewrite123.txt3
sort rewrite123.txt3 >rewrite123.txt3.sorted
uniq -dc rewrite123.txt3.sorted >rewrite123.txt3.sorted.uniqd
sort -nr rewrite123.txt3.sorted.uniqd >rewrite123.txt3.sorted.uniqd.sorted
more rewrite123.txt3.sorted.uniqd.sorted

3 be traveling with
2 you want to
2 want to have
2 student group bookings.
2 part of their
2 on the other
2 notarized letter of
2 made with tap
2 entertainment , activities
2 and want to

Plot a wav file - wav2png

#!/bin/sh
#usage: wav2png.sh file.wav


BASE=${1%.wav}
PNG=$BASE.png
WAV=$BASE.wav
DAT=$BASE.dat

#echo $BASE,$PNG,$WAV,$DAT
sox $WAV $DAT
grep -v '^;' $DAT >$DAT.clean
FREQ=`head -1 $DAT|tr -d ';'`

echo -e "set terminal png;set title '$FREQ';set output '$PNG'; plot '$DAT.clean'" |gnuplot

Split a midi file into separate tracks

Get the program:

wget http://interglacial.com/~sburke/pub/midi_splitter.pl


perl midi_splitter.pl /tmp/vanilla.mid
file vanilla*.mid
midi2abc vanilla.mid
midi2abc vanilla_t01_c00.mid >vanilla_t01.abc

Access internet hotspots in client mode with WRT54G

Linksys WRT54G (below version 5 because linksys later switched to vmware)

Load the sveasoft firmware. (I am using Alchemy-pre5.3 v2.04.4.8sv)

Put it in client mode with loopback off.

Here is a good tutorial:
http://www.engadget.com/2005/05/24/how-to-connect-your-linksys-wrt54g-network-to-the-internet/

ssh into the router and type:
~ # wl scan
~ # wl scanresults|tr -d '\n' |sed 's/noise/\n/g' |tr ']' '\n'|grep ^SSID|awk '{print $(NF-1),$2}' |sed 's/Mode://'|sort -n
-96 "zeus"
-94 "LF-X1U.00014A10A7C8"
-94 "wireless"
-89 "Doug"
-88 "GHOST"
-86 "COR-MDA-LAN"
-86 "linksys"
-86 "linksys"
-83 "default"
-83 "hpsetup"
-83 "linksys"
-78 "hpsetup"
-78 "linksys"
-74 "hirsh
-51 "link"

You will get a sorted list of the most powerfull access points in range.

Split a polyphonic midi file into monophonic tracks

perl midi_splitter.pl /tmp/vanilla.mid

Music secrets revealed

" Using Sox, LAME, FLAC, and madplay - I put together this "dictionary" of shell commands that can be run to convert audio from one format to other, including dialup-modem-streamable lofi mono MP3 files, and including shorter clips of music."

-Derek Silvers is the founder and sole programmer of http://cdbaby.com

Well done!

See the article here:
www.oreillynet.com/pub/wlg/5894


Here are a few other usefull linux audio commands:

mp3info /tmp/07_The_Worst_Hangover_Ever.mp3
48 kbit/s j-stereo (2m 57s)

sox /tmp/orig.07_The_Worst_Hangover_Ever.wav -e stat
Samples read: 15695872
Length (seconds): 177.957732
Scaled by: 2147483647.0
Maximum amplitude: 0.999969
Minimum amplitude: -1.000000
Midline amplitude: -0.000015
Mean norm: 0.237282
Mean amplitude: -0.000037
RMS amplitude: 0.312397
Maximum delta: 1.965851
Minimum delta: 0.000000
Mean delta: 0.163578
RMS delta: 0.230147
Rough frequency: 5170
Volume adjustment: 1.000

Using pdftk to process pdf files

pdftk multimodal.pdf dump_data output report.txt; cat report.txt

pdftk multimodal.pdf dump_data_fields output report.txt; cat report.txt

Yahoo hosting MySQL tricks

To load a csv file into a MySQL database.

1) FTP the csv file to /tmp/afile.csv directory on your yahoo server.

2)From yourdomain.com/phpMySql run this sql:

LOAD DATA LOCAL INFILE '/tmp/afile.csv' REPLACE INTO TABLE `carrier`
FIELDS TERMINATED BY ''
OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '\\'
LINES TERMINATED BY '\n';


Here is another example:

LOAD DATA LOCAL INFILE "/tmp/IP.CSV" INTO TABLE iploc FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\r\n'

Lame is sweet

Lame is a pretty cool linux program you can use to batch process MP3 files:

lame --resample 22.05 --scale 10 -b 48 -a HowBizarre.wav -o HowBizarreLoud.mp3

while i'm at it; here is the whole script

#!/bin/bash
for k in *
do
lame --resample 22.05 --scale 10 -b 48 -a $k -o ${k%.wav}.mp3
done

Garbled man pages in Linux?

I use the putty windows client to connect to my linux boxes and always run into this problem:

>man sort

NAME
sort â sort lines of text files
SYNOPSIS
sort [OPTION]... [FILE]...
DESCRIPTION
Write sorted concatenation of all FILE(s) to standard output.
Ordering options:

Mandatory arguments to long options are mandatory for short options
too.
âb, ââignoreâleadingâblanks ignore leading blanks
âd, ââdictionaryâorder
consider only blanks and alphanumeric characters
âf, ââignoreâcase
fold lower case to upper case characters


Luckily there is an easy fix:

export LANG="en_US"

I don't know why Fedora sets the LANG variable to something that breaks your terminal.

SMS service centers

t-mobile 12063130004
cingular 13123149810

Fix a hosed Linux terminal

Are you seeing this:
[⎼⎺⎺├@▒1▒ ·]#
[⎼⎺⎺├@▒1▒ ·]# ⎽├├≤ ␋␌▒␌⎺┼

Instead of this?
[root@a1a ~]#
[root@a1a ~]# ls

Your screen can get hosed if you cat a binary file or for some reason your screen recieved a ^n character. This trash on your screen is called graphics mode. You can fix it by typing:

echo ^O

-while you are typing, it will look like garbage on the screen. Don't worry.
-to print a control O you can type: echo ctl-v ctl-o

More 2 Cent Tips... | Linux Gazette: "To force it back you need to have a shell or other program to send the
^o to the stricken display. I often switch to another virtual terminal
and do an 'echo -e '\017' >/dev/tty2'"