Some Ideas

2011-11-23T12:05:00.001-08:00

ngrep2curl - How to replay a browser web request

Many times, I found myself downloading a file twice because Cookies or some other required browser headers prevent curl from doing a direct download from the command line.

To simplify this process, I wrote a script that:
1) Listens on port 80 to grab the http headers for a download on a browser
2) Generates a curl command using the captured headers.

Old workflow (file transfered twice):
Web Browser -> browse to file -> download to local machine
sftp to remote server

New workflow (file transfered just once):
run ngrep2curl on local machine
Web Browser -> browse to file -> download just the beginning of the file -> cancel download
run the generated curl command on remote server

#!/bin/sh
ngrep port 80 -Wbyline |grep -vE '^#|^T '|head -20|sed 's/\.$//'|awk '/GET/{url=$2} /Host:/{host=$2} /User-Agent:/{gsub("User-Agent: ","");ua=$0} /Cookie:/{gsub("Cookie: ","");cookie=$0} /Referer:/{gsub("Referer: ","");ref=$0} /Accept:/{accept=$0} END{print "curl -L","\x27"host url"\x27","-A","\""ua"\"","-b\""cookie"\"","-e \x27"ref"\x27","-H","\x27"accept"\x27"}'

High Performance Text Processing -Taco Bell style

2011-09-26T21:45:00.000-07:00

Following the Taco Bell programming methodology, we will process a huge amount of data using only a few simple ingredients (i.e. unix command line tools).

Most people won't think twice about writing thousands of lines of code to accomplish what a line or two of bash script will handle.

Some anti-pattterns to avoid come to mind:
NIH (Not Invented Here)
Golden Hammer (Treat every new problem like it's a nail.)
Re-inventing the wheel

Text processing is composed of four primary activities. Filtering, Transforming, Sorting and Joining.

To achieve the fastest processing speed, you should try grouping all of your filtering, transforming and joining tasks together in one pipeline.

Stream processing tasks (filtering, transforming, joining) are limited by disk io only so take advantage of your disk scan and apply all operations as co-routines at the time you read the file.

Lets say I need to apply 5 regular expressions to a file:

Example (as co-routines- equally fast ways):

time cat bigfile \
|grep -vE "[^a-z0-9 ][^a-z0-9 ]|[^a-z0-9] [^a-z0-9]||\. |[a-z]' " \
> bigfile.clean

Another example (same thing- but 5 times slower):
time cat bigfile|grep -v '[^a-z0-9 ][^a-z0-9 ]'>tmpfile1
time cat tmpfile1|grep -v '[^a-z0-9] [^a-z0-9]'>tmpfile2
time cat tmpfile2|grep -v '' >tmpfile3
time cat tmpfile3|grep -v '\. '>tmpfile4
time cat tmpfile4|grep -v "[a-z]' " >bigfile.clean

Using temp files here causes the equivalent of 5 full text scans on the data when you should really only be reading the data once.

Word.

2011-09-16T09:11:00.000-07:00

Select 'Text Processing' from UNIX like 'a BOSS'

2011-09-07T08:14:00.000-07:00

1) Learn the commands, avoid re-inventing wheels
Bootstrap method:
First, read every man page and understand that each command can be fed into any other command. There are around 2000 commands on a typical UNIX box but you only need to know a few hundred of these. If you have a specific task you can usually figure out which commands to use with the "apropos" command

Read the entire man page for important commands but just skim the top description for the others (Like encryption or security programs)

Here's how to get a list of commands available on your system (logged as root):
(The script command will screen capture into a file named typescript)
Press the tab key twice to see available commands-
#script
-tab- -tab-

Display all 2612 possibilities? (y or n)
y
! icc2ps ppmtobmp
./ icclink ppmtoeyuv
411toppm icctrans ppmtogif
7za iceauth ppmtoicr
: icnctrl ppmtoilbm
Kobil_mIDentity_switch icontopbm ppmtojpeg
MAKEDEV iconv ppmtoleaf
--More--

...
#ctl-d

# exit
Script done, file is typescript

Now that you have captured all the commands available on your system, process that file to extract commands with man pages.
Get rid of dos newlines
#unix2dos typescript
Manually cleanup "--More--" if needed using emacs or vi editor.
#emacs typescript
Loop over the commands and keep only commands with a man page
# (for k in `cat typescript`; do echo -n "$k="; man $k|wc -l ; echo; done 2>/dev/null)|sort|grep '=[1-9]' >command.list
Look over the list
awk=1612
badblocks=119
baobab=28
base64=60
basename=52
bash=4910
batch=177
bc=794
bccmd=116
bdftopcf=71

2) Start "Reading The Fine Manpages"

# man man

man(1) Manual pager utils man(1)
NAME
man - an interface to the on-line reference manuals
SYNOPSIS
man [-c|-w|-tZHT device] [-adhu7V] [-m system[,...]] [-L locale] [-p string] [-M path] [-P pager] [-r prompt] [-S list] [-e extension] [[section] page ...] ...
man -l [-7] [-tZHT device] [-p string] [-P pager] [-r prompt] file ...
man -k [apropos options] regexp ...
man -f [whatis options] page ...
DESCRIPTION
man is the system’s manual pager. Each page argument given to man is normally the name of a program, utility or function. The manual page associated with each of these arguments is then found and displayed. A section, if provided, will direct man to look only in that section of the manual. The default action is to search in all of the available sections, following a pre-defined order and to show only the first page found, even if page exists in several sections.
...

Commands:

ENTER – scroll down
”b” – take you back
”/ keyword” – search the man page for a word
asumming more as page reader

Finding Commands:

# apropos CPU
top (1) - display top CPU processes
# whatis apropos
apropos (1) - search the manual page names and descriptions
NOTE: apropos is man -k (for ”keyword search”) and whatis is man -f
(for ”fullword search”)

3) Study and practice combining these key text processing tools:
I can't even tell you how important this is.
AWK, SED, TR, GREP, SORT, UNIQ, CUT,PASTE,JOIN, HEAD, TAIL, CAT, TAC

4) Learn how to manage running processes
PS, TOP, FG, BG, KILL, vmstat, iostat, netstat, lsof, fuser, strace
ctrl-d
ctrl-c
ctrl-z

5) Learn the BASH shell
UNIX pipes combine and process streams:

Integer value	Name
0	Standard Input (stdin)
1	Standard Output (stdout)
2	Standard Error (stderr)

6) Keep going
Check out history, wget, curl, xml2, xmlformat and GNU parallel.. tons more....

(You'll be glad you did)

NCD for news article data mining

2011-09-02T12:56:00.000-07:00

The Normalized Compression Distance (NCD) is a powerful formula that can be used to discover hidden relationships in almost any data.

C=compressor program like Zip
x and y are the 2 strings you want to compare

I works by simply compressing three things. The first text string C(x), the second C(y) and then the concatenation both strings together c(xy).

Apply the formula and you get a distance score between the 2 strings from 0 to 1 (plus a slight error amount). 0 means the articles are identical, and 1 means they are absolutely dissimilar.

NCD was discovered by Rudi Cilibrasi and Paul Vitanyi as described in thier 2005 paper "Clustering by Compression"
http://arxiv.org/PS_cache/cs/pdf/0312/0312044v2.pdf

Consider the following article:

What If He Had Gone on Vacation-

Jack Kilby describes how he developed the world's first integrated circuits.

"After several interviews, I was hired by Willis Adcock of Texas Instruments. My duties were not precisely defined, but it was understood that I would work in the general area of microminiaturization. Soon after starting at TI in May 1958, I realized that since the company made transistors, resistors, and capacitors, a repackaging effort might provide an effective alternative to the Micro-Module. I therefore designed an IF amplifier using components in a tubular format and built a prototype. We also performed a detailed cost analysis, which was completed just a few days before the plant shut down for a mass vacation.

"As a new employee, I had no vacation time coming and was left alone to ponder the results of the IF amplifier exercise. The cost analysis gave me my first insight into the cost structure of a semiconductor house. The numbers were high — very high — and I felt it likely that I would be assigned to work on a proposal for the Micro-Module program when vacation was over unless I came up with a good idea very quickly. In my discouraged mood, I began to feel that the only thing a semiconductor house could make in a cost-effective way was a semiconductor. Further thought led me to the conclusion that semiconductors were all that were really required — that resistors and capacitors, in particular, could be made from the same material as the active devices.

"I also realized that, since all of the components could be made of a single material, they could also be made in situ, interconnected to form a complete circuit. I then quickly sketched a proposed design for a flip-flop using these components. Resistors were provided by bulk effect in the silicon, and capacitors by p-n junctions.

"These sketches were quickly completed, and I showed them to Adcock upon his return from vacation. He was enthused but skeptical and asked for some proof that circuits made entirely of semiconductors would work. I therefore built up a circuit using discrete silicon elements. Packaged grown-junction transistors were used. Resistors were formed by cutting small bars of silicon and etching to value. Capacitors were cut from diffused silicon power transistor wafers, metallized on both sides. This unit was assembled and demonstrated to Adcock on August 28, 1958.

"Although this test showed that circuits could be built with all semiconductor elements, it was not integrated. I immediately attempted to build an integrated structure as initially planned. I obtained several wafers, diffused and with contacts in place. By choosing the circuit, I was able to lay out two structures that would use the existing contacts on the wafers. The first circuit attempted was a phase-shift oscillator, a favorite demonstration vehicle for linear circuits at that time.

"On September 12, 1958, the first three oscillators of this type were completed. When power was applied, the first unit oscillated at about 1.3 megacycles.

"The concept was publicly announced at a press conference in New York on March 6, 1959. Mark Shepherd said, "I consider this to be the most significant development by Texas Instruments since we divulged the commercial availability of the silicon transistor." Pat Haggerty predicted the circuits first would be applied to the further miniaturization of electronic computers, missiles, and space vehicles and said that anyapplication to consumer goods such as radio and television receivers would be several years away."

A television program in 1997 said about the integrated circuit and Jack Kilby, "One invention we can say is one of the most significant in history -- the microchip, which has made possible endless numbers of other inventions. For the past 40 years, Kilby has watched his invention change the world. Jack Kilby — one of the few people who can look around the globe and say to himself 'I changed how the world functions.'"

Can NCD work as a relationship classifier in news articles?

First, I ran a part of speech tagger on the article text and then looked-up all entities (proper nouns) in Wikipedia. The Wikipedia articles were then used to build a related word-list for each entity. Next I compute the normalized compression distance (NCD) pair-wise for each entity to get a distance matrix. Cluster the matrix and a nice binary graph appears using dot and the relationships begin to appear.

From the 50 entities mapped below, a few relations are apparent. (Probably should have just picked the top 25 entities though, because the clustering works better with fewer items.)

A) Mark Shepherd and Willis Adcock are related through Jack Kilby

B) The integrated circuit, transistor, capacitor and resistor are related by silicon.

C) Concepts and ideas are related through thought.

Move your web content to Rackspace Cloudfiles

2011-08-15T08:33:00.000-07:00

I 'm gonna make this quick

To start hosting your web content on Rackspace Cloudfiles, you need to complete the following tasks:

Create Cloudfile containers to hold your web content ( plan on storing up to 10K items per container so you can use cloudfuse to mount your containers to the filesystem). You need to come up with an organizational structure that makes sense for your site. For example, you could use a container for your audio named c_audio and c_pics for pictures. If you know you will have over 10K files you should try to break them up like this: c-pics000,c-pics001, c-pics002, ...
Click "Publish to CDN" - This will give you your CDN URL.
Add cname entries into your DNS to create a predictable name for your containers.
Write a PHP script to query your database and copy all content listed in your database into the cloud. Then save the new cloud location for the content back to your database. For example, if you use auto-increment and have 30 thousand files you can put the first 10k into c-pics000 container, second 10K into c-pics001 container and so on.
For user contributed content- update your website file upload script to save the content to the cloud instead of the filesystem and save the new content location in the database. For images you may want to create the different sizes you will need before uploading to the cloud. Example: me.jpg and me_thumb.jpg
Connect your Cloudfile containers to your filesystem with cloudfuse.

Add the following entry to /etc/fstab:

cloudfuse /mnt/cloud fuse defaults,allow_other,username=your_username,api_key=your_api_key

Then mount /mnt/cloud and you will see all your containers as directories. This is a great way to see what you have using ls and also to make backups of your cloudfiles to another hard-drive on your server.

You're done!

Linux virtualization with chroot

2011-08-15T04:22:00.000-07:00

Big savings with virtualization (Exponential?)

According to Moore's Law we get better, and generally cheaper servers every year. In the data center, we can take advantage of this by leasing servers instead of buying- if we are using Linux virtualization.

There are many virtualization containers that work perfectly well for Linux. The three I have used most over the years are Xen, VMware and Oracle VM VirtualBox. Each has it's own quirks you need to learn and requires a software install to use. The one disadvantage they all share though, is that they add complexity and there is a cost to switching containers. For a pure linux shop there is a simpler way to virtualize without adding software.

Good old chroot will do the trick

As long as the host and the virtualized OS will run on the same kernel, you don't need any special software to virtualize in Linux. Using chroot, you can build an OS inside an OS. I currently run CentOS as the host OS and Gentoo for my virtualized instances.

Here is how I set it up:
# cat /etc/redhat-release
CentOS release 5.5 (Final)

#mkdir /mnt/gentoo
#cd /mnt/gentoo

First: Download a tar ball of a running Gentoo OS and unpack it
#wget http://distro.ibiblio.org/pub/linux/distributions/gentoo/releases/amd64/autobuilds/current-install-amd64-minimal/stage3-amd64-20110811.tar.bz2

#bunzip2 stage3-amd64-20110811.tar.bz2
#tar xvf stage3-amd64-20110811.tar

Next: Setup networking and devices, copy these 3 lines into the /etc/rc.local of your host OS (CentOS in my case)
mount -t proc none /mnt/gentoo/proc
cp -L /etc/resolv.conf /mnt/gentoo/etc/resolv.conf
chroot /mnt/gentoo su -c '/etc/init.d/start_gentoo.sh' -

Last: add a shell script into your virtualized OS (/mnt/gentoo) to start your webservers, databases and any other programs you need to run on a reboot.

To login to your new virtualized server
#chroot /mnt/gentoo

Come back in 1 year

To copy it to a new leased machine in a year once prices have dropped and CPU cores have doubled:
Stop the running programs (whatever you put into start_gentoo.sh) so that you don't copy things like your database tables during a write.
#tar cvfz /mnt/gentoo/gentoo.20110815.tar.gz /mnt/gentoo

You can now ftp this to your new server, make a new /mnt/gentoo directory, untar the tar ball, setup your start script and /etc/rc.local and you are up and running on your new, faster machine.
#chroot /mnt/gentoo

Improving a good thing

So that's it! I have had this setup up and working for the past 5 years and every year I change to new servers it gets easier (Hint- if you use the cloud to host your web content, upgrading to new servers gets even easier- see my next post on this topic).

Now I am planning to switch my setup from Gentoo to Ubuntu because Ubuntu has better support in the community. I will write that up once I get it working (I started using Ubuntu initially because it is a source distribution, and due to that, there are tar balls that unpack and work easily in chroot). If anyone has tips on how to get Ubuntu running in a chroot environment, please leave a comment.

HOWTO - Simple Parallel Sort in Linux

2009-02-05T05:57:00.000-08:00

More data than memory

I had duplicate records in a MySQL table with 180 million rows and a varchar index. One requirement I had in de-duping the records was that I needed to keep the row with the highest value in a certain column. This is a VERY common scenario for developers and DBAs but there seems to be no good way to do this in MySQL or Postgreql. A simple "group by order by" does not work in MySQL (5.0.22) for example because it does not know how to presort rows in a group by.

This means that if you want to show the top seller for each day by using a "group by" it will actually pick the first row it finds instead of sorting first to give you the top seller for the day.

I read tons of posts and tried several ways to dedupe the table. Basically everything works and everyone has great advice and tips. But try de-duping a hundred million records that don't fit into memory with whatever query you want on a database, it's just too slow.

Trying MySQL

Since "group by" would not work without an inner query, my idea was to copy the table structure, add a unique key constraint to the primary key on the new table and then "insert ignore" (ordering by my value to get the highest in first) from old table to new. This would leave me with the new table correctly deduped and I would avoid an expensive subquery. I ran this query for 1 week though and it was still killing the server and not even half finished. (I was not batching these inserts so that is something to try next time.)

Try Postgres

I tried postgresql at this point thinking I could take advantage of the DISTINCT ON(field) construct that they have. This seemed like a good way to de-dupe the records so I loaded the data. After getting the data in though, I discovered that building the indexes to just get started was actully slower than in my sql... (I'm not saying postgresql in general is slower than MySQL, it could be faster but it was not going to save the day here)

On to Linux - The Sort Command

I decided to try presorting and deduping the records in Linux PRIOR to loading in the database.

Hardware: 2 Dual-Core AMD Opteron(tm) Processor 2214 HE
OS: 2.6.18-92.1.22.el5 #1 SMP x86_64 GNU/Linux

I have 2 servers like this. Each has 4 cores and 2G ram to work with.

First thing to do is clean up the data and remove any junk rows before starting the sort. You want the file small enough to fit into memory if possible.

step1: left pad any numbers so they sort correctly (integers need 10 places)
awk -F',' '{printf "%s,%s,%10d\n",$3,$4,$5}' file.csv >file.padded.csv

step2: remove non-alphanumeric and junk rows (whatever you can do to get the filesize down)
grep -v '[^a-Z,0-9 ]' file.padded.csv | grep -vE '^junk,|,junk,|^foo,|,foo,' >file.padded.clean.nojunk.csv

step3: sort the rows and keep only uniques (I used reverse sort to keep the max value rows)
sort -fru -S1800M file.padded.clean.nojunk.csv > file.padded.clean.nojunk.sorted.csv

The Job completed in 12 hours on one server. Blowing away the performance of the databases.
But it could be faster.

Linux sorting in parallel (Distributed, Muti-Core)
With 2 servers and 4 cores available to each, it doesn't make sense to run a large sort on only one.

Split the file to be sorted into 4:
wc -l file.padded.clean.nojunk.csv
140000000
split -l 35000000 file.padded.clean.nojunk.csv

files:
xaa
xab
xac
xad

Copy 2 files to the second machine for processing then begin sorting: (4 parallel sorts)
Machine#1
nohup sort -fr -S900M xaa > xaa.sorted
nohup sort -fr -S900M xab > xab.sorted

Machine#2
nohup sort -fr -S900M xac > xac.sorted
nohup sort -fr -S900M xad > xad.sorted

4 Hours later - merge the 2 sort results in parallel (merges are very fast anyway)
Machine#1
nohup sort -frmu xaa.sorted xab.sorted > xa.ab.sorted

Machine#2
nohup sort -frmu xac.sorted xad.sorted > xa.cd.sorted

4 Minutes later - copy the result from server2 to server1 and complete the merge.

Machine#1
nohup sort -frmu xa.ab.sorted xa.cd.sorted > file.padded.clean.nojunk.sorted.csv

All Done

Took just a little over 4 hours this way and with more machines you could get the sort time down to minutes. You are mainly limited by disk write speeds as long as you have gigabit ethernet between servers.

Time everything and make sure to experiment with memory allocation. I noticed that sort merges were much slower when I allocated a lot of extra memory for them.

Follow-up notes:

The main takeaway from this is to process the data until you can fit it into memory. Splitting the data on a Multi-Core server into subsets will allow you to process data in parallel.

For a speed boost change your locale from en_US.UTF-8 to C with export LC_ALL=C
Thanks to Tapajyoti Das for this tip. http://tdas.wordpress.com/2008/02/03/speed-up-grep/
(this also will change the way results are filtered, for example a-Z will match only simple ascii and no accented characters under Locale C- this is often what you want)

Absolute Best Way to Embed Flash in HTML

2008-12-03T11:27:00.000-08:00

*update- doesn't seem to work with Google browser... not sure about safari. i'll update if i get a fix

This is the best way to embed flash into a webpage without javascript. It is a Markup Only solution and because it does not use the embed tag, it will validate.

The HTML object method (nested objects + IE conditional comments):

<object classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000" id="smix" codebase="http://fpdownload.macromedia.com/get/flashplayer/current/swflash.cab" height="30" width="100">
<param name="movie" value="file.swf">
<param name="wmode" value="transparent">
<param name="allowScriptAccess" value="sameDomain">

<object name="smix" type="application/x-shockwave-flash" data="file.swf" height="30" width="100">
<param name="wmode" value="transparent">
<param name="allowScriptAccess" value="sameDomain">
</object>

</object>

Here is a compatability chart for this method from the flash test suite by Bobby van der Sluis.

Browser	IE 5, 5.5, 6	IE 7, 8b2	FF 1, 2, 3Moz 1.7	Saf 1.3, 2, 3*Chrome 0.2	Opera 9, 9.5
Basic	yes	yes	yes	yes	yes
Streaming	yes	yes	yes	yes	yes
Params	yes	yes	yes	yes	yes
Communication	yes	yes	yes	no fscommand	no fscommand

Since I require javascript communication, I was a little worried about Safari support but then realized that I use ExternalInterface instead of the old flash way to communicate with javascript called fscommand.

Flash 8 uses ExternalInterface to do javascript communication without using SWLiveConnect. This was a always a problem with Safari using the Object code above but no more with Flash 8 and up.

Recommendations to use SWFObject are all over the forums and is well promoted but it is too much overhead for my taste. Just remember that IE7 and under will require an extra click from the users unless you use one of the javascript methods. This is due to the stupid Ebola patent that claimed rights to HTML embed technology.

ImageMagick - How to shrink images in batch mode

2008-04-09T09:06:00.001-07:00

find . -size +1000k -name \*.png awk '{print $1,$1}' | xargs -n2 convert -scale 1024x\>

Setup a Future Rescue with the 'at' Comand

2008-02-01T19:53:00.000-08:00

With a leased server, you never want to lock yourself out of your box. If you do get locked out, KVM style console access can save you but it's not cheap. It usually costs $30 per use. And that's if your ISP even supports it.

There are 3 risky things that can get you locked out of your server.
1) service iptables restart
2) service network restart
3) changing etc files like /etc/sudoers ... and then rebooting

For me, iptables is the riskiest to edit because one bad rule can lock you out of ssh on port 22.

To protect yourself, use 'at' to revert the file change 5 minutes in the future:
echo 'mv ifcfg-eth0.save ifcfg-eth0' at now + 5 minute

*this will work even if you are kicked out (as long as the server is running).

--update-- here is another way to prevent getting locked out by iptables during testing:

*/10 * * * * iptables -P INPUT ACCEPT; iptables -P OUTPUT ACCEPT; iptables -P FORWARD ACCEPT; /sbin/iptables -F

This will flush all the rules every 10 minutes, just in case you lock yourself out. When you're happy with the results of your work, remove the line from your crontab, and you're in business.

Consolidate your web architecture -w- vmware

2007-05-05T18:25:00.000-07:00

Good and Cheap Hardware
CPU: AMD64 Dual Core Processor 4400+
Clock: 2.2 GHz
Ram: 1 Gig

1) Select a CentOS 4.4 install (With 5 usable ip addresses)
2) Install vmware server (See last weeks post)
3) Download CentOS 5 virtual appliance and cp into /vm directory
http://www.vmware.com/vmtn/appliances/directory/820
4) Download MySQL Appliance and cp into /vm directory
5) Download memcached Appliance and cp into /vm directory

6) Edit /etc/sysconfig/network-scripts/ifcfg-eth0-range0 to free up 3 ip addresses
## START
#IPADDR_START=72.xxx.xxx.19
IPADDR_START=72.xxx.xxx.22
IPADDR_END=72.xxx.xxx.22
CLONENUM_START=0
##END

7) service network restart
New ifconfig wil show IP's up on eth0 and and one ethernet alias eth0:0

8) run vmware_config.pl
for networking, select vmnet0 to bridge on eth0
set vmware default appliance directory to /vm

9) From a pc start vmware client and connect to the remote server.
10) "Open existing virtual machine" and browse to each appliance to install.
11) Start up both vm appliances and setup networking.
ifconfig eth0 72.xxx.xxx.19
route add default gw [use the same gateway that is used on the host server]
make changes permanent

/etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=static
HWADDR=00:0c:22:32:s0:d4
IPADDR=72.xxx.xxx.19
NETMASK=255.255.55.248
ONBOOT=yes

# vi /etc/sysconfig/network
Append/modify configuration as follows:

NETWORKING=yes
HOSTNAME=www1.fubar.com
GATEWAY=72.xxx.xxx.xxx

Set the correct DNS server defined in /etc/resolv.conf file:
# vi /etc/resolv.conf
nameserver 72.yyy.yyy.yyy

service network restart
ping google.com

12) Assign DNS names to the new servers

13) Install vmware tools. This is not required if you will usually login to the server via ssh.
http://www.thoughtpolice.co.uk/vmware/howto/centos-5-vmware-tools-install.html

Install Vmware on 64 bit CENTOS 4.4

2007-04-29T13:57:00.000-07:00

Here are detailed instructions on how to successfully install vmware on a 64 bit server running CENT0S 4.4

The server:
OS: CentOS 5 x86_64
Hardware: AMD 64 Athlon 4400 DC/1024MB/160GB SATA

# uname -a
Linux xx.xx.xx.xx 2.6.9-42.0.10.ELsmp #1 SMP Tue Feb 27 09:40:21 EST 2007 x86_64 x86_64 x86_64 GNU/Linux

# arch -a
x86_64

#cat /etc/redhat-release
CentOS release 5 (Final) *

[Thanks to my hosting provider LayeredTech. They Rock!]

First, download and install the latest VMware Server. Since there are no x64 rpm files I used the x86. I hope it will efficiently using this CPU.

sh-3.00# rpm -ivh VMware-server-1.0.3-44356.i386.rpm

sh-3.00# vmware-config.pl
The correct version of one or more libraries needed to run VMware Server may be
missing. This is the output of ldd /usr/bin/vmware:
linux-gate.so.1 => (0xffffe000)
libm.so.6 => /lib/tls/libm.so.6 (0xf7fd0000)
libdl.so.2 => /lib/libdl.so.2 (0xf7fcc000)
libpthread.so.0 => /lib/tls/libpthread.so.0 (0xf7fba000)
libX11.so.6 => not found
libXtst.so.6 => not found
libXext.so.6 => not found
libXt.so.6 => not found
libICE.so.6 => not found
libSM.so.6 => not found
libXrender.so.1 => not found
libz.so.1 => not found
libc.so.6 => /lib/tls/libc.so.6 (0x001e6000)
/lib/ld-linux.so.2 (0x001cd000)

This program cannot tell for sure, but you may need to upgrade libc5 to glibc
before you can run VMware Server.

Hit enter to continue.

This is the first hint of trouble. Next after a few more prompts you will see an error message about a missing X library file.

/usr/lib/vmware/bin/vmware-vmx: error while loading shared libraries: libX11.so. 6: cannot open shared object file: No such file or directory
Please enter your 20-character serial number.

This is because vmware server is compiled against X libraries on the x86 architecture. To me this is an odd dependency because most servers do not run Xwindows. Personally, I never use Xwindows on servers, but to get past the vmware-config.pl installation you must install the xorg-x11-libs for x86 platform (even if you are on a x64).

Here is how it went:
rpm -ivh xorg-x11-libs-6.8.2-1.EL.13.37.7.i386.rpm
error: Failed dependencies:
libGL.so.1 is needed by xorg-x11-libs-6.8.2-1.EL.13.37.7.i386
libexpat.so.0 is needed by xorg-x11-libs-6.8.2-1.EL.13.37.7.i386
libfontconfig.so.1 is needed by xorg-x11-libs-6.8.2-1.EL.13.37.7.i386
libfreetype.so.6 is needed by xorg-x11-libs-6.8.2-1.EL.13.37.7.i386
libz.so.1 is needed by xorg-x11-libs-6.8.2-1.EL.13.37.7.i386

OK, so first I need to download the libraries that are needed.

wget ftp://ftp.nluug.nl/pub/os/Linux/distr/smeserver/releases/testing/7.2/smeos/i386/CentOS/RPMS/xorg-x11-libs-6.8.2-1.EL.13.37.7.i386.rpm
wget ftp://ftp.nluug.nl/pub/os/Linux/distr/smeserver/releases/7.1/smeos/i386/CentOS/RPMS/zlib-1.2.1.2-1.2.i386.rpm
wget ftp://ftp.nluug.nl/pub/metalab/distributions/smeserver/releases/7.1/smeos/i386/CentOS/RPMS/expat-1.95.7-4.i386.rpm
wget ftp://ftp.nluug.nl/pub/ibiblio/distributions/smeserver/releases/7.1/smeos/i386/CentOS/RPMS/fontconfig-2.2.3-7.centos4.i386.rpm
wget ftp://ftp.nluug.nl/pub/os/Linux/distr/CentOS/4.4/updates/i386/RPMS/freetype-devel-2.1.9-5.el4.i386.rpm
wget ftp://ftp.nluug.nl/pub/sunsite/distributions/smeserver/releases/testing/7.2/smeos/i386/CentOS/RPMS/xorg-x11-Mesa-libGL-6.8.2-1.EL.13.37.7.i386.rpm
wget ftp://ftp.nluug.nl/pub/os/Linux/distr/CentOS/4.4/updates/x86_64/RPMS/freetype-2.1.9-5.el4.i386.rpm

rpm -ivh xorg-x11-libs-6.8.2-1.EL.13.37.7.i386.rpm xorg-x11-Mesa-libGL-6.8.2-1.EL.13.37.7.i386.rpm expat-1.95.7-4.i386.rpm fontconfig-2.2.3-7.centos4.i386.rpm freetype-devel-2.1.9-5.el4.i386.rpm zlib-1.2.1.2-1.2.i386.rpm freetype-2.1.9-5.el4.i386.rpm
Preparing... ########################################### [100%]
1:zlib ########################################### [ 14%]
2:freetype ########################################### [ 29%]
3:expat ########################################### [ 43%]
4:fontconfig ########################################### [ 57%]
5:freetype-devel ########################################### [ 71%]
6:xorg-x11-Mesa-libGL ########################################### [ 86%]
7:xorg-x11-libs ########################################### [100%]

OK, so now I got xorg-x11 library installed I can retry the vmware-config.pl script.

Please enter your 20-character serial number.

Type XXXXX-XXXXX-XXXXX-XXXXX or 'Enter' to cancel:

Starting VMware services:
Virtual machine monitor [ OK ]
Virtual ethernet [ OK ]
Bridged networking on /dev/vmnet0 [ OK ]

The configuration of VMware Server 1.0.3 build-44356 for Linux for this running
kernel completed successfully.

Success!!

Thanks to Karanbir Singh for his blog post on installing vmware on CENTOS 4.4.
vmware_server_on_x86_64_centos4_redhat_e

Easiest possible way to port forward

2007-04-20T13:04:00.000-07:00

Say you have a mysql server that you do not want to make public. Port 3306 is blocked by your firewall but port 22 is open for ssh.

Just connect tp your mysql server from putty on port 22 and modify one setting.
Connection->SSH->Tunnels add "L3306 127.0.0.1:3306"

After you make this change you can Click on Session and save your changes.

Once you connect through putty, you will be able to access the remote mysql server by localhost:3306.

Warning: Don't forget that this port is local now! If you forget that port 3306 is forwarded, you may wonder why you have a mysql server running on your machine.
If you decide later to install mysql on your local machine you should delete the port forward from the putty config to avoid unintentional changes your remote server.

Awking Mysql

2006-08-19T12:07:00.000-07:00

How do you perform awk type functions in Mysql?

Say you have a list of urls and you want to find the most common domain names hosting gif files.

http://www.fileden.com/123/foo.gif
http://h1.ripway.com/11.gif
http://www.hostmyfile.info/22/web.gif
http://www.fileden.com/24/fun.gif

If this list is in a file you can use standard linux commands to find the most common domains.

grep '\.mp3' my_pictures |awk -F'/' '{print $3}' |sort |uniq -c|sort -nr

If the list is in a mysql database you can use locate() and substring() to accomplish the same thing!

SELECT count(*) as count, substring(url,1,locate('/',url,9)) as f FROM `my_pictures` WHERE url IS NOT NULL and url !='' and filename like '%.mp3' group by f order by count desc

Happy Awking!

phpMyAdmin #1045 - Access denied for user 'root'@'localhost' (using password: NO)

2006-07-11T14:27:00.000-07:00

When installing phpMyAdmin the most common error is not being able to connect to the database after installation.

#1045 - Access denied for user 'root'@'localhost' (using password: NO)

Here is the solution:
Make sure your config file in the phpMyAdmin directory is named "config.inc.php"!

In the installation instuctions it says to copy config.default.php into the phpMyAdmin directory and then customize it (usually the only change needed is to change the 'auth_type' is changed from 'config' to 'http'). Make sure to rename config.default.php to config.inc.php.

Get the Latidude and Longitude from an IP

2006-07-04T07:31:00.000-07:00

ip2location shell script

for k in `cat ip_list`;
do
id=`echo $k|awk -F_ '{print $1}'`;
ip_num=`echo $k|awk -F_ '{print $2}'|awk -F. '{ printf "%10d\n",$1 * 16777216 + $2 * 65536 + $3 * 256 + $4 }'`;
echo id=$id;
echo ip_num=$ip_num;
echo "SELECT ip_latitude,ip_longitude,ip_city FROM ip2location WHERE (ip_from <= $ip_num) AND (ip_to >= $ip_num) LIMIT 1"|mysql -u doug crostel_store;
done

UPDATE my_space_users u,
ip2location i SET u.user_location = i.ip_city WHERE (

ip_from <=1091213045
) AND (
ip_to >=1091213045
) LIMIT 2

mysql> update my_space_users u, ip2location l set u.ip_country=l.country_short, u.ip_region=l.ip_region, u.ip_city=l.ip_city, u.ip_latitude=l.ip_latitude, u.ip_longitude=l.ip_longitude, u.ip_isp=l.ip_isp, u.ip_domain=l.ip_domain WHERE (l.ip_from <= u.reg_ip_num) AND (l.ip_to >= u.reg_ip_num);

Using Wikipedia as a natural language corpus

2006-04-13T09:34:00.000-07:00

1) download the english yahoo abstracts from

http://en.wikipedia.org/wiki/Sort-Merge_Join

wiki.txt
Anarchism is derived from the Greek Î±Î½Î±ÏÏÎ¯Î± ("without archons (ruler, chief, king)"). Anarchism as a political philosophy, is the belief that rulers, governments, and hierarchal social relationships are unnecessary and should be abolished, although there are differing interpretations of what this means.
|
Albedo is the measure of reflectivity of a surface or body. It is the ratio of electromagnetic radiation (EM radiation) reflected to the amount incident upon it.
Abu Dhabi (Arabic: Ø£Ø¨Ù Ø¸Ø¨Ù Ê¼AbÅ« áºaby) is the largest of the seven emirates that comprise the United Arab Emirates and was also the largest of the former Trucial States. Abu Dhabi is also a city of the same name within the Emirate that is the capital of the country, in north central UAE.

wiki.txt1
Anarchism
is
derived
from
the

wiki.txt2
is
derived
from
the
Greek

wiki.txt12
Anarchism is
is derived
derived from
from the
the Greek

wiki.txt.sorted
wiki.txt.sorted.1
wiki.txt.sorted.nonums
wiki.txt.sorted.nonums1
wiki.txt.sorted.nonums1.uniq
wiki.txt.sorted.nonums1.uniq.2words

export LC_ALL=C

cat
wordnet.index
wordnet.index.sorted
wordnet.index.sorted.uniq
wiki.txt.sorted.nonums1.uniq.2words.cols
wiki.txt.sorted.nonums1.uniq.2words.cols.LC_ALL
wordnet.index.sorted.uniq.LC_ALL
wiki-wordnet.join

for k in `cat wordnet.index.sorted.uniq.LC_ALL`; do look $k wiki.txt.sorted.nonums1.uniq.2words.cols.LC_ALL; done >/tar-backups/wiki-wordnet.join

3 word n-grams
grep -v '[^[:print:]]' wiki.txt1 >wiki1.txt
tail +2 wiki1.txt >wiki2.txt
tail +2 wiki2.txt >wiki3.txt

grep "[[:print:]]\+\t[[:print:]]\+\t[[:print:]]" wiki123.txt >wiki123.txt3

find the most used 3 word phrases in a text file
sort rewrite123.txt3 >rewrite123.txt3
grep ".\+\t.\+\t.\+" rewrite123.txt >rewrite123.txt3
sort rewrite123.txt3 >rewrite123.txt3.sorted
uniq -dc rewrite123.txt3.sorted >rewrite123.txt3.sorted.uniqd
sort -nr rewrite123.txt3.sorted.uniqd >rewrite123.txt3.sorted.uniqd.sorted
more rewrite123.txt3.sorted.uniqd.sorted

3 be traveling with
2 you want to
2 want to have
2 student group bookings.
2 part of their
2 on the other
2 notarized letter of
2 made with tap
2 entertainment , activities
2 and want to

Plot a wav file - wav2png

2006-03-18T15:10:00.000-08:00

#!/bin/sh
#usage: wav2png.sh file.wav

BASE=${1%.wav}
PNG=$BASE.png
WAV=$BASE.wav
DAT=$BASE.dat

#echo $BASE,$PNG,$WAV,$DAT
sox $WAV $DAT
grep -v '^;' $DAT >$DAT.clean
FREQ=`head -1 $DAT|tr -d ';'`

echo -e "set terminal png;set title '$FREQ';set output '$PNG'; plot '$DAT.clean'" |gnuplot

Split a midi file into separate tracks

2006-03-18T12:57:00.000-08:00

Get the program:

wget http://interglacial.com/~sburke/pub/midi_splitter.pl

perl midi_splitter.pl /tmp/vanilla.mid
file vanilla*.mid
midi2abc vanilla.mid
midi2abc vanilla_t01_c00.mid >vanilla_t01.abc

Access internet hotspots in client mode with WRT54G

2006-03-16T07:03:00.000-08:00

Linksys WRT54G (below version 5 because linksys later switched to vmware)

Load the sveasoft firmware. (I am using Alchemy-pre5.3 v2.04.4.8sv)

Put it in client mode with loopback off.

Here is a good tutorial:
http://www.engadget.com/2005/05/24/how-to-connect-your-linksys-wrt54g-network-to-the-internet/

ssh into the router and type:
~ # wl scan
~ # wl scanresults|tr -d '\n' |sed 's/noise/\n/g' |tr ']' '\n'|grep ^SSID|awk '{print $(NF-1),$2}' |sed 's/Mode://'|sort -n
-96 "zeus"
-94 "LF-X1U.00014A10A7C8"
-94 "wireless"
-89 "Doug"
-88 "GHOST"
-86 "COR-MDA-LAN"
-86 "linksys"
-86 "linksys"
-83 "default"
-83 "hpsetup"
-83 "linksys"
-78 "hpsetup"
-78 "linksys"
-74 "hirsh
-51 "link"

You will get a sorted list of the most powerfull access points in range.

Split a polyphonic midi file into monophonic tracks

2006-02-28T13:59:00.000-08:00

perl midi_splitter.pl /tmp/vanilla.mid

Music secrets revealed

2005-11-29T10:40:00.000-08:00

" Using Sox, LAME, FLAC, and madplay - I put together this "dictionary" of shell commands that can be run to convert audio from one format to other, including dialup-modem-streamable lofi mono MP3 files, and including shorter clips of music."

-Derek Silvers is the founder and sole programmer of http://cdbaby.com

Well done!

See the article here:
www.oreillynet.com/pub/wlg/5894

Here are a few other usefull linux audio commands:

mp3info /tmp/07_The_Worst_Hangover_Ever.mp3
48 kbit/s j-stereo (2m 57s)

sox /tmp/orig.07_The_Worst_Hangover_Ever.wav -e stat
Samples read: 15695872
Length (seconds): 177.957732
Scaled by: 2147483647.0
Maximum amplitude: 0.999969
Minimum amplitude: -1.000000
Midline amplitude: -0.000015
Mean norm: 0.237282
Mean amplitude: -0.000037
RMS amplitude: 0.312397
Maximum delta: 1.965851
Minimum delta: 0.000000
Mean delta: 0.163578
RMS delta: 0.230147
Rough frequency: 5170
Volume adjustment: 1.000

Using pdftk to process pdf files

2005-11-22T08:40:00.000-08:00

pdftk multimodal.pdf dump_data output report.txt; cat report.txt

pdftk multimodal.pdf dump_data_fields output report.txt; cat report.txt

Yahoo hosting MySQL tricks

2005-10-17T22:45:00.000-07:00

To load a csv file into a MySQL database.

1) FTP the csv file to /tmp/afile.csv directory on your yahoo server.

2)From yourdomain.com/phpMySql run this sql:

LOAD DATA LOCAL INFILE '/tmp/afile.csv' REPLACE INTO TABLE `carrier`
FIELDS TERMINATED BY ''
OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '\\'
LINES TERMINATED BY '\n';

Here is another example:

LOAD DATA LOCAL INFILE "/tmp/IP.CSV" INTO TABLE iploc FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\r\n'

Lame is sweet

2005-10-06T19:59:00.000-07:00

Lame is a pretty cool linux program you can use to batch process MP3 files:

lame --resample 22.05 --scale 10 -b 48 -a HowBizarre.wav -o HowBizarreLoud.mp3

while i'm at it; here is the whole script

#!/bin/bash
for k in *
do
lame --resample 22.05 --scale 10 -b 48 -a $k -o ${k%.wav}.mp3
done

Garbled man pages in Linux?

2005-09-29T09:08:00.000-07:00

I use the putty windows client to connect to my linux boxes and always run into this problem:

>man sort

NAME
sort â sort lines of text files
SYNOPSIS
sort [OPTION]... [FILE]...
DESCRIPTION
Write sorted concatenation of all FILE(s) to standard output.
Ordering options:

Mandatory arguments to long options are mandatory for short options
too.
âb, ââignoreâleadingâblanks ignore leading blanks
âd, ââdictionaryâorder
consider only blanks and alphanumeric characters
âf, ââignoreâcase
fold lower case to upper case characters

Luckily there is an easy fix:

export LANG="en_US"

I don't know why Fedora sets the LANG variable to something that breaks your terminal.

SMS service centers

2005-08-02T15:05:00.000-07:00

t-mobile 12063130004
cingular 13123149810

Fix a hosed Linux terminal

2005-08-02T06:42:00.000-07:00

Are you seeing this:
[⎼⎺⎺├@▒1▒ ·]#
[⎼⎺⎺├@▒1▒ ·]# ⎽├├≤ ␋␌▒␌⎺┼

Instead of this?
[root@a1a ~]#
[root@a1a ~]# ls

Your screen can get hosed if you cat a binary file or for some reason your screen recieved a ^n character. This trash on your screen is called graphics mode. You can fix it by typing:

echo ^O

-while you are typing, it will look like garbage on the screen. Don't worry.
-to print a control O you can type: echo ctl-v ctl-o

More 2 Cent Tips... | Linux Gazette: "To force it back you need to have a shell or other program to send the
^o to the stricken display. I often switch to another virtual terminal
and do an 'echo -e '\017' >/dev/tty2'"

GPRS and WAP settings

2005-07-05T12:07:00.000-07:00

Here is a good link to get worldwide wap settings for pre-paid and contract GSM phones.

http://www.taniwha.org.uk/gprs.html

Send an email to any phone

2005-06-30T09:21:00.000-07:00

Here is a list of email addresses for all cellphones alphabetized by carrier.
http://www.avtech.com/Support/Email/Email_A.htm

Nokia secret codes

2005-06-27T08:33:00.000-07:00

* # 06 # = show the serial number IMEI -
*#92702689# (= *#war0anty#) = show warranty information

Nokia Ringtone Capabilities

2005-06-27T06:13:00.000-07:00

1100 mono -sms delivery
v120t manual
5165 mono - sms delivery
5125 mono - sms delivery
2600 - cingular email

SI WAP PUSH Calculator

2005-06-26T12:56:00.000-07:00

echo -n 0051000B919145557705F400F5AA560605040B8423F0DC0601AE
02056A0045C60C033230322e392e39382e35343a383038302f4347536572
7665722f73657276652e6a73703f69643d31323334353600010349742773
206120424f595c215c215c21000101 |
awk -v FIELDWIDTHS="10 12 6 2 38 90 6 32 6" '{printf "%s,%s,%s,%s,%s,%s,",$1,$2,$3,$4,$5,$6; system("echo "$6" |xxd -r -p"); printf ",%s,%s,",$7,$8; system("echo "$8" |xxd -r -p"); printf ",%s,",$9}' |tr ',' '\n'

0051000B91 #top header
9145557705F4 #11 digit number
00F5AA
56 #bytes in payload
0605040B8423F0DC0601AE02056A0045C60C03 #payload header
3230322E392E39382E35343A383038302F43475365727665722F73657276652E6A73703F69643D313233343536
202.9.98.54:8080/CGServer/serve.jsp?id=123456 000103
49742773206120424F595C215C215C21 #message in HEX
It's a BOY\!\!\!
000101 #footer

Comparing numerical data with sed and diff

2005-06-25T15:59:00.000-07:00

A common situation that occures when working with Hex data is you will have a piece of data that works and one that doesn't. The idea is to compare the 2 HEX strings without pulling your hair out.

Here is a quick way to compare 2 HEX data strings.

"works_good"
0051000B919145557705F400F5AA44605040B8423F0DC0601AE02056A
0045C60C0336362e3137362e3231322e3137342f62622f7377696e672
e67696600010349742773206120424f59202120212021000101

"works_bad"
0051000B919145557705F400F5AA440605040B8423F0DC0601AE02056
A0045C60C0336362e3137362e3231322e3137342f62622f7377696e67
2e67696600010349742773206120626f795c215c215c21000101

Rather than checking the data with a pencil one byte at a time, we can use sed to break up the data into chunks of 10 and then the diff command to show where the first difference is.

echo 0051000B919145557705F400F5AA44605040B8423F0DC0601AE02056A
0045C60C0336362e3137362e3231322e3137342f62622f7377696e672e6769
6600010349742773206120424f59202120212021000101 |sed 's/........../&\n/g'>works_bad

echo 0051000B919145557705F400F5AA440605040B8423F0DC0601AE02056A
0045C60C0336362e3137362e3231322e3137342f62622f7377696e672e67696
600010349742773206120626f795c215c215c21000101|sed 's/........../&\n/g'>works_good

[root@a1a ~]# diff --side-by-side a b |grep --color -C500 '|'

0051000B91 0051000B91
9145557705 9145557705
F400F5AA44 F400F5AA44
605040B842 | 0605040B84
3F0DC0601A | 23F0DC0601
E02056A004 | AE02056A00
5C60C03363 | 45C60C0336
62e3137362 | 362e313736
e3231322e3 | 2e3231322e
137342f626 | 3137342f62
22f7377696 | 622f737769
e672e67696 | 6e672e6769
6000103497 | 6600010349
4277320612 | 7427732061
0424f59202 | 20626f795c
1202120210 | 215c215c21
00101 | 000101

We can easily see that 'works_bad' is missing a 0 in position 40.

Linux Hexadecimal HOWTO

2005-06-24T15:07:00.000-07:00

There are 4 things you need to do with HEX values in LINUX

For the impatient:

1) Convert Hex to Decimal

echo 68 65 6c 6c 6f 20 77 6f 72 6c 64 |tr '[a-z]' '[A-z]' |sed 's/ / p /g' |sed 's/$/ p/'|awk '{print "16i "$0}'|dc |tr '\n' ' '

104 101 108 108 111 32 119 111 114 108 100

4) Convert Hex to ASCII

echo 68656c6c6f20776f726c64 |sed 's/../& /g' |tr '[a-z]' '[A-z]' |sed 's/ / p /g' |sed 's/$/ p/'|awk '{print "16i "$0}'|dc |tr ' ' '\n' |awk '{printf("%c",$0)}'
Even Better -
echo 68656c6c6f20776f726c64 | xxd -r -p

hello world

2)Convert Decimal to Hex

echo 87 87|tr ' ' '\n'|awk '{printf "%x ",$1}'

57 57

3) Convert ASCII to HEX

echo -n hello world |od -tx1 |cut -c8-|tr -d ' \n'

68656c6c6f20776f726c64

More Examples:

If it wasn't so simple, you could put it in a script and call it ascii2hex.sh.

This command is usefull to encode ascii text into 8 bit Hex (UCS2)
for sending the payload of a PDU sms message.

If it wasn't so simple, you could put it in a script and call it ascii2hex.sh.

echo -n "202.9.98.54:8080/CGServer/serve.jsp?id=123456" |od -tx1
0000000 32 30 32 2e 39 2e 39 38 2e 35 34 3a 38 30 38 30
0000020 2f 43 47 53 65 72 76 65 72 2f 73 65 72 76 65 2e
0000040 6a 73 70 3f 69 64 3d 31 32 33 34 35 36
0000055

Just ignore the first column or cut it out and remove the non-text characters with "| cut -c 8- |tr -d ' \n' "
3230322e392e39382e35343a383038302f43475365727665722f73657276652e6a73703f69643d313233343536

Here is a standard 8 bit ascii chart to verify the results.

x="C0 AA BB CC"

echo "16i C0 p AA p BB p CC p" | dc
192
170
187
204

OR

echo -e "16i\t$x"|sed 's/ / p /g' |sed 's/$/ p/'|dc
192
170
187
204

_____________

Single number conversion

Hex to Integer
echo 16i 57 p |dc
87

Integer to Hex
echo 87 16 o p |dc

57

SI WAP PUSH for Linux Hackers

2005-06-24T14:00:00.000-07:00

#send a wap push - si
echo -ne "at+cmgs=101\r" >/dev/modem;sleep 1;echo -ne "0051000B919145557705F400F5AA570605040B8423F0DC0601AE02056A0045C60C033230322E392E39382E35343A383038302F43475365727665722F73657276652E6A73703F69643D313233343536000103616D757A6F2E636F6D2043472030303031000101^Z" >/dev/modem

00 default SMSC
51
00 message ID 00 means let phone set
0B length of destination mobile number = 11
91 the international + sign
0621723364F4 the phone number 60 12 27 33 46 4
00 protocol identifier
F5
AA validity period 4 days

57 length of content
06
05
04
0B84 dest port no
23F0 origin port no
DC Transaction ID (Push ID)
06 PDU type (Push PDU)
01 Header Length (just leave it at 01)
AE Content Type=application/vnd.wap.sic (0x80 | 0x2E)
02
05
6A
00
45
C6
0C http://
03 (next is an ASCII string for the URL, terminate with 00)

3230322E392E39382E35343A383038302F43475365727665722F73657276652E6A73703F69643D313233343536
which means = 202.9.98.54:8080/CGServer/serve.jsp?id=123456

00 terminate string with 00
01
03 (next is an ASCII string for title, terminate with 00)

616D757A6F2E636F6D2043472030303031
which means = amuzo.com CG 0001

00 terminate string with 00
01
01

Calculate the size of the payload message in Hex : (goes after AA)

This works:

echo -n "06 05 04 0B 84 23 F0 DC 06 01 AE 02 05 6A 00 45 C6 0C 03 32 30 32 2E 39 2E 39 38 2E 35 34 3A 38 30 38 30 2F 43 47 53 65 72 76 65 72 2F 73 65 72 76 65 2E 6A 73 70 3F 69 64 3D 31 32 33 34 35 36 00 01 03 61 6D 75 7A 6F 2E 63 00 01 01" |tr '[a-z]' '[A-z]' |sed 's/ / p /g' |sed 's/$/ p/'|awk '{print "16i "$0}'|dc |tr ' ' '\n' |awk '{printf("%c",$0)}'|wc -c |tr ' ' '\n'|awk '{printf "%x ",$1}'
4d

And this is an easier way:

echo -n 0605040B8423F0DC0601AE02056A0045C60C033230322E392E39382E35343A383038302F43475365727665722F73657276652E6A73703F69643D313233343536000103616D757A6F2E63000101 |wc -c |awk '{print $1" * .5"}'|bc |awk -F. '{printf "%x\n",$1}'
4d

Here is how to calculate the overall message length: (headers + payload)
echo -ne "51000B919145557705F400F5AA570605040B8423F0DC0601AE02056A0045C60C033230322E392E39382E35343A383038302F43475365727665722F73657276652E6A73703F69643D313233343536000103616D757A6F2E636F6D2043472030303031000101" |wc -c |awk '{print $1" * .5 "}'|bc |awk -F. '{print $1}'
101

Important AT error messages

2005-06-24T01:07:00.000-07:00

You can find a list of CMS error codes here:

http://www.nowsms.com/discus/messages/1/829.html

CMS Errors:
300: ME Failure;
302: Operation not allowed;
303: Operation not supported;
304: Invalid PDU mode parameter;
305: Invalid text mode parameter;
320: memory failure;
321: invalid memory index;
322: memory full;
330: SCA unknown;
500: Unknown error;

message center number - cingular - +1404727259060

AT Commands - Part 2

2005-06-23T23:44:00.000-07:00

ln -s /dev/rfcomm1 /dev/modem

rfcomm
rfcomm1: 00:60:57:4D:D3:32 channel 1 closed

stty -a < /dev/modem speed 9600 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = ;
eol2 = ; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W;
lnext = ^V; flush = ^O; min = 1; time = 0;
-parenb -parodd cs8 hupcl -cstopb cread clocal -crtscts
ignbrk brkint ignpar -parmrk -inpck istrip -inlcr -igncr icrnl ixon -ixoff
-iuclc -ixany -imaxbel
opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
isig icanon iexten -echo echoe echok -echonl -noflsh -xcase -tostop -echoprt
echoctl echoke

ln -s /dev/ttyACM0 /dev/modem #motorola v188 usb

cat /dev/modem &
echo > /dev/modem #for some reason the first echo gets ignored so we send a dummy
echo -ne "at+cmgf=1\r" >/dev/modem
echo -ne 'at+cmgl="ALL"\r' >/dev/modem #get messages in ASCII

+CMGL: 4,"STO UNSENT","+9545577504",,

How are you?

OK

If you are operating in Text mode AT+CMGF=1 then the command is
AT+CMGL="ALL"

This will list all the SMS in the SIM

If you are in PDU mode then you should use

AT+CMGL=4

>>3) Delete messages from sim

AT+CMGD= is the command to delete the message at position

echo -ne "at+cmgf=0\r" >/dev/modem #read messages in binary
echo -ne 'at+cmgl=4\r' >/dev/modem #ALL does not work in binary mode

+CMGL: 4,2,,24

07913121139418F011000A9159547557400000AA0CC8F71D14969741F977FD07

#hello word in 7 bin binary
echo -ne "at+cmgs=23\r" >/dev/modem;sleep 1
echo -ne "07913121139418F011000A9159346367820000AA0BE8329BFD06DDDF723619^Z" >/dev/modem

+CMGS: 59

OK

#10 digit number beginning with area code "hello world"
echo -ne "at+cmgs=23\r" >/dev/modem;sleep 1;echo -ne "0011000A9159346367820000AA0BE8329BFD06DDDF723619^Z" >/dev/modem

#11 digit number including country code
echo -ne "at+cmgs=24\r" >/dev/modem;sleep 1;echo -ne "0011000B919145337626F80000AA0BE8329BFD06DDDF723619^Z" >/dev/modem

[root@a1a ~]# echo -ne "at+cmgf=0\r" >/dev/ttyACM0
[root@a1a ~]# at+cmgf=0
OK

[root@a1a ~]# echo -ne "at+cmgs=24\r" >/dev/ttyACM0
[root@a1a ~]# at+cmgs=24
>
[root@a1a ~]# echo -ne "0011000B919145337626F80000AA0BE8329BFD06DDDF723619^Z" >/dev/ttyACM0[root@a1a ~]# 0011000B919145337626F80000AA0BE8329BFD06DDDF723619
+CMGS: 3

send a binary message to 9546638946 "hello world" (please change number before trying this)
echo -ne "at+cmgf=0\r" >/dev/ttyS0
echo -ne "at+cmgs=24\r" >/dev/ttyS0; sleep 1;
echo -ne "0011000B919145668349F60000AA0BE8329BFD06DDDF723619^Z" >/dev/ttyS0

echo -ne "at+cmgs=15\r" >/dev/modem;sleep 1;echo -ne "0011000B919145337626F80004AA0122^Z" >/dev/modem #send ok but phone gets a "message could not be displayed"

Cellphone forum links

2005-06-17T08:07:00.000-07:00

forums.wirelessadvisor.com

Phone Museum

2005-06-15T13:15:00.000-07:00

Here are the phones I own:

NOKIA
3390b- GSM 14.99 (t-mobile) + 10 s/h =$24.99 - OR-andale = $20 , 572 offered, 291 sold - found a lot of 45 used for $900= $20 each
3650- GSM -
6010- GSM (supposedly datacable DKU-5)
3120b - GSM -(data cable, DKU-5 $20)
3595- GSM (data cable 3950 - $6.99)
6340i - GSM 850 1900 ALL TDMA frequencies and Analog, WAP 1.2.1, wireless modem 9.6 kbps,infrared, Nokia 6340 PC : 10 new for $400 on ebay=$40
5165p - tdma
5165p - tdma

MOTOROLA
v120T- tdma
v171 - gsm TF - EMS version 5.0

SAMSUNG
samsung SPHA740 - sprint - bob

SONY ERICSSON
T68 gsm-features

SEIMENS
CF62 - t-mobile prepaid

Phones I would like to buy:
3395- wap 1.1 19.99 + 12.50 s/h = $32.49
motorola c155 -TF- EMS version 5.0

A Simple ASCII to Hex Convertor

2005-06-15T06:14:00.000-07:00

This command is usefull to encode ascii text into 8 bit Hex (UCS2)
for sending the payload of a PDU sms message.

If it wasn't so simple, you could put it in a script and call it ascii2hex.sh.

echo -n "202.9.98.54:8080/CGServer/serve.jsp?id=123456" |od -tx1
0000000 32 30 32 2e 39 2e 39 38 2e 35 34 3a 38 30 38 30
0000020 2f 43 47 53 65 72 76 65 72 2f 73 65 72 76 65 2e
0000040 6a 73 70 3f 69 64 3d 31 32 33 34 35 36
0000055

Just ignore the first column or cut it out and remove the non-text characters with "| cut -c 8- |tr -d ' \n' "
3230322e392e39382e35343a383038302f43475365727665722f73657276652e6a73703f69643d313233343536

HEX to ASCII

echo 31 32 33 34 35 36 |sed 's/ / p /g' |sed 's/$/ p/'|awk '{print "16i "$0}'|dc |tr ' ' '\n' |awk '{printf("%c\n",$0)}'
1
2
3
4
5
6

Here is a standard 8 bit ascii chart to verify the results.

HOWTO - Setup a Serial Connection to a Bluetooth Phone

2005-06-14T15:06:00.000-07:00

I am running Fedora Core 3 - FC3 with a kensington usb bluetooth adapter. FC3 comes with the bluez software so I didn't need to install anything for bluetooth to work.

0) # service bluetooth status; service bluetooth stop; service bluetooth start; service bluetooth status
1) unplug the usb bluetooth adapter then plug it back in
2) # hcifconfig hci0 down
3) # hcifconfig hci0 up
4) # hcifconfig -a

hci0: Type: USB
BD Address: 00:0C:55:F8:5E:73 ACL MTU: 377:10 SCO MTU: 16:0
UP RUNNING PSCAN ISCAN
RX bytes:378 acl:0 sco:0 events:16 errors:0
TX bytes:325 acl:0 sco:0 commands:17 errors:0
Features: 0xff 0xfe 0x0d 0x38 0x08 0x08 0x00 0x00
Packet type: DM1 DM3 DM5 DH1 DH3 DH5 HV1 HV2 HV3
Link policy: RSWITCH HOLD SNIFF PARK
Link mode: SLAVE ACCEPT
Name: 'a1a-0'
Class: 0x120104
Service Classes: Networking, Object Transfer
Device Class: Computer, Desktop workstation
HCI Ver: 1.2 (0x2) HCI Rev: 0x0 LMP Ver: 1.2 (0x2) LMP Subver: 0x6965
Manufacturer: Broadcom Corporation (15)

5) #hcitool scan
Scanning ...
99:66:55:44:D1:35 Nokia3650

6) rfcomm bind /dev/rfcomm1 99:66:55:44:D1:35 1
7) You should now be able to use the serial connection to communicate with your bluetooth phone.

If you get the following error:

# cat /dev/rfcomm1
cat: /dev/rfcomm1: Connection refused

1) Then turn the phone off and back on.
2) hcitool scan
Scanning ...
99:66:55:44:D1:35 Nokia3650
3)cat /dev/rfcomm1
Waiting with no error

This usually fixes the problem.

Using the Nokia 3650 for internet access -Linux FC3

2005-06-10T13:18:00.000-07:00

Carrier: Cingular
Plan: $19.99 monthly internet
Phone: Nokia 3650
OS: Fedora Core 3 - FC3
Bluetooth device: Kensington USB adapter

The output from a successfull GPRS ppp connection:

[root@a1a peers]# pppd call gprs
--> WvDial: Internet dialer version 1.54.0
--> Initializing modem.
--> Sending: ATH
NO CARRIER
ATH
OK
--> Sending: ATE1
ATE1
OK
--> Sending: AT+CGDCONT=1,"IP","isp.cingular","",0
AT+CGDCONT=1,"IP","isp.cingular","",0
OK
--> Modem initialized.
--> Sending: ATD*99#
--> Waiting for carrier.
ATD*99#
CONNECT
~[7f]}#@!}!} } }2}#}$@#}!}$}%\}"}&} }*} } g}%~
--> Carrier detected. Waiting for prompt.
~[7f]}#@!}!} } }2}#}$@#}!}$}%\}"}&} }*} } g}%~
--> PPP negotiation detected.
Serial connection established.
using channel 6
Using interface ppp0
Connect: ppp0 <--> /dev/modem
sent [LCP ConfReq id=0x1 ]
rcvd [LCP ConfRej id=0x1 ]
sent [LCP ConfReq id=0x2 ]
rcvd [LCP ConfAck id=0x2 ]
rcvd [LCP ConfReq id=0x0 ]
sent [LCP ConfAck id=0x0 ]
sent [PAP AuthReq id=0x1 user="ISPDA@CINGULARGPRS.COM" password="CINGULAR1"]
rcvd [PAP AuthAck id=0x1 ""]
PAP authentication succeeded
sent [CCP ConfReq id=0x1 ]
sent [IPCP ConfReq id=0x1

]
rcvd [IPCP ConfReq id=0x0 ]
sent [IPCP ConfNak id=0x0 ]
rcvd [LCP ProtRej id=0x0 80 fd 01 01 00 0c 1a 04 78 00 18 04 78 00]
rcvd [IPCP ConfNak id=0x1

]
sent [IPCP ConfReq id=0x2

]
rcvd [IPCP ConfReq id=0x1 ]
sent [IPCP ConfAck id=0x1 ]
rcvd [IPCP ConfAck id=0x2

]
not replacing existing default route to eth0 [192.168.1.1]
local IP address 166.138.237.249
remote IP address 10.0.0.1
primary DNS address 66.209.10.201
secondary DNS address 66.209.10.202
Script /etc/ppp/ip-up started (pid 5672)
Script /etc/ppp/ip-up finished (pid 5672), status = 0x0
Terminating on signal 2.
Script /etc/ppp/ip-down started (pid 5797)
sent [LCP TermReq id=0x3 "User request"]
rcvd [LCP TermAck id=0x3]
Connection terminated.
Connect time 31.6 minutes.
Sent 62981 bytes, received 152749 bytes.

Now from another xterminal you can rin ifconfig and look for the new interface - ppp0
ppp0 Link encap:Point-to-Point Protocol
inet addr:166.138.195.72 P-t-P:10.0.0.1 Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1
RX packets:4 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:3
RX bytes:64 (64.0 b) TX bytes:116 (116.0 b)

After this connection was made there are 2 options:
1) replace my default gateway with 10.0.0.1.
route del default gw 192.168.1.1; route add default gw 10.0.0.1
or
2) add a route to get to the cingular gateway for my phone (ip 66.209.11.61 -taken from my phone settings)
route add -net 66.209.11.0 netmask 255.255.255.0 dev ppp0

Destination Gateway Genmask Flags Metric Ref Use Iface
10.0.0.1 * 255.255.255.255 UH 0 0 0 ppp0
66.209.11.0 * 255.255.255.0 U 0 0 0 ppp0
192.168.1.0 * 255.255.255.0 U 0 0 0 eth0
default 192.168.1.1 0.0.0.0 UG 0 0 0 eth0

tested by:
ping 66.209.11.61
and
browsing to 64.233.161.99 (this is www.1.google.com)

Works OK except for DNS.

My Favorite Nokia AT commands

2005-06-10T10:09:00.000-07:00

#/bin/sh
#make the rfcomm device files
c=0;
while [ $c -lt 256 ]; do
echo c=$c
if [ ! -c /dev/rfcomm$c ]; then
mknod -m 666 /dev/rfcomm$c c 216 $c
fi
c=`expr $c + 1 `
done

service bluetooth status
service bluetooth start
service bluetooth status
hciconfig hci0 down && hciconfig hci0 up
rfcomm bind 1 00:60:57:4D:D3:32 1
rfcomm show
rfcomm1: 00:60:57:4D:D3:32 channel 1 closed
rfcomm release 1
rfcomm bind 1 00:60:57:4D:D3:32 1
rfcomm show
rfcomm1: 00:60:57:4D:D3:32 channel 1 clean

[root@a1a peers]# hcitool scan
Scanning ...
00:60:57:4D:D3:32 Nokia3650

hcitool rssi 00:60:57:4D:D3:32; echo -e "at+csq\r" >/dev/modem
RSSI return value: 0
echo -e 'AT+CSCA?\r' >/dev/modem #to get the message center number from the phone
AT+CMGF=?
echo -e 'at+cmgs="9543367628"\r' >/dev/modem; echo -e 'Hi Bonita^Z' >/dev/modem
echo -e 'at+cmgl="ALL"\r' >/dev/modem
echo -e 'at+cbc\r' >/dev/modem #check battery
at+cbc

+CBC: 0,45

OK

2005-06-09T22:43:00.000-07:00

Nokia AT GPRS command set

Bluetooth GPS ????

2005-06-09T11:44:00.000-07:00

In this Thesis, Bluetooth is promoted as a technogy to enable location
aware services. The authors discuss the problems of trying to use RSSI
radio signal strength to triangulate position. They built a test system with
30 DLINK DBT-120 USB Bluetooth adapters costing $30 each and
using USB hubs with an external power supply. The USB hubs allow the
Bluetooth adapters to be initialized on a laptop and then disconnected from
the computer once initialized. As long as the power supply is not interrupted,
the beacons will continue to broadcast.

http://people.csail.mit.edu/albert/ pubs/ashuang-sm-thesis-2005.pdf

Using the Nokia 3650 for internet access -Windows XP

2005-06-08T15:30:00.000-07:00

I bought the Nokia 3650 to use as a backup internet connection for emergencies or when I'm on the road. After a day of hassling with it I finally got it setup and working in Windows XP with my Kensington bluetooth usb adaptor.

There is a lot of information on the web about connecting this phone to the internet with a cingular sim card.

I saw that there are basically 4 steps:
1) Download and install the Nokia 3650 PC Suite and the nokia drivers "Nokia 3650 Modem Setup " http://www.nokiausa.com/support/phones/software/1,3048,3650,00.html

2) Turn on bluetooth in the phone and pair the connection to the computer.
3) Create a new dialup connection
4) Enter the username, password and number to dial
5) Connect

On step 5 I could not get past this error message when trying to connect:

"Error with PPP connection: 734"
"Error 734: The PPP link control protocol was terminated."

As a test, I downloaded and ran the Cingular connection manager,
CingularSetup_4_2_7_1.exe -> http://www.cingular.com/media/downloads

But I was getting the same error using both applications. I decided to call Cingular. It turns out that I was not provisioned to use GPRS modem because I was on the 2.99 internet express plan. They said I needed the 19.99 internet connection plan so I signed up for that and they gave me a user/pass to connect.

Dialup Connection Information:
User Name: ISPDA@CINGULARGPRS.COM
Password: CINGULAR1
Dial: *99#

I have seen a lot of comments on the internet saying you can leave the username and password blank, or put in random text but this did not work for me. Without the correct username and password I get the error message:

"Error 734: The PPP link control protocol was terminated."

They changed the number to dial to *99***1# but I also tested *99# for the number and it worked just as well.

2005-06-05T08:55:00.000-07:00

XFS undelete HOWTO:

How to undelete a file in a linux XFS filesystem:

"I moved to the next possibility and clicked ‘undeploy’.
Suddenly- all my application and
data files were deleted by tomcat."

Background (How I accidently deleted all my data):

At the end of a 4 week project to make a shipping portal website, I decided to try out the Jakarta-Tomcat web admin interface to start and stop the application instead of sticking to the command line startup and shutdown scripts that I am familiar with.

I logged into the admin console (jakarta-tomcat-5.5.9) and proceeded to restart the application by clicking ‘start’ and then ‘stop’. This did not show the changes I was expecting so I needed to try something else. With each button click, a generic window pops up to say “Are you sure?” Am I sure about what? I wondered. I moved to the next possibility and clicked ‘undeploy’. Suddenly- all my application and data files were deleted by tomcat.

Reminder- to tomcat, undeploy means you want all your code and project files deleted.

I ran df –k praying that this filesystem was your friendly ext3 but instead it read xfs. Dread.

Linux XFS FAQ

Q: Does the filesystem have a undelete function?

There is no undelete in XFS, in fact once you delete something, the
chances are the space it used to occupy is the first thing reused.
Undelete is really something you have to design in from the start.
Getting anything back after a accidental rm -rf is near to impossible.

This called for extreme measures. I had to bring the files back.

The Undelete:

They say it is near to imposible to recover files in XFS but I did it, and here’s how:

df –k /usr/local/jakarta-tomcat/webapps

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda7 26245376 13740048 12505328 53% /usr/local

I knew that it was important not to disturb the file system where the files had been deleted, but I had production data on part of that disk so I could not unmount it. Also, I did not have 26 Gig’s of free space on the machine to make a safe copy. This meant that I had to work fast and avoid all file creation on the hda7 filesystem.

I used the time command on every search I ran because when grepping 26 Gig files it is very important to manage the amout of time spent searching for data.

First I ran some tests on 1 Gig of raw XFS data to find the fastest way to search for strings. I searched for '$WGET' because it was a variable i remembered that was in the lost file.

time dd if=/dev/hda7 bs=1024 count=1000000 | strings | cut -c0-50 | grep '$WGET'

$WGET --post-data="prefix1=$PREFIX&number1=$NUMBER
$WGET --post-data="prefix1=$PREFIX&number1=$NUMBER
1000000+0 records in
1000000+0 records out
real 4m35.243s
user 1m24.390s
sys 0m20.810s

Now the better way.

time dd if=/dev/hda7 bs=1024 count=1000000 | grep -a '$WGET' | strings

1000000+0 records in
1000000+0 records out
$WGET --post-data="prefix1=$PREFIX&number1=$NUMBER&DoIt=Do+it"
$WGET --post-data="prefix1=$PREFIX&number1=$NUMBER&DoIt=Do+it"
real 0m55.836s
user 0m4.350s
sys 0m18.310s

Both command statements searched 1 million records of data (1,000,000 x 1024=1Gig) but moving the strings command to after the grep and taking out the cut is 500% faster. This is important because I need to search 26 times this amount of data.

Now I began the search using a unique keyword that I knew was present in the deleted file.

time grep -a –B100 –A100 "$WGET" /dev/hda7 | strings

[3%?%p1%{1}%=%t4%e%p1%{3}%=%t6%e%p1%{4}%=%t1%e%p1%{6}%=%t3%e%p1%d%;m
#6t1~
fo< x=")&" i="n?&s%p1%d%;m">[99H

#!/bin/sh
#045 24630815
usage () {
echo "Usage:"
echo
echo '-p 3 digit prefix'
…
…
…
my_count=100
WGET='wget -q --user-agent="Internet Explorer 5.5" --wait=1 --timeout=10 --tries=2 -O -'
#begin the web get

$WGET --post-data="prefix1=$PREFIX&number1=$NUMBER&DoIt=Do+it" http://www.demo.com/Ingles.jsp grep -A60 "Details" grep -E "::' sed 's:::' sed 's:::g'tr '<' ',' sed 's/>/,_/g' awk -F, '{print $3}' grep -v _Time tr -d '_' paste -d= 5.template -

#script done

#6t1~
fo< ~b~Ht s3Q+ k&s@ vAJ,%

real 19m57.031s user 0m35.090s sys 4m19.860s

As you can see, in 20 minutes I found my deleted file stashed in 26 Gigs of binary junk!

It was not practical to attempt recovery of all my files this way so I just recovered the most valuable programs. Other jsp’s I rewrote based on html I was able to recover from the browser cache of another computer and the most recent backups I had.