XFS undelete HOWTO:

How to undelete a file in a linux XFS filesystem:

"I moved to the next possibility and clicked ‘undeploy’.
Suddenly- all my application and
data files were deleted by tomcat."

Background (How I accidently deleted all my data):

At the end of a 4 week project to make a shipping portal website, I decided to try out the Jakarta-Tomcat web admin interface to start and stop the application instead of sticking to the command line startup and shutdown scripts that I am familiar with.

I logged into the admin console (jakarta-tomcat-5.5.9) and proceeded to restart the application by clicking ‘start’ and then ‘stop’. This did not show the changes I was expecting so I needed to try something else. With each button click, a generic window pops up to say “Are you sure?” Am I sure about what? I wondered. I moved to the next possibility and clicked ‘undeploy’. Suddenly- all my application and data files were deleted by tomcat.

Reminder- to tomcat, undeploy means you want all your code and project files deleted.

I ran df –k praying that this filesystem was your friendly ext3 but instead it read xfs. Dread.

Linux XFS FAQ

Q: Does the filesystem have a undelete function?

There is no undelete in XFS, in fact once you delete something, the
chances are the space it used to occupy is the first thing reused.
Undelete is really something you have to design in from the start.
Getting anything back after a accidental rm -rf is near to impossible.

This called for extreme measures. I had to bring the files back.

The Undelete:

They say it is near to imposible to recover files in XFS but I did it, and here’s how:

df –k /usr/local/jakarta-tomcat/webapps

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda7 26245376 13740048 12505328 53% /usr/local

I knew that it was important not to disturb the file system where the files had been deleted, but I had production data on part of that disk so I could not unmount it. Also, I did not have 26 Gig’s of free space on the machine to make a safe copy. This meant that I had to work fast and avoid all file creation on the hda7 filesystem.

I used the time command on every search I ran because when grepping 26 Gig files it is very important to manage the amout of time spent searching for data.

First I ran some tests on 1 Gig of raw XFS data to find the fastest way to search for strings. I searched for '$WGET' because it was a variable i remembered that was in the lost file.

time dd if=/dev/hda7 bs=1024 count=1000000 | strings | cut -c0-50 | grep '$WGET'

$WGET --post-data="prefix1=$PREFIX&number1=$NUMBER
$WGET --post-data="prefix1=$PREFIX&number1=$NUMBER
1000000+0 records in
1000000+0 records out
real 4m35.243s
user 1m24.390s
sys 0m20.810s

Now the better way.

time dd if=/dev/hda7 bs=1024 count=1000000 | grep -a '$WGET' | strings

1000000+0 records in
1000000+0 records out
$WGET --post-data="prefix1=$PREFIX&number1=$NUMBER&DoIt=Do+it"
$WGET --post-data="prefix1=$PREFIX&number1=$NUMBER&DoIt=Do+it"
real 0m55.836s
user 0m4.350s
sys 0m18.310s

Both command statements searched 1 million records of data (1,000,000 x 1024=1Gig) but moving the strings command to after the grep and taking out the cut is 500% faster. This is important because I need to search 26 times this amount of data.

Now I began the search using a unique keyword that I knew was present in the deleted file.

time grep -a –B100 –A100 "$WGET" /dev/hda7 | strings

[3%?%p1%{1}%=%t4%e%p1%{3}%=%t6%e%p1%{4}%=%t1%e%p1%{6}%=%t3%e%p1%d%;m
#6t1~
fo< x=")&" i="n?&s%p1%d%;m">[99H

#!/bin/sh
#045 24630815
usage () {
echo "Usage:"
echo
echo '-p 3 digit prefix'
…
…
…
my_count=100
WGET='wget -q --user-agent="Internet Explorer 5.5" --wait=1 --timeout=10 --tries=2 -O -'
#begin the web get

$WGET --post-data="prefix1=$PREFIX&number1=$NUMBER&DoIt=Do+it" http://www.demo.com/Ingles.jsp grep -A60 "Details" grep -E "::' sed 's:::' sed 's:::g'tr '<' ',' sed 's/>/,_/g' awk -F, '{print $3}' grep -v _Time tr -d '_' paste -d= 5.template -

#script done

#6t1~
fo< ~b~Ht s3Q+ k&s@ vAJ,%

real 19m57.031s user 0m35.090s sys 4m19.860s

As you can see, in 20 minutes I found my deleted file stashed in 26 Gigs of binary junk!

It was not practical to attempt recovery of all my files this way so I just recovered the most valuable programs. Other jsp’s I rewrote based on html I was able to recover from the browser cache of another computer and the most recent backups I had.