Archive for the ‘Bash’ Category

Rename your photos from the comand-line using jhead

Monday, July 12th, 2010

Anyone who want to simply rename his or her photo’s to a date format, will appreciate the ease of batch renaming them from the command-line.

jhead is the tool of choice for manipulation of images on the command-line using EXIF tags. There is a version available for all operating systems.

I like to rename my photos to the date-time the image was taken so I will use this command:

jhead -autorot -nf%y%m%d-%H%M%S *.jpg

But that is just the start of what this little program can do for you:
jhead can extract the following from an Exif  jpeg file:

  • Camera make and model
  • Time and date picture was taken
  • Integral low-res Exif thumbnail
  • Shutter speed
  • Camera F-stop number
  • Flash used (yes/no)
  • Distance camera was focused at
  • Focal length and calculate 35 mm equivalent focal length
  • Image resolution
  • GPS info, if stored in image
  • IPTC header
  • XMP data

You can download this program and find more examples on retrieving data on the authors website.h

Backup large databases with mysqldump by splitting up the backup files

Sunday, May 30th, 2010

The primary responsibility of MySQL professionals is to establish and run proper backup and recovery plans.

The most used method to backup a MySQL database is the mysqldump utility. This mysqldump utility creates a backup file for one or more MySQL databases that consists of DDL/DML statements needed to recreate the databases with their data. To restore a database, the backup file is simply read back into the MySQL utility command prompt as an input file.

This is fine for smaller databases, but for large databases the resulting sql file will be very, very large. If this file is too large to handle, it will be impossible to open it in an editor and fix problems in the script if you need to. This will make the recovery of your database very difficult. For example my trusted vim editor would not load a 2GB file at all, and handling slightly smaller files is too slow to do any work within the file.

One way to solve this is to split up the restore script into multiple parts. The most logical way is to split it up by table name.
You will end up with a large amount of sql scripts that you need to restore.

Since this is a cumbersome job to do, we will write a script that does the following:

  • backup multiple databases. For each database:
    • Backup the DDL of all tables in a separate sql file.
    • Make a backup per table name.
    • Write a script that we can run to restore the DDL and all tables in one command.
    • Compress the generated scripts.

As an added value you will have the possibility to edit the restore script and uncomment the generation of the DDL, exclude some tables for restoration etc. without opening one very large file.

I will not cover sending the compressed file by FTP and/or sending emails and scheduling cron jobs. The script below will be stripped from all but the essential code to perform the task of backing up the databases in small parts.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
#!/bin/bash
### change the values below where needed.....
DBNAMES="MyDb1 MyDb2 MyDb3"
HOST="--host=localhost"
USER="--user=root"
PASSWORD="--password=MyPassword"
BACKUP_DIR="/MyBackupDirectory"

#### you can change these values but they are optional....
OPTIONS="--default-character-set=latin1 --complete-insert --no-create-info --compact -q"
RESTORESCRIPT="$BACKUP_DIR/__restoreData.sql"
DATE=`/bin/date '+%y%m%d_%H%M%S'`

#### make no changes after this....
#### start script ####
echo removing old temporary files if they exists...
rm -f ${BACKUP_DIR}/*.sql > /dev/null 2>&1
rm -f ${BACKUP_DIR}/*.tar > /dev/null 2>&1
cd ${BACKUP_DIR}

for DB in $DBNAMES
do
    echo "=========================================="
    echo ${DB}
    echo "=========================================="
    echo 'SET FOREIGN_KEY_CHECKS=0;' > $RESTORESCRIPT

    mysqldump --no-data $HOST $USER $PASSWORD $DB > ${BACKUP_DIR}/__createTables.sql
    echo 'source __createTables.sql;' >> $RESTORESCRIPT

    for TABLE in `mysql $HOST $USER $PASSWORD $DB -e 'show tables' | egrep -v 'Tables_in_' `; do
        TABLENAME=$(echo $TABLE|awk '{ printf "%s", $0 }')
        FILENAME="${TABLENAME}.sql"
        echo Dumping $TABLENAME
        echo 'source' $FILENAME';' >> $RESTORESCRIPT
        mysqldump $OPTIONS $HOST $USER $PASSWORD $DB $TABLENAME > ${BACKUP_DIR}/${FILENAME}
    done

    echo 'SET FOREIGN_KEY_CHECKS=1;' >> $RESTORESCRIPT

    echo making tar...
    tar -cf ${DB}_${DATE}.tar *.sql  > /dev/null 2>&1

    echo compressing...
    gzip -9 ${DB}_${DATE}.tar > /dev/null 2>&1

    echo removing temporary files...
    rm -f ${BACKUP_DIR}/*.sql > /dev/null 2>&1
    rm -f ${BACKUP_DIR}/*.tar > /dev/null 2>&1

    echo "done with " $DB
done

echo "=========================================="
echo "            done with all database!       "
echo "=========================================="

Before you run this script:

  • Make sure the backup directory exists and you have the permissions to write and delete files in this directory otherwise the script will hang.
  • Have a list of database names and the credentials to make the backup, usually this is the root user.

You can uncompress the files with:

$ gunzip MyDatabaseName.tar.gz
$ tar -xf MyDatabaseName.tar

And you can restore the database with:

$ mysql -uroot -pYourPassword -h YourHost YourDatabaseName < __restoreData.sql

And finally the most important thing in the whole process; Test your backups now!
If you make daily backups but you do take the time to test your recovery plan regularly, you will end up with a big problem when you have to do it under pressure of time.

Exclude items in a list in a bash script

Saturday, May 22nd, 2010

In a Linux bash script you can loop over a set of data like a list of directories, database table names etc.
But you do not always want to use all the items in this list so you need to filter the results.

In this example I will use a generated list of directory names.
Filtering the directories we want to exclude is done by removing them from the directory listing using sed.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
DIRS=`ls -l --time-style="long-iso" $MYDIR | egrep '^d' | awk '{print $8}'`

# make a list of directories you want to exclude
$DIREXCLUDE="dir1 dir3 dir5"

# now remove the excluded directories from the directory list by looping over the directories and remove the excluded directories with sed:
for EXCLUDE in $DIREXCLUDE
do
    DIRS=`echo $DIRS | sed "s/\b$EXCLUDE\b//g"`
done

# and finally loop over the cleaned up directory list.
for DIR in $DIRS
do
    echo  ${DIR} :
done

* see a previous article about selecting directory names in a cron job.

List directory names in bash shell

Tuesday, April 6th, 2010

Here is a little trick you should know when selecting directories in a Linux bash script:

Getting a list of directory names in a bash shell is a simple task:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#!/bin/bash

$MYDIR="/var/log"

DIRS=`ls -l $MYDIR | egrep '^d' | awk '{print $8}'`

# "ls -l $MYDIR"      = get a directory listing
# "| egrep '^d'"           = pipe to egrep and select only the directories
# "awk '{print $8}'" = pipe the result from egrep to awk and print only the 8th field

# and now loop through the directories:
for DIR in $DIRS
do
echo  ${DIR}
done

A script like this is usually needed for a job that needs to be scheduled with cron, for example to make nightly backups, scheduled svn updates etc.

But there is a little catch;
When running a script in cron, the environment in which the script runs can be different and the “ls -l” command may result in a different layout.

The problem is in the time format which can throw off the result that the awk part of the script returns.

Sometimes you need to use: awk '{print $8}'

And in another environment you need to use: awk '{print $9}'

To prevent maintaining two versions of the same script you need to make sure the time format is always the same, without altering the environment variables, which could have side effects.

The solution is something like this:
DIRS=`ls -l --time-style="long-iso" $MYDIR | egrep '^d' | awk '{print $8}'`

the: –time-style=”long-iso” makes sure that the same format for the date-time string is the same in all environments.