Swapping your linux boot drive

My main home server's drive had been giving me SMART errors for a while.
It's a Samsung 1TB disk that I've been using for a couple of year, so it was time to change it

This server runs my MythTV backend, and various services that I've talked about here (most important being my energy monitor: http://htpc.avenard.org/power). It is powered by Linux Ubuntu 10.04 LTS

So after considering the issue, I decided to replace it with an Intel 520 60GB SSD drive.

Changing your hard drive is usually not a too complex task, you plug the new hard drive in, copy the file across with your tool of choice, re-install the GRUB boot loader, disconnect the old drive, put the new one instead. Reboot

Unfortunately here, my PC has *only* 8 SATA ports, 1 for the boot disk, and 6 used for the RAID5 array (made of 6, 2TB disks), and one use for the Bluray optical disk.

So where am I going to connect my new SSD drive. I could use a USB caddy, but that would be slow as.

I had a spare PC, so I connected the disk on the spare PC.

Using gparted I created a GPT partition format (my PC BIOS is an EFI one).
I created a partition taking the whole disk, aligned to 1MB start.
This was a mistage, as I found out later, as to install GRUB boot loader on the disk, you need space at the beginning of the disk.

So if using GPT, create a partition at least 1MB in size, that you mark in GRUB as boot_grub, and another ext4 (or ext3 your choice) partition taking the remaining space (also aligned to 1MB)

I then mounted the new ext4 partition on /ssd as follow:

sudo mkdir /ssd
sudo mount /dev/sdd1 /ssd

/dev/ssd1 is the device for the big partition I created above, you can see what device to use in gparted.

I used rsync to copy all the files from my boot disk on the server into the new disk
the copy the file across

sudo rsync --archive --stats --numeric-ids --delete-during --partial --inplace --hard-links --compress --exclude-from=exclude.default.linux --rsh="/usr/bin/ssh -i /home/jyavenard/.ssh/id_rsa" --log-file=sync.log root@htpc:/ /ssd/

/home/jyavenard/.ssh/id_rsa is the path to my RSA private key that I use to login into my server: htpc
Make sure that root login is allowed on the server you want to clone, so is defined in /etc/ssh/sshd_config and add or uncomment the line:

PermitRootLogin yes

the file exclude.default.linux contains the list of all the directories and file I do not want to copy (mainly temporary files, cache, and /data which is where the RAID array is mounted)

The content of the file is as follow:

/sys/*
/proc/*
*mozilla/firefox/*/Cache/**
/var/lib/vservers/vs1/home/*
*/.googleearth/Cache/**
*/.googleearth/Cache/temp/**
/var/spool/squid/**
/backup/*
/var/spool/cups/**
/var/log/**.gz
*/cache/apt/archives/**
/var/lib/vservers/vs1/var/tmp/**
/home/programs/tmp/**
/home/programs/vmware/**
/home/**/.thumbnails/**
/home/**/.java*/deployment/cache/**
/home/**/profile/**
/home/**/.local/Trash/**
/home/**/.macromedia/**
/export/*
**.core
/usr/ports/**
/dev/**
/usr/compat/linux/proc/**
/pool/**
/data/**
/.amd_mnt/*
/.Spotlight-V100/**
/Network/Servers/**
/export/shares/**

Note that at no time during rsync was htpc offline. It was still doing his duty in the mean time.

Once rsync had finished, I rebooted htpc in single user mode and started networking.

I re-ran the rsync on the remote PC once again, so make sure I had all the latest changes.

Once it was completed, I unmounted the SSD from the remote PC, disconnected it and then shut down the server.
Removed the boot disk, and installed the new SSD.

Now as it is, this new setup will not boot. There aren't any bootloader installed.
I had a copy of the MythBunutu LiveCD handy, so I booted htpc with it.

To install the bootloader, I needed grub2 as my disk is using GPT.
So I mounted the SSD with:

sudo mount /dev/sda1 /mnt
sudo mount --bind /dev /mnt/dev
sudo mount --bind /proc /mnt/dev

The commands above make sure the device directory and proc can be seen on the mnt mount point.
/dev/sda1 is the device for the new partition

chroot /mnt
grub-install /dev/sda
grub-mkconfig --output=/boot/grub/grub.cfg

That last line is important as your existing grub.cfg from the previous disk wouldn't work. Most Linux distribution use the disk UUID to identify a partition. That UUID is invalid on the new disk.

Before I could update, I had to make sure /etc/fstab was correct ; you should normally only have to edit the line where it mount the / root directory
For me I changed it from:

/dev/sda1 / ext3 rw,relatime,errors=remount-ro 0 0

to

/dev/disk/by-uuid/d62feed7-8003-4be7-9eae-8a24a83beb6f / ext4 rw,relatime,errors=remount-ro,barrier=1,data=ordered 0 0

To find out what the line should be, look into:
/proc/mount
and check where / is mounted.

Now we're all done and you can enjoy your new faster disk

If you had a spare SATA port on the PC you want to clone, a very similar approach could have been taken, it would only have been much faster.
You would have mounted the disk to /ssd using the same command as before; add /ssd/** to the exclude.default.linux file and run the command:

sudo rsync --archive --stats --numeric-ids --delete-during --partial --inplace --hard-links --exclude-from=exclude.default.linux / /ssd/

You could also install grub without using the LiveCD too.

Copying a disk with rsync is the fastest way of doing it, it only copies what needs to be copied, it can be interrupted and resumed at any time.

How to change all svn:user entry in one go

Over the years, our username has changed. Like over the years I have been avenardj, jyavenard, jeanyves_avenard and finally jean-yves.avenard

In the 7 years I've been using our SVN repository, looking at the history, it's now all over the place.

So I wanted to make my username consistent...

Preliminary steps: allow changing property in the svn repository by adding a hooks/pre-revprop-change file.

it woud contain only one thing:
exit 0

Because in our setup we use ViewVC, we also needed a hooks/post-revprop-change with:


REPOS="$1"
REV="$2"
USER="$3"
PROPNAME="$4"
ACTION="$5"

/var/www/viewvc/bin/svndbadmin update "$REPOS" "$REV"


I didn't feel like writing much code do to what seemed to be a trivial task.

Looking at various solutions I found this interesting project: XMLSH. Alas, it's quite poorly documented.

After going through the examples, I came up with:



svn log --xml | xread logs
entries=<[ $logs/log/logentry ]>
for entry in $entries; do
rev=<[$entry/@revision]>
echo "entry: " $rev
author=<[$entry/author/string()]>
echo $author
author2=$(echo $author | sed -e 's/(avenardj|jeanyves_avenard|jyavenard)$/jean-yves.avenard/g')
if [ "$author" != "$author2" ]; then
echo " -> $author2"
svn propset --revprop -r $rev svn:author $author2
fi
done


Obviously, replace the regex with the one suitable for you.

How do solar panels perform according to solar exposure..

The Australian bureau of Meteorology provides daily solar exposure data on their web site.

This map shows the amount of energy received from the Sun in a given location.

This was very interesting stuff (well, at least for me).

I wrote a little utility to retrieve the data in a form suitable for post-processing in a spreadsheet and for use in my power tracking system .

This utility is available there. It's written in python, run it with --help as an argument to see the various options available.

One direct application, was measuring on how well the CEEG panels I have installed performed according to the amount of energy they receive from the sun.

It gave me this graph:

The black line shows the average of Export/Import. Any points above the line means the system performed better than average, any points below the line, mean they performed worse.

A clear trend shows that my system performs really well when there's a lot of sun, but rather poorly in low light conditions.

It's rather a worry, as it means they will likely perform poorly during winter.

Stay in touch!

Energy tracking...

I got a solar system installed early December 2009 ; it's a 2kW system made of 12 CEEG 170W panels with a Power-One Aurora Inverter PVI-2000

I had to track the electricity production ; but also wanted to track my power usage.

I couldn't find any proper solutions letting me record everything at once:

• How much is used in the house
• How much is produced by the solar panels
• How much is imported from the electric network grid
• How much is exported back to the grid

So I designed my own...

I bought a CurrentCost CC128 which can be interfaced to a PC through a serial interface.
The Aurora inverter has a RS485 interface; a cheap RS232 converter on eBay for $6.50 did the trick.

The system runs on almost any machines (tested on FreeBSD, Linux, MacOS).

It is made of 3 parts.
-Daemons polling the various sensors ; they poll every 6s ; compute an average every minute and feed the data in a RRDtool database (round robin database).
-A Daemon computing the import and export value in 30 minutes interval (it wakes up every 30 minutes, calculate the import/export over the past 30 minutes and feed another database
-A web page

The daemons are written in python ; as long as you have python running on your machine with the pyserial module you'll be fine.

For the HTML interface ; it's written in PHP on the server side; and javascript/ajax on the client side (though it will work even with javascript disabled, albeit with limit interactivity).

It's probably easier to get it working with the Apache web server and the mod_php5 module...

The PC needs to be on 24/7 as neither the CC128 nor the Aurora inverter keeps any data in memory...

The system is done in such a way that you can easily use a different power monitor or a different inverter.
Everythin can be configured through a simple configuration file: colours, shapes, labels, database used etc...
You don't even need a solar system ; I set the same system at my office to monitor our power usage.

The response to this tool has been very overwhelming.
I have been approach by various parties to integrate with some professional solutions.

I contacted the Google PowerMeter project in order to feed the data through their system, but at this stage they weren't interested, prefering to work utility providers. Their loss !

Some real-time graph samples are shown below.

http://htpc.avenard.org/power.

If you are interested, drop me an email.

Easy FreeBSD Upgrade ...

Over the past few weeks, I updated all my machines from FreeBSD 6.4 to the latest FreeBSD 8.

Upgrading FreeBSD has always been a bit of a pain (even makes me miss linux here and all its binary distributions) as I do not have physical access to most machine and always do so remotely.

I found this great site:
http://people.freebsd.org/~rse/upgrade/

I followed all the steps gron from FreeBSD 6 to FreeBSD 7 then FreeBSD 8.

Painless upgrades; all remotely !

Update for Kindle DX with Global Wireless (DX International)

Amazon just released a new Kindle DX which can now works outside the US using the GSM network.

It has a new device id and 3rd party packages written for the previous Kindle DX won't work.

It's really just a matter of repackaging them and the OS is virtually the same.

So I updated the packager to support this new device ; and repackaged the usbnetwork and screen saver hack package..

Packager (version 0.8):
http://www.avenard.org/kindle2/packager-0.8.tar.gz

USB network (version 0.11)
http://www.avenard.org/kindle2/usbnetwork23-0.11.tar.gz

Screen saver (version 0.1)
http://www.avenard.org/kindle2/screensaverhack-0.1.tar.gz

Amazon's Kindle new Firmware 2.3

Amazon released firmware 2.3 a few days ago.

Before you can update to firmware 2.3 ; you *must* first uninstall all hacks previously installed ; in particular the screen saver hack and savory.

Other than the advertised new features such as PDF native reader ; what's more interesting (at least for me) is that the same kind of security that got added with Firmware 2.2 on the Kindle 2 International is now found on all Kindle model.

Each software installed on the Kindle, must be signed using Amazon's RSA key; which obviously we don't have.

Luckily, the same trick I did for the Kindle 2 International works fine with the 2.3 firmware...

You can find the new packager there:
http://www.avenard.org/kindle2/packager-0.7.tar.gz

I have changed the options format, so it's more in line with other unix command. Make sure to read the README.txt file included in the archive for the syntax.
In short, for firmware 2.3 ; you will want to use the --sign option (which will sign the packages using our own key) and the --ex (which will add auto-jailbreaking to the package)

For Kindle 2 US only users ; once you have updated your Kindle to firmware 2.3 ; the various hacks you enjoyed won't work anymore. You need to get new versions of those

The most popular hack is the screen saver one ; you can find an updated version that works with 2.3 (all kindle model) here:
http://www.avenard.org/kindle2/screensaverhack-0.1.tar.gz

USB network hack:
http://www.avenard.org/kindle2/usbnetwork23-0.9.tar.gz