Thursday, December 5, 2013

How to write random data to disk FAST

Background:
I got into a heated discussion about the use /dev/zero with the unix utility dd in order to benchmark speed disk.

To me, writing zero serves little purpose, it is highly compressible to start with. Aside writing data linearly, is hardly a method to benchmark your disk speed!

To deal with compression, you must write data that just do not compress: that mean randomisation.

But one of the opposite argument was that you couldn't because you would have to use /dev/random as source of random data, and that's too slow.

/dev/random requires high level of entropy as to provide good random values; it is also very CPU intensive. But one of the main issue with /dev/random is that it is blocking if it doesn't have enough entropy. And it will block until it gets some.

So yes, /dev/random isn't a proper source of data for dd.
On Linux /dev/urandom is a better choice, but that's still too slow (and on FreeBSD /dev/urandom is just a link to /dev/random)

Well, you don't have to use /dev/random to start with, there are other ways to generate random data at much faster rate.

And so come dd2.

dd2 provides similar features to dd (though, at this stage, it doesn't work with pipes), only files or block devices.

It takes the following optional arguments
-i : input file name (equivalent to if=)
-o : output file name (equivalent to of=)
-b : block size for read/write (equivalent to bs=)
-n : number of blocks to write in total (equivalent to count=)
-z : write zeros instead of random data (this is much faster than reading from /dev/zero: 175 times faster on OS X 10.9)

block size and number of blocks takes an integer. So if you want bs=1k, use -b 1024 etc..

The difference with dd is that by default, it generates random data very quickly. Quickly enough as to make it negligible.

There are multiple randomisation algorithms available.
  • INTELSSE: a SSE2 accelerated LCG (Linear Congruential Generator) Algorithm.
    Original code found here. Only modifications I made was to make it compile with gcc and clang
  • ARCRANDOM: arc4 random number generator. Available on BSD systems (including OSX). It uses the key stream generator as employed by the arc4 cypher
  • RANDOMC: a collection of C of seven fast random generators by George Marsalia. 
INTELSSE is very fast, the fastest; giving over 7GB/s of random generation on a i7-2600 CPU. But it's far from perfect as far as random quality is concerned.
ARCRANDOM is the slowest, and really only provided there for comparison purposes. It is too slow for testing disk speed (97MB/s on my system)
RANDOMC provides seven methods: LIFB4, SWB, KISS, CONG, SHR3, MWC and FIB. You can read all about them there.

All methods but INTELSSE or RANDOMC-FIB generates data incompressible by either gzip or bzip2.
INTELSSE and RANDOMC-FIB will compress at around 50%: but they are *FAST*

I recommend to use SWB; it's fast enough (close to 2GB/s) for testing all spinning hard disk, including striped raid.


You can find the source code of dd2 here

with gcc compile with:

gcc -O -msse2 -o dd2 dd2.c

with clang:
clang -O -msse2 -o dd2 dd2.c

-msse2 isn't required unless you use the INTELSSE randomiser.

Enjoy.

Wednesday, November 13, 2013

ZFS / RAIDZ Benchmarks - Part 2

Table of Content


  • Introduction
  • Hardware Setup
  • Description
  • Hiccups
  • Benchmarks
  • Intel NASPT
    • Bonnie++
    • iozone
    • phoronix test suite
  • Conclusions

Introduction


I became in need for a NAS solution, something to store all my files and provide a great level of safety and redundancy.

ZFS was always going to be my choice of filesystem; it worked well, provide lots of useful features (especially snapshots) and is very reliable.

I looked at the existing professional solutions, but none of them provided the flexibility I was after.
FreeNAS' backer iXsystems has an interesting FreeNAS Mini; but only allows a maximum of four disks; and their professional solutions was just outside my budget.


It's been a while since I had done a ZFS benchmarks (check zfs-raidz-benchmarks I wrote in 2009).
So I'm at it again.

Hardware Setup


- Supermicro SC826TQ-500LPB 12 bays chassis
- Supermicro X10SL7-F motherboard
- Intel E3-1220v3 processor
- 32GB RAM (made of 4*8GB Kingston DD3 1600MHz ECC KVR16E11/8EF)
- 6 x WD RED 4TB



from the top

 two chassis: 24 bays total


Description


The chassis comes with a 500W platinum rated redundant power supplies; it's rated at 94% for 20% load and 88% at 10% load. Even with 12 disks, it won't ever go over 25% load so this power supply is overkilled, but it's the smallest Supermicro has.

The X10SL7-F has 6 onboard SATA connectors plus a LSI 2308 SAS2 with 8 SAS/SATA ports.

ZFS shouldn't run on top of hardware RAID controller, it defeat the purpose of ZFS. The LSI was flashed with an IT firmware, making the 2308 a plain HBA.

The plan was to use RAIDZ2 (ZFS equivalent to RAID6), which provides redundancy for two simultaneous disk failures). RAIDZ2 with six 4TB disks would give me 16TB (14TiB) of available space.

The system could later be extended with six more disks... As this is going to be used as a MythTV storage center, I estimate that it will reach capacity in just one year (though MythTV perfectly handles auto-deleting low-priority recordings)

The choice came between the new Seagate NAS drives and WD. My primary concern was power consumption and noise: the WD being 5400 drives win power-wise, but the Seagate are a tiny bit more quiet. Anandtech review also found that IOPS on the WD Red were slightly better: this and the lower power consumption made me go for the WD: the noise difference being minimal.

While I like to fiddle with computers, I'm not as young as I used to and as such, I wanted something that would be easy to use and configure: so my plan was to use FreeNAS.

FreeNAS is a FreeBSD based distribution that makes everything web configurable... It's still not for the absolute noob, and requires that you have good understanding of the underlying file system: ZFS.

FreeNAS runs of a USB flash drive in read-only mode, and let you install all of the FreeBSD ports in a jail residing in the ZFS partitions..


Hiccups


My plans became slightly compromised once I put everything together and realise how noisy that setup was. The Supermicro chassis being enterprise-grade, its only concern is to keep everything cool. But damn, that thing is noisy: no way I could ever have this in any room or office.

There's nothing in the motherboard BIOS allowing you to change the fans speed. The IPMI web access let you choose the fan speed mode: but the choice ends up being between "Normal" which wouldn't let anyone sleep, and "Heavy" which would for sure wake-up the neighbours.

The fans on this motherboard are controlled by a Nuvoton NCT6776D.  On linux the w83627ehf kernel module let you control most of the PWM including the fans speed, unfortunately I found no equivalent on FreeBSD. So if I'm to run FreeBSD I would have to use an external voltage regulator to lower the fans: something that doesn't appeal to me.

Also, this motherboard and chassis provides SGPIO interface to control the SAS/SATA backplane and indicates the status of the drives. This is great to identify which disk is faulty as you can't always rely on the device name provided by the OS.

However, I connected all my drives to the LSI 2308 controller.
Despite my attempts, I couldn't get the front LEDs to show the status of the disk in FreeBSD. Something I could easily do under Linux using the sas2ircu utility...

I like FreeBSD, and always used it for servers, but its lack of hardware gimmick like this started to annoy me. As part of my involvement in the MythTV projects (www.mythtv.org), I have switched to Linux for all my home servers, and I've grown familiar to it over the years.

A few months ago, I would have never considered anything but FreeBSD as I wanted to use ZFS, however the ZFS On Linux (ZOL) project recently called their drivers as "ready for production".... So could it be that linux be the solution?

So FreeBSD or Linux?

I ran various benchmarks, here are the results...


Benchmarks



lz4 compression was enabled on all ZFS partitions.

Intel NAS Performance Toolkit (via Windows and samba: gigabit link)




FreeNASUbuntu RAIDZ2Ubuntu md+ext4
Test NameThroughput(MB/s)Throughput(MB/s)DifferenceThroughput(MB/s)
HDVideo_1Play93.402102.6269.88%101.585
HDVideo_2Play74.33195.03127.85%101.153
HDVideo_4Play66.39595.93144.49%99.255
HDVideo_1Record104.36987.922-15.76%208.868
HDVideo_1Play_1Record63.99197.15651.83%96.807
ContentCreation10.36110.4330.69%10.734
OfficeProductivity51.62756.86710.15%11.405
FileCopyToNAS56.4250.427-10.62%55.226
FileCopyFromNAS66.86885.65128.09%8.367
DirectoryCopyToNAS5.69216.812195.36%13.356
DirectoryCopyFromNAS19.12727.2942.68%0.638
PhotoAlbum10.40310.884.59%12.413

Bonnie++

FreeNAS 9.1


Version 1.97Sequential OutputSequential InputRandom
Seeks
Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
ports64G19799741314745226586346995153176268612.6316++++++++++++++++++++++++1153736++++++++2203171
Latency55233us117ms4771ms129ms760ms817msLatency16741us78us126us145ms23us92942us

Ubuntu - ZOL 0.6.2


Version 1.97Sequential OutputSequential InputRandom
Seeks
Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
ubuntu63G199991102305916865847749898153986266445.41016++++++++++++++++++++++++3017396++++++++++++++++
Latency50051us60474us326ms79062us93147us133msLatency20511us236us252us41664us10us356us

Ubuntu - MD - EXT4


Version 1.97Sequential OutputSequential InputRandom
Seeks
Sequential CreateRandom Create
SizePer CharBlockRewritePer CharBlockNum FilesCreateReadDeleteCreateReadDelete
K/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPUK/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU/sec% CPU
ubuntu63G10869513749211133601757588557541815571.8716298520++++++++++++++++++++++++++++++++++++++++
Latency18937us146ms634ms23816us86087us141msLatency47us223us227us39us17us35us

iozone


This NAS will mostly deal with very big files, so let's specifically test those (ext4 perform especially poorly here)
started with iozone -o -c -t 8 -r 128k -s 4G


FreeNASUbuntu ZOL 0.6.2Ubuntu md+ext4
TitleKB/sKB/sKB/s
Children see throughput for 8 initial writers38100.2738065.716141.23
Parent sees throughput for 8 initial writers38096.537892.056140.56
Min throughput per process4762.324749.14767.58
Max throughput per process4763.044769.23767.74
Avg throughput per process4762.534758.21767.65
Min xfer4193664.00 KB4176640.00 KB4193536.00 KB
Children see throughput for 8 rewriters36189.4736842.595938.99
Parent sees throughput for 8 rewriters36189.1236842.265938.93
Min throughput per process4523.494602.7742.25
Max throughput per process4524.14609742.52
Avg throughput per process4523.684605.32742.37
Min xfer4193792.00 KB4188672.00 KB4192896.00 KB
Children see throughput for 8 readers4755369.064219519.44541271.02
Parent sees throughput for 8 readers4743778.254219187.22540861.93
Min throughput per process593043.56527155.6958198.66
Max throughput per process596547.44527774.5671080.23
Avg throughput per process594421.13527439.9367658.88
Min xfer4169728.00 KB4189696.00 KB3438592.00 KB
Children see throughput for 8 re-readers4421317.254648511.624961596.72
Parent sees throughput for 8 re-readers4416015.24648083.464593093.04
Min throughput per process539363.12580726.531874.36
Max throughput per process558968.815815854619527.5
Avg throughput per process552664.66581063.95620199.59
Min xfer4048000.00 KB4188288.00 KB29184.00 KB
Children see throughput for 8 reverse readers5082555.841929863.779773067.07
Parent sees throughput for 8 reverse readers4955348.811898050.519441166.11
Min throughput per process426778183929.87041.22
Max throughput per process991416.19381972.759710407
Avg throughput per process635319.48241232.971221633.38
Min xfer1879040.00 KB2075008.00 KB3072
Children see throughput for 8 stride readers561014.62179665.1911963150.31
Parent sees throughput for 8 stride readers559886.81179420.5411030888.26
Min throughput per process57737.7319092.5611788.88
Max throughput per process107340.936176.449238066
Avg throughput per process70126.8322458.151495393.79
Min xfer2268288.00 KB2221312.00 KB5376
Children see throughput for 8 random readers209240.1693627.6413201790.94
Parent sees throughput for 8 random readers209234.7493625.1212408594.53
Min throughput per process25897.3811702.1972349.58
Max throughput per process27949.8111704.49059793
Avg throughput per process26155.0211703.451650223.87
Min xfer3886464.00 KB4193536.00 KB34688
Children see throughput for 8 mixed workload91072.2924038.27Too Slow
Cancelled
Parent sees throughput for 8 mixed workload17608.6323877.52
Min throughput per process2305.942990.74
Max throughput per process20461.233021.75
Avg throughput per process11384.043004.78
Min xfer472704.00 KB4151296.00 KB
Children see throughput for 8 random writers36929.7737391.58
Parent sees throughput for 8 random writers36893.0536944.62
Min throughput per process4615.374643.6
Max throughput per process4617.824704.09
Avg throughput per process4616.224673.95
Min xfer4192128.00 KB4140416.00 KB
Children see throughput for 8 pwrite writers37133.2336726.14
Parent sees throughput for 8 pwrite writers37131.3136549.34
Min throughput per process4641.494575.27
Max throughput per process4641.974600.88
Avg throughput per process4641.654590.77
Min xfer4193920.00 KB4171008.00 KB
Children see throughput for 8 pread readers4943880.54806370.25
Parent sees throughput for 8 pread readers4942716.334805915.05
Min throughput per process617373.75595302.62
Max throughput per process619556.38603041.88
Avg throughput per process617985.06600796.28
Min xfer4179968.00 KB4140544.00 KB

phoronix test suite


results are available here (xml result file here)

Comparison including md+ext4 raid6 here


Conclusions


Ignoring some of the nonsensical data found by the benchmarks above which indicates pure cache effect; the ZFS On Linux drivers are doing extremely well, and Ubuntu manages on average to surpass FreeBSD: that was a surprise...

Maybe time to port FreeNAS to use linux as kernel? That would be a worthy project...

Wednesday, February 6, 2013

Windows Phone 8: one month on

I got my Lumia 920 exactly a month ago...

I had an iPhone 5 before that and I wanted something new, something different. I had worked with a HTC Mozart (WP 7.x) two years earlier and I was impressed on well it worked.

It's been a month, and I don't know on how to choose between hating the thing but put up with it, or just go back to using an iPhone.

I like the physical shape of the phone, it's big, but not too big. While the specs state it's a much heavier phone than the iPhone, you can't feel it. The screen is beautiful. Navigating in the phone is very smooth, I like the look and feel. I love the tiles and the swiping left/right to see different views.
From a navigation perspective, very little have changed since windows phone 7: it was good then, it's still good now.

Using the phone for its primary purpose : making phone calls has actually been a challenge. Every time I use it, I get frustrated on how cumbersome it is.

First, there's no different volume settings. If I want to phone ring to be loud, I have to bump the volume to the max. Then everything is loud. I just want the ringtone to be loud. I don't need to be deafened when I type a number in the dialer !

Phone Dialer

The default one is terrible.
I found RapDialer which is a brilliant app and brings most of what's missing, but it doesn't show incoming calls or missed calls so you still have to use two dialers.
Favorites: I can't call a person with a single or two taps. I have to go into People, choose a person, then choose the number to call. Sure you can create groups but that only narrows the selection, still can't define a single number as favorite.
Even RapDialer whenever I select a favourite ask me if I really want to dial... Duh! why else would I tap on a name in my favourite?
I don't care to see facebook update status, or what their latest photos are when all I want to do is make a phone call.

Some people will comment that you can pin a contact to the start screen. That's true. But first, all you see there is a photo, usually attached to the facebook account: which can change, making it hard to find the right person. Then, how crowded can the start screen be, there's no pages there, it's a long scroll view...

It's fine to make a fancy, very socially oriented phone, and I think they've succeeded. But not at the expense of making the primary function of a phone cumbersome.

Answering a call

So I have to swipe up *AND* then select to answer the call???
Why can't it just answer the call when I swipe the screen up like any phones out there?

Maps/Location/Search:


What a disaster that thing...
So yesterday, I search for bunnings warehouse. I live in Melbourne. the first result was for Bunnings in Ballarat, 2 hours away. 2nd one was in Sydney.

The results list showed the address and phone number and how many km away I was from it. Not sure how it's sorted, but it's certainly not by distance.

So I tap on the entry, and it open a new thing with "about / buzz / apps"... no phone number. So how do I call it? well, tough luck I only have to remember the number from the result list, go into the dialer and dial it manually, no copy paste for you !

So let's drive to that place.. I tap on the address field, and it opens maps. Great... but I want to drive there... I tap on the name in map, and ..... it reopens the "about / buzz / apps" screen
Oh, there I can "drive" using an app, get a choice between gMaps Pro or Nokia Drive
Nokia Drive.
Ooops, can't find the address I've just entered and the thing just sit there doing nothing.
Funnily, if I search for the exact same address (again, no copy paste there) it does find it.
And don't get me started for when I search: tiles. Even going into "local" shows me stuff that aren't local except a *carpet* place that I know is around the corner.
Search is a big fail IMHO compare to what I could get on google... Unfortunately, can't have anything but Bings in there.

Another example...
But search say "Tiles Ringwood"
Ringwood being a suburb
I get 3 answers:
1) Ringwood Melbourne, Australia
2) Ringwood Melbourne, Australia
3) Tills Dr, Warrandyty, 3113 Melbourne
Totally useless
Nokia Maps provides much more valuable answer !
1) Ringwood Melbourne, Australia
2) Ringwood Melbourne, Australia
3) Ringwood Health Centre
4) Ringwood Private Hospital
Very useful indeed.
Mind you, the only reason I tried to narrow to Ringwood is because it couldn't find any tiles place in Ringwood
Bing is a tad better , it does find a tiles store located in Ringwood and that's the first result. Interestingly the first result is in ringwood, the next result in ringwood is the 8th result and the 16th. None of the other are in Ringwood
maps.google.com.au finds 10 tiles store, all in ringwood.



Web browser

well, nothing extraordinary there. I find navigating easier on an iPhone. on the iphone I can create a new tab with one tap. Here I have to go into the ... menu and select tabs.
More annoying is search in the address bar: auto-spelling is disabled there and with the very cramped keyboard it's much more difficult to get it right the first time. so you keep correcting what you typing.
Let's start about auto-entry here.
The auto-correction is brilliant, and does a great job. But really, why make the space bar so tiny when keys like [.] aren't required (I can get it with a double tap on space). I don't care about a dedicated smiley key, I only ever use type :( or :). Keep hitting the , key when I just want to enter a space. Maybe it's due to the length of my thumb when I type single-handed
I have my keyboard settings set to UK because I want to type colour not colour. Good luck finding a $. Have to type a pound, hold it and wait for the $ to appear. Cumbersome. Where are my Australian keyboard settings !!??

There's a big screen, yet typing a SMS, there's less than ¼ of the screen space available to what I'm typing. And how many times have I started typing something single-handed to find that I inaverdently tap on the search button: Bing here I come.

Which brings up to the back button. So you press back. Well too bad with what you were currently doing. It's a stack, it pops a level... can't get back to where you were earlier.

Oh, sure, I can tap the back button, hold it, wait, to see a list of where to go back to. Funnily the first thing you can go back to is the current application. Because really, I press the back button and hold it by accident very often. So it's useful to go back to where I am currently.

Internet Browser / Text Editor:

I can never easily select anything and copy it.. I just don't get it onw how selection works.
Trying to position the cursor to correct a word is convoluted. First you tap it puts the cursor where you tap great, but try to bit more precise and it places the cursor about 1cm above where you're holding it: weird. iOS or Android way is so much more intuitive
No password saving ??!! no text reflow so it automatically adjusts and reformats text so it's easier to read like on iOS..

Skype

Microsoft has an answer to Apple's iMessage and FaceTime. It's called skype.
So why can't I type a skype message from within the messaging application.
Why does entering Skype takes such a long time with first "loading...." then "resuming..."
Why can't I start a phone call using skype from the phone application? or even enter the skype details of a contact within the "People" entry.
No, it needs to have two address book and be in a separate applications.
What a let down.
And most of the time when people call me on my skype account, only skype on my desktop rings. Phone stays silent.
I find I'm using the phone less and less during the day, which is great for battery life cause when I'm using the phone for web browsing, it won't even last a day.

Applications

I like The Age application, it's actually more nicely done than on other platforms.
I've trawled the application store for hours and hours... it's pretty hard to find anything interesting or that works well with windows 8; they all seem to be Windows 7 that luckily still works on W8.
Content there is piss-poor, somehow I know doubt it will ever take off but one can hope.

Printing

Don't hope to print with it, cause there's nothing available.

Mail

There's a massive screen, yet the content shown is less than on a much smaller iPhone. And when you reply, the keyboard and icon bar takes over 2/3rd of the screen.
Destination recipient are dumb, I explained:
I receive an email, I press reply type my answer send.
Oh, I forgot to say something. so in the mail list I type reply again... And I reply.... to *myself*. I must select the original message in the thread, and then press reply.
Surely, few people will ever try to reply to their own message hoping to be the only one reading it !
It's all tiny usability details like that, that make a frustrating experience.

System Settings

Ok.. Can someone tell me how things are organised in there?
It's not sorted by alphabetical order, nor by type. Options that I access often are located toward the bottom, things I will access once are at the top like theme or email accounts.

No Wifi Setup

When you start the phone, you must link it a live account, and you can't change later. To change you must reset the phone.

Oh, that reminded me on something so dumb with WP8 I couldn't believe it they would miss something like this...
You can't setup you phone with just wifi access. You are being asked to set wifi *after* you've set everything else.
Knowing that a restore from a microsoft backup is only offered during the initial setup, if you have no sim card you're stuck. You can't do restore


There's a suggestion list there:
http://windowsphone.uservoice.com/forums/101801-feature-suggestions

It's unfortunate that most of the top ones are already implemented on other phones (either iOS or Android)

If you've never used anything as a "smart" phone a windows phone, you probably won't know what you're missing and like it very much.
To me, it's a let down, because things that should be very easy and straight forward aren't.
If you're using your phone for facebook primarily, you'll be very happy. integration is brilliant.
Unfortunately, I still make phone calls and try to find things in shops.

Wednesday, February 15, 2012

Swapping your linux boot drive

My main home server's drive had been giving me SMART errors for a while.
It's a Samsung 1TB disk that I've been using for a couple of year, so it was time to change it

This server runs my MythTV backend, and various services that I've talked about here (most important being my energy monitor: http://htpc.avenard.org/power). It is powered by Linux Ubuntu 10.04 LTS

So after considering the issue, I decided to replace it with an Intel 520 60GB SSD drive.

Changing your hard drive is usually not a too complex task, you plug the new hard drive in, copy the file across with your tool of choice, re-install the GRUB boot loader, disconnect the old drive, put the new one instead. Reboot

Unfortunately here, my PC has *only* 8 SATA ports, 1 for the boot disk, and 6 used for the RAID5 array (made of 6, 2TB disks), and one use for the Bluray optical disk.

So where am I going to connect my new SSD drive. I could use a USB caddy, but that would be slow as.

I had a spare PC, so I connected the disk on the spare PC.

Using gparted I created a GPT partition format (my PC BIOS is an EFI one).
I created a partition taking the whole disk, aligned to 1MB start.
This was a mistage, as I found out later, as to install GRUB boot loader on the disk, you need space at the beginning of the disk.

So if using GPT, create a partition at least 1MB in size, that you mark in GRUB as boot_grub, and another ext4 (or ext3 your choice) partition taking the remaining space (also aligned to 1MB)

I then mounted the new ext4 partition on /ssd as follow:

sudo mkdir /ssd
sudo mount /dev/sdd1 /ssd


/dev/ssd1 is the device for the big partition I created above, you can see what device to use in gparted.

I used rsync to copy all the files from my boot disk on the server into the new disk
the copy the file across
sudo rsync --archive --stats --numeric-ids --delete-during --partial --inplace --hard-links --compress --exclude-from=exclude.default.linux --rsh="/usr/bin/ssh -i /home/jyavenard/.ssh/id_rsa" --log-file=sync.log root@htpc:/ /ssd/


/home/jyavenard/.ssh/id_rsa is the path to my RSA private key that I use to login into my server: htpc
Make sure that root login is allowed on the server you want to clone, so is defined in /etc/ssh/sshd_config and add or uncomment the line:
PermitRootLogin yes

the file exclude.default.linux contains the list of all the directories and file I do not want to copy (mainly temporary files, cache, and /data which is where the RAID array is mounted)

The content of the file is as follow:

/sys/*
/proc/*
*mozilla/firefox/*/Cache/**
/var/lib/vservers/vs1/home/*
*/.googleearth/Cache/**
*/.googleearth/Cache/temp/**
/var/spool/squid/**
/backup/*
/var/spool/cups/**
/var/log/**.gz
*/cache/apt/archives/**
/var/lib/vservers/vs1/var/tmp/**
/home/programs/tmp/**
/home/programs/vmware/**
/home/**/.thumbnails/**
/home/**/.java*/deployment/cache/**
/home/**/profile/**
/home/**/.local/Trash/**
/home/**/.macromedia/**
/export/*
**.core
/usr/ports/**
/dev/**
/usr/compat/linux/proc/**
/pool/**
/data/**
/.amd_mnt/*
/.Spotlight-V100/**
/Network/Servers/**
/export/shares/**


Note that at no time during rsync was htpc offline. It was still doing his duty in the mean time.

Once rsync had finished, I rebooted htpc in single user mode and started networking.

I re-ran the rsync on the remote PC once again, so make sure I had all the latest changes.

Once it was completed, I unmounted the SSD from the remote PC, disconnected it and then shut down the server.
Removed the boot disk, and installed the new SSD.

Now as it is, this new setup will not boot. There aren't any bootloader installed.
I had a copy of the MythBunutu LiveCD handy, so I booted htpc with it.

To install the bootloader, I needed grub2 as my disk is using GPT.
So I mounted the SSD with:
sudo mount /dev/sda1 /mnt
sudo mount --bind /dev /mnt/dev
sudo mount --bind /proc /mnt/dev

The commands above make sure the device directory and proc can be seen on the mnt mount point.
/dev/sda1 is the device for the new partition


chroot /mnt
grub-install /dev/sda
grub-mkconfig --output=/boot/grub/grub.cfg
That last line is important as your existing grub.cfg from the previous disk wouldn't work. Most Linux distribution use the disk UUID to identify a partition. That UUID is invalid on the new disk.

Before I could update, I had to make sure /etc/fstab was correct ; you should normally only have to edit the line where it mount the / root directory
For me I changed it from:
/dev/sda1 / ext3 rw,relatime,errors=remount-ro 0 0
to
/dev/disk/by-uuid/d62feed7-8003-4be7-9eae-8a24a83beb6f / ext4 rw,relatime,errors=remount-ro,barrier=1,data=ordered 0 0


To find out what the line should be, look into:
/proc/mount
and check where / is mounted.

Now we're all done and you can enjoy your new faster disk

If you had a spare SATA port on the PC you want to clone, a very similar approach could have been taken, it would only have been much faster.
You would have mounted the disk to /ssd using the same command as before; add /ssd/** to the exclude.default.linux file and run the command:

sudo rsync --archive --stats --numeric-ids --delete-during --partial --inplace --hard-links --exclude-from=exclude.default.linux / /ssd/
You could also install grub without using the LiveCD too.

Copying a disk with rsync is the fastest way of doing it, it only copies what needs to be copied, it can be interrupted and resumed at any time.

Wednesday, August 17, 2011

How to change all svn:user entry in one go

Over the years, our username has changed. Like over the years I have been avenardj, jyavenard, jeanyves_avenard and finally jean-yves.avenard

In the 7 years I've been using our SVN repository, looking at the history, it's now all over the place.

So I wanted to make my username consistent...

Preliminary steps: allow changing property in the svn repository by adding a hooks/pre-revprop-change file.

it woud contain only one thing:
exit 0

Because in our setup we use ViewVC, we also needed a hooks/post-revprop-change with:


REPOS="$1"
REV="$2"
USER="$3"
PROPNAME="$4"
ACTION="$5"

/var/www/viewvc/bin/svndbadmin update "$REPOS" "$REV"


I didn't feel like writing much code do to what seemed to be a trivial task.

Looking at various solutions I found this interesting project: XMLSH. Alas, it's quite poorly documented.

After going through the examples, I came up with:



svn log --xml | xread logs
entries=<[ $logs/log/logentry ]>
for entry in $entries; do
rev=<[$entry/@revision]>
echo "entry: " $rev
author=<[$entry/author/string()]>
echo $author
author2=$(echo $author | sed -e 's/(avenardj|jeanyves_avenard|jyavenard)$/jean-yves.avenard/g')
if [ "$author" != "$author2" ]; then
echo " -> $author2"
svn propset --revprop -r $rev svn:author $author2
fi
done


Obviously, replace the regex with the one suitable for you.

Saturday, February 6, 2010

How do solar panels perform according to solar exposure..

The Australian bureau of Meteorology provides daily solar exposure data on their web site.


This map shows the amount of energy received from the Sun in a given location.

This was very interesting stuff (well, at least for me).

I wrote a little utility to retrieve the data in a form suitable for post-processing in a spreadsheet and for use in my power tracking system .

This utility is available there. It's written in python, run it with --help as an argument to see the various options available.


One direct application, was measuring on how well the CEEG panels I have installed performed according to the amount of energy they receive from the sun.

It gave me this graph:

The black line shows the average of Export/Import. Any points above the line means the system performed better than average, any points below the line, mean they performed worse.

A clear trend shows that my system performs really well when there's a lot of sun, but rather poorly in low light conditions.

It's rather a worry, as it means they will likely perform poorly during winter.

Stay in touch!

Saturday, January 23, 2010

Energy tracking...

I got a solar system installed early December 2009 ; it's a 2kW system made of 12 CEEG 170W panels with a Power-One Aurora Inverter PVI-2000


I had to track the electricity production ; but also wanted to track my power usage.

I couldn't find any proper solutions letting me record everything at once:
  • How much is used in the house
  • How much is produced by the solar panels
  • How much is imported from the electric network grid
  • How much is exported back to the grid
So I designed my own...

I bought a CurrentCost CC128 which can be interfaced to a PC through a serial interface.
The Aurora inverter has a RS485 interface; a cheap RS232 converter on eBay for $6.50 did the trick.

The system runs on almost any machines (tested on FreeBSD, Linux, MacOS).

It is made of 3 parts.
-Daemons polling the various sensors ; they poll every 6s ; compute an average every minute and feed the data in a RRDtool database (round robin database).
-A Daemon computing the import and export value in 30 minutes interval (it wakes up every 30 minutes, calculate the import/export over the past 30 minutes and feed another database
-A web page

The daemons are written in python ; as long as you have python running on your machine with the pyserial module you'll be fine.

For the HTML interface ; it's written in PHP on the server side; and javascript/ajax on the client side (though it will work even with javascript disabled, albeit with limit interactivity).

It's probably easier to get it working with the Apache web server and the mod_php5 module...

The PC needs to be on 24/7 as neither the CC128 nor the Aurora inverter keeps any data in memory...

The system is done in such a way that you can easily use a different power monitor or a different inverter.
Everythin can be configured through a simple configuration file: colours, shapes, labels, database used etc...
You don't even need a solar system ; I set the same system at my office to monitor our power usage.

The response to this tool has been very overwhelming.
I have been approach by various parties to integrate with some professional solutions.

I contacted the Google PowerMeter project in order to feed the data through their system, but at this stage they weren't interested, prefering to work utility providers. Their loss !

Some real-time graph samples are shown below.

http://htpc.avenard.org/power.

If you are interested, drop me an email.