You are currently browsing the Gentoo Linux category
Displaying 1 - 6 of 6 entries.

ZFS: Deduplicating is not a myth!

  • Posted on February 9, 2013 at 20:44

Long time no see!

After having it put away for, I guess almost 2 years, I took a look at ZFS again.
In ‘the early days’ ZFS only had a Linux-implementation using FUSE. An implementation which I liked from a Nerdish point-of-view, but not so much as a serious replacement for XFS on my operational Linux machines.

Since SSD is commonly available now, and all my operational servers have at least 16 cores, it was time to reevaluate the possibilities of ZFS on Linux again.

I was not disappointed! My oh my…

 

The ZFSonLinux Gentoo 64bit Walk-through:
(Using VMware Fusion 5)

  • Create a Gentoo Linux 64bit VM, add 4GB RAM at least, and the disks:
  • 1 Boot disk, 80GB
  • 4 Data disks, each 2TB. Single file, Do not Preallocate diskspace! Important!
  • 1 Cache disk, 20GB. Preallocating is advised. Not necessary.
  • Install Gentoo
  • Install sys-fs/zfs sys-fs/zfs-kmod
  • add ‘modules_3_6=”zfs”‘ to /etc/conf.d/modules
  • insmod /lib64/modules/3.6.11-gentoo/addon/zfs/zfs/zfs.ko
  • zpool create deduptestvol raidz -f /dev/sdf /dev/sdg /dev/sdh /dev/sdi
  • zpool add -f deduptestvol cache /dev/sdj
  • zfs set atime=off deduptestvol
  • zfs set dedup=on deduptestvol

The ZFS-volume, named deduptestvol, should be up and running right now. Typically it is mounted automatically under /.
Let’s check:

 

Testing the dedup-capabilities

  • I made one volume, with the same size, also RAIDZ
  • Created one file of exactly 1000MegaBytes
    (# dd if=/dev/random of=/data bs=100M count=10)
  • Copied that file over for 32 times
  • Then I copied that whole directory to the volume with deduplicating switched on.
  • On my host machine, I took a look a the disk space consuming.

Some proofs

Some recursive MD5’s over both volumes:

 

Conclusion

For saving 32Gigabyte of data in traditional RAID5:

The ‘normal’ ZFS-Volume consumed 44G of virtual disk-space.
The Deduplicated ZFS-Volume consumed 3.6G of virtual disk-space.

“ZFS is the shit!”

Knowing your dynamic IP-address

  • Posted on July 25, 2011 at 18:53

The troubling situation
Being a nerd, there is nothing more frustrating than having a dynamic IP-address.
Being a nerd and doing some hosting locally actually makes it even wors!

There’s not much use to running services like Apache or Sendmail with an IP-address that changes at random times.
Random in this particular case means: “When we of Ziggo want it to!”.

Alert by email
So what you basically want is to be alerted whenever you IP has changed.
I wrote some perl-code that does exactly that! And as a bonus, it does some handy logging.
The script can be initiated from command-line, but the best way is making a cronjob of course.

Realworld example
I created two new e-mailadresses; newipkrusjme@gmail.com and newipkrusjme@hotmail.com. Both addresses are now configured on my iPhone to receive all mail automatically,
In the cron, I set it to run as an hourly-job,
And then I did “the Dutch approach“…

Sourcecode
Thanks to Vim it was quite easy to make the sourcecode readable using :TOhtml.
If you love to see the amazing Vim-output or you’re actually seriously interested: You can find the sourcecode here: http://www.krusj.nl/files/newip-2.0.pl.html
Download directly: http://www.krusj.nl/files/newip-2.0.pl

Installing

  • download the script, (for example to /usr/local/bin)
  • change the email-addresses to your preferred ones,
  • change the interface-card, (eth1, eth0 or whatever)
  • mkdir /var/log/newip /var/newip,
  • chmod 755 /usr/local/bin/newip-2.0.pl,

Testing / Forcing
If you want to test it, just run the script from command-line using:
# /usr/local/bin/newip-2.0 --force-mail

Processes and ports

  • Posted on July 19, 2011 at 08:04

Ever wondered which process is behind which port? Well, I do pretty often!
And also pretty often I have to look it up again…
Let’s change that for once and for all!

Here’s a really obvious example with port number 80 in use by Apache. (Such a surprise!)

Netstat: The syntax
# netstat -a -n -p | egrep 'Proto|LISTEN' | grep 80
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1840/apache20

 

Forcing open doors
This output shows that Apache2.0 runs with PID 1840 and it’s listening to port 80.

After this I did a ‘ps aux | grep 1840’ and it turned out to be true! 🙂

 

 

/etc/fstab and bind-mount

  • Posted on January 17, 2011 at 03:22

I use a Software RAID0 for data
And on this software-raid-volume, there are my MySQL-Databases, /home and other stuff that should be fast…
And, like all volumes, this software-raid-volume is mounted during boot-time.
But since my /home is also on that volume, it should be mounted using a bind-mount.
So first there is a mount using a mdadm: /dev/md127 -> /mnt/data,
And second there is a mount using bind-mount /mnt/data/home -> /home

It took me quite a while to figure out how to get this done at boot-time, but I managed!
Here’s what my /etc/fstab looks like:

# /etc/fstab: static file system information.
#
# noatime turns off atimes for increased performance (atimes normally aren't
# needed; notail increases performance of ReiserFS (at the expense of storage
# efficiency).  It's safe to drop the noatime options if you want and to
# switch between notail / tail freely.
#
# The root filesystem should have a pass number of either 0 or 1.
# All other filesystems should have a pass number of 0 or greater than 1.
#
# See the manpage fstab(5) for more information.


# <fs>            <mountpoint>    <type>        <opts>        <dump/pass>


# NOTE: If your BOOT partition is ReiserFS, add the notail option to opts.
LABEL=boot        /boot        ext2        noauto        1 2
#/dev/sda3        /boot        ext2        noauto        1 2


LABEL=root        /        ext3        defaults    0 1
#/dev/sda4        /        ext3        noatime        0 1

# DATA
LABEL=data        /mnt/data    ext3        defaults    0 1

# HOME (bindmount)
/mnt/data/home         /home        bind        defaults,bind    0 0


# SWAP like RAID0. The higher the value the higher the priority. (max=32767)
/dev/sda5        none        swap        sw,pri=1    0 0
/dev/sdb5        none        swap        sw.pri=1    0 0

#/dev/cdrom        /mnt/cdrom    auto        noauto,ro    0 0
#/dev/fd0        /mnt/floppy    auto        noauto        0 0

# glibc 2.2 and above expects tmpfs to be mounted at /dev/shm for
# POSIX shared memory (shm_open, shm_unlink).
# (tmpfs is a dynamically expandable/shrinkable ramdisk, and will
#  use almost no memory if not populated with files)
shm            /dev/shm    tmpfs        nodev,nosuid,noexec    0 0
proc            /proc        proc        defaults        0 0

For the attentive readers:
Yes, I did put my boot on a separate partition. And getting this done on a PowerMac G5 is quite an exhibition! Maybe someday, someday I’m gonna blog about that too!

SABnzbd and Gentoo

  • Posted on January 11, 2011 at 01:15

Apparently the portage-tree doesn’t contain SABnzbd. Too bad! I really needed some binary newsgroup downloader. And I did have some experience with running it on my local machine, a MacPro. Wonderful App! Or tool, or server, or whatever…

But there I was, left alone, helpless and disillusioned…
What to do now?

Well, basically…… This!
(keep in mind, I’m using a PPC64)

Installing
Edit /etc/portage/package.keywords, and add:
# SABnzbd
dev-python/* **
app-arch/par2cmdline *

And now run (not specifically in this order):

# emerge dev-python/pysqlite
# emerge dev-python/cheetah
# emerge dev-python/pyopenssl
# emerge sqlite
# emerge par2cmdline
# emerge unrar

Download the SABnzbd-sources:

# wget http://sourceforge.net/projects/sabnzbdplus/files/sabnzbdplus/sabnzbd-0.5.6/SABnzbd-0.5.6-src.tar.gz

Unzip/Unpack SABnzbd.zip to an appropriate directory,
cd into newly made directory,
Edit /.sabnzbd/sabnzbd.ini, and change the host-value ‘localhost’ to 0.0.0.0,
Now run SABnzbd.py and you can configure the rest of SABnzdb using your browser, (Ahhhhh….)

# ./SABnzbd.py

Security
Although this post isn’t about security, be aware that in this particular example SABnzbd is now running as root. And that ain’t necessary at all!

It is possible, and advised by me ;-), to create a sabnzbd-user, and let it run under that newly created sabnzbd-account.
Naturally, you have to do some chowning to the SABnzbd-directory. (Also to the .-directories which are ignored by default by chown!)

Fun stuff
Further more a tip for extra fun and carefree usage:
Create a SABnznbd-writeable directory /var/log/sabnzdb,
In the sabnzbd.ini in the ./sabnzbd-dir: Change the logdir (logs) to /var/log/sabnzbd

You are maybe asking yourself: “What’s the fun of that?”
Well, just take a look at xtail.pl, and you’ll find out!

Unmasking autounmask

  • Posted on January 9, 2011 at 18:36

I found out that a lot of Gentoo-stuff has not yet been tested on the PPC64 platform, but appears to work just fine. Take WordPress for example!

Unmasking is basically, or usually, the only thing that needs to be done. But immediately, there is the first barrier: How to get autounmask to work?
Well, this is what worked for me:

Installing
Edit /etc/portage/package.keywords, and add:
# autounmask
app-portage/autounmask *
dev-perl/PortageXS *
virtual/perl-Term-ANSIColor *
perl-core/Term-ANSIColor *

And now run “emerge autounmask” (again).
Et voila! Autounmask will emerge having no trouble at all!