GHOST 4 UNIX

Now all of you wondering what is this ghost for unix. I found this software here http://www.feyrer.de/g4u/#kernel. (Donate him if you like this software :)

Now the problem could be. You are out of HDD space and lazy to do reinstalation of the OS. So what you do is buy a new hard drive, bigger one of course and copy your existing hard drive to the new one. but then its not as easy as select all and drag all the content of old hard drive to the new one using window explorer. IT WON"T WORK!!!!

solution:
1. buy a norton ghost, make an image and deploy your old drive image into the new one.
2. get this G4U at no cost (donation to the author is welcome:), copy your hardrive into the new hardrive using this tools and you will have everything transfered in to time(depend on how big is your old drive).
want more? this toools can also put your hardrive image in FTP server so anytime you ruin your partition just out the whole new OS inside your drive again. Feel like FRESH drive :)

1. What is it?

g4u ("ghost for unix") is a NetBSD-based bootfloppy/CD-ROM that allows easy cloning of PC harddisks to deploy a common setup on a number of PCs using FTP. The floppy/CD offers two functions. The first is to upload the compressed image of a local harddisk to a FTP server, the other is to restore that image via FTP, uncompress it and write it back to disk. Network configuration is fetched via DHCP. As the harddisk is processed as an image, any filesystem and operating system can be deployed using g4u. Easy cloning of local disks as well as partitions is also supported.

For the curious, I've added a few screenshots:
1. Booting g4u in bochs
2. Device detection
3. Welcome to g4u!
4. Some random g4u commands
5. Uploading a disk image with uploaddisk
6. Restoring with slurpdisk

2. Why not one of the alternatives?

o Server-part often runs (only) under DOS/Windows. I wanted to use a Unix based server.
o Supported filesystems include everything from Microsoft, but others are not handled properly (Solaris/x86, NetBSD, ...)
o I don't want to deal (ever again) with making a DOS-based bootfloppy, that gets its IP-number via DHCP.
o I've played with doing multicast-based deployment based on imm, but the result was so slow I decided not to persue it.

3. Requirements & Download

o Two empty 1.44MB floppy disks, or an empty CD-R/RW
o A FTP-server with some GB free space
o A DHCP-server

In addition to that, you may want:

o The g4u 2.1 floppy images (zipped/ uncompressed floppy one and floppy two))
o The g4u 2.1 ISO CD image (zipped/uncompressed)
o The g4u 2.1 source

Older versions of g4u are available as well:
o The g4u 1.17 floppy images (zipped/ uncompressed floppy)
o The g4u 1.17 ISO CD image (zipped/uncompressed)
o The g4u 1.17 source

You can also download from one of these mirrors:
o Australia
o Canada

4. Using it

4.1 Preparations
o Using the g4u floppy images:
1. Download the two floppy images, g4u-2.1-1.fs and g4u-2.1-2.fs or g4u-2.1.fs.zip, which contains these two files.
2. If you downloaded the g4u-2.1.fs.zip file, unpack it to get g4u-2.1-1.fs and g4u-2.1-2.fs
3. Write the two images to two seperate floppy disk. Under Unix, a simple "cat g4u-2.1-1.fs >/dev/diskette" (and same for -2.fs) will do. Make yourself familiar with the name of your floppy device, some common ones are:

+ NetBSD: /dev/fd0a
+ Solaris: /dev/diskette
+ Linux: /dev/fd0

There are also similar devices for USB sticks, but you need to grab the g4u.fs from the ISO to put there:

+ NetBSD: /dev/sd0d
+ Linux: /dev/sd0

If you're using DOS, use rawrite.exe. There's also a Windows-based program available called rawr32.zip.
o Using the g4u CDROM ISO image:
1. Download the CDROM ISO image, g4u-2.1.iso or g4u-2.1.iso.zip
2. If you downloaded the g4u-2.1.iso.zip file, unpack it to get g4u-2.1.iso
3. Please consult your CDROM writing software (Nero, DiskJuggler, WinOnCD, cdrecord, ...) 's manual on how to write the g4u.iso file to a CDROM. Note that the image is bootable.

o On a FTP server of your choice, create an user-account called "install", and protect it with some password. Make sure the 'install' user can login via ftp (/etc/shells...)

If you want to use a different account, you can specify "login@server" for slurpdisk, uploaddisk etc..
o Make sure you have a working DHCP server that hands out IP addresses and other data needed to access the FTP server from your workstation (name server, netmask, default gateway). Else you will have to set the IP-number manually..

4.2 Image creation
o Boot the CD or floppy on the machine you want to clone. See it read the kernel from disk, then print out all the devices found in the machine. It will do DHCP next, asking for an IP number - be sure you have DHCP configured properly! At the end you'll get a text description of possible commands, and a shell prompt.

o Whole harddisk:
Type "uploaddisk your.ftp.server.com filename.gz" to read out the machine's harddisk (rwd0d), and put it into the "install" account of your FTP server under the given filename. The disk image is compressed (with gzip -9), so maybe use a ".gz" file suffix. You don't have to, though. Before putting the file on the FTP server, the "install" account's password is requested.

If you want to clone your second IDE disk, add it's name on the uploaddisk command line: "uploaddisk your.ftp.server.com filename.gz wd1". Similarly, if you use SCSI instead of IDE disks, use "uploaddisk your.ftp.server.com filename.gz sd0".

If you want to use a different account name than "install", use "account@your.ftp.server.com" for both uploaddisk and slurpdisk.

o One partition only:
Get an overview of disks recognized by g4u by typing "disks", a list of partitions on a certain disk is available via "parts disk", where disk is one of the disks printed by "parts", e.g. wd0, wd1, sd0, etc. Partitions are numbered with letters starting from 'a', where partitions a-d are usually predefined, with your partitions starting at 'e'. Partitions here are BSD-partitions, which have little in common with DOS MBR partitions. To specify a partition, use something like "wd0e" or "sd0f": "uploadpart your.ftp.server.com filename.gz wd0e". Run "uploadpart" without arguments for more examples.

o Wait until you're back at the shell prompt (ignore the errors :-). Depending on your network, CPU, harddisk hardware and contents, image creation can take several hours!
o You can switch off the machine now. Type "halt" or simply press reset/power button - there are no filesystems mounted so no harm will result.
o Check that your FTP server's "install" account now has the image file.

4.3 Image deployment
o Boot the CD or floppy to the shell prompt again, see above.

o Whole harddisk:
Type "slurpdisk your.ftp.server.com filename.gz". This will log into the FTP server's "install" account, verify the password, then retrieve the image, uncompress it and write it back to /dev/rwd0d.

If you want to restore to a SCSI disk, add the disk's name to the slurpdisk command line, e.g. "slurpdisk your.ftp.server.com filename.gz sd0".

See above if you want to use an account name other than "install".
o One partition:
Use "slurppart your.ftp.server.com filename.gz wd0e" or whatever values you passed to uploadpart. Please note that the partition information is taken from your MBR, which is expected to be the same as before image creation - expect surprises if you change something between image creation and deployment. In case of inevitable changes, check the start sector and size values given by "parts". For an image that includes the MBR, do a full backup with "uploaddisk".

o Reboot the machine (type "reboot" or press reset button), and see if your machine comes up as expected - it should!

4.4 Copying a disk locally
If you just want to copy one local disk to another one with no network & server involved, the "copydisk" command is what you want. E.g. to copy the first IDE disk to the second IDE disk, use "copydisk wd0 wd1", to do the same for SCSI disks run "copydisk sd0 sd1".

Beware! All data on the target disk will be erased!

A list of disks as found during system startup can be found using the "disks" command.

4.5 Copying a partition locally
If you want to only copy one local partition to another local partition (similar to what 'uploadpart' and 'slurppart' do, just without the network and FTP in between), this can be done with the 'copypart' command. It takes two partition names as arguments, and copies the contents of one partition to the other. As an example if you found you want to copy your first local partition 'wd0e' to the second one 'wd0f', run:

copypart wd0e wd0f

A list of disks can be found using the 'disk' command, to list all the partitions on a disk use the 'parts' command. Partitions have the form of "wd0d", "w1e", "sd1f".

Be aware that the partitions to copy should have identical size (down to the sector), else funny things will happen. When copying a 'big' partition into a 'small' one, g4u won't thrash the data behind the 'small' partition, but of course the copy is not complete either. Take special note that that case could happen when you restore a copy made that way, and which went fine when you first copied your small working partition to your big backup partition!

5. FAQs and hints on disk cloning

5.1 Supported filesystems
One of the questions arising a lot is "what filesystems does g4u support". The answer is: "all of them". g4u reads the disk bit by bit, starting from byte #0 to the end. It includes any MBR, boot record, partition table and the partitions themselves without further investigating the structure of the data stored in these partitions.

5.2 Supported Operating Systems
The question on operating systems that can be deployed with g4u is the same as for the filesystems: any. Given the image-approach again, g4u is able to handle any operating system. Systems that were cloned successfully include NetBSD, Linux, Novell Netware 4.11 and 5.1, Solaris/x86, Windows NT, 2000 and XP.

By moving the harddisks to a PC, g4u can even be used to deploy operating systems for non-PC based SCSI machines running HP-UX, Irix, Solaris, AIX etc.

5.3 Supported Hardware
The system running g4u itself can have IDE, SATA, SCSI or RAID disks with various controllers (Adaptec, ...) as well as wide range of PCMCIA, Cardbus, ISA and PCI network cards. Please see the g4u kernel config for the full list of supported hardware.

If you're unsure if your hardware is supported, simply boot g4u and see if your network card gets listed by "ifconfig -a" and if your disks get listed by the "disks" command. If not, adding relevant parts of "dmesg" output (from g4u; press space bar to scroll down) is required for analysis if you ask for help. See "Reporting problems" for more information.

5.4 A word on disk sizes
The question how g4u deals with different disk sizes arises a lot too. The general answer is, g4u works best with identical disk sizes & geometry. Putting an image from a small disk on a big disk works, putting an image from a big disk to a small disk is likely to cause problems.

If you cannot avoid preparing an image on a big disk that'll get deployed to a small disk later, make sure the "extra" space is not occupied by a active partition or filesystem, else data loss is very likely to occur!

If you intend to deploy a "small" image to a "big" disk, the extra space that's not covered by g4u can be used for creating a partition and a filesystem. You will have to do that on your own, e.g. using your operating systems' post installation steps.

5.5 Changing compression level
Per default, images uploaded to the FTP server are compressed with "gzip -9". This saves as much disk space as possible, but also takes a long time - several hours are not uncommon. You can reduce the gzip level for "uploaddisk" by setting the GZIP environment variable:

# GZIP=-1 uploaddisk your.ftp.server.com filename.gz

You can change compression levels between 1 (fast, little compression) and 9 (slow, maximum compression). Of course you can specify all the usual options to uploaddisk.

5.6 List of recognized disks
During startup of g4u, all devices recognized are listed, but very fast. To get a list of recognized disks, use the 'disks' command:

# disks
wd0 at pciide0 channel 0 drive 0:
wd0: drive supports 16-sector pio transfers, lba addressing
wd0: 6149 MB, 13328 cyl, 15 head, 63 sec, 512 bytes/sect x 12594960 sectors
wd0: 32-bit data port
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 2
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2 (using DMA data transfers)

The above example shows a 6GB IDE harddisk.

5.7 Problems with images at 2GB
Do you experience g4u aborting file transfers after the image has grown to 2 GB on the FTP server? The problem here is not g4u, but most likely your FTP server. Some older Linux distributions are known to only allow files of up to 2GB filesize, and even if there is a Linux 2.4 kernel running, that's no guarantee for a properly working server. Make sure that your ftp daemon is upto date, or install a decent operating system.

So far, whatever FTP server comes with NetBSD, Solaris and Windows 2000 has been used without problems.

5.8 Can you add feature XXX?
I got requests for adding many features to g4u:

+ using TFTP
+ using SSH/scp
+ using NFS
+ adding a X or curses based GUI
+ writing images to CDROM / deployment from CDROM
+ bzip2 compression

After moving two a two-floppy set for g4u, some of these features may be added in the future, while others (X...) are not likely. Stay tuned!

5.9 Problems with network performance
If upload performance is weak (less than 5MBytes/sec on a 100BaseT Ethernet switch) even with a small compression level or a fast CPU and the harddisk is idle this means the network sucks. A common problem in switched Ethernet is a duplex mismatch between the NIC and the switch. In NetBSD, the default is to negotiate speed and duplex automatically. Other settings can be set manually.

Enforcing 100BaseTX/Full-duplex:

# ifconfig fxp0 media 100BaseTX mediaopt Full-duplex
# ifconfig -a
fxp0: flags=[...]
media: Ethernet 100baseTX full-duplex

Using autonegotiation (default):

# ifconfig fxp0 media auto
# ifconfig -a
fxp0: flags=[...]
media: Ethernet autoselect (100baseTX)

For more information, please see the ifconfig(8) manpage as well as the Auto-Negotiation Valid Configuration Table featuring "Why Can't the Speed and Duplex Be Hardcoded On Only One Link Partner?".

5.10 Reducing the image size
People complain that the image resulting from g4u is very big. This is normal as g4u clones the whole disks with all blocks, not attributing if they contain any valid data or if they are empty/unused. To find empty/unused blocks (and not clone them), g4u would need intimate understanding of the contained filesystem, which is different again for each filesystem - Windows FAT, Linux Ext2/3/ReiserFS/..., BSD FFS, Solaris UFS, etc. Given both tight space limitations on the floppy as well as shortage on filesystem documentation and implementations available, teaching g4u to ignore empty blocks is not likely to happen.

But there is an easy way to circumvent the problem: use the native operating system's understanding (and implementation) of the filesystem, and make sure it prepares empty/unused blocks in a way so they don't contain random garbage data but values which can be compressed easily by g4u, thus resulting in small image sizes.

Effectively, you just fill up the disk's unused blocks with zero-bytes. Open file for writing, stuff in 0-bytes until the disk is full, then close the file and remove it. The result is that all unused blocks were used by the file, and filled with data that g4u can then compress easily. Usually the operating system will just mark the blocks as unused, without changing the actual data content.

Using this technique on a 20GB disk that had 6GB Solaris 8/x86 and the rest Windows 2000 Workstation shrunk the image from ~6GB compressed to ~2GB compressed. You can probably imagine the effect of this on deployment time too. :)

To perform the filling of unused data blocks with zero-bytes, there are several ways, depending on what operating system you use on your computer, and what software you have available:

Standard Unix:
This works on any Unix variant - Linux, NetBSD, Solaris, etc.:

dd if=/dev/zero of=/0bits bs=20971520 # bs=20m
rm /0bits

Windows Perl solution:
This one needs perl installed. In a command shell, type:

cd /d c: c:\win-preclone.pl c:

Click here to download the win-preclone.pl perl script.

Windows Pascal solution:
This pascal program was contributed by Matthias Jordan [mjordan at code-fu dot de]:
+ nullfile-1.02.exe
+ nullfile-1.02.pas
The programs are provided here without warranty.

64bit Windows binary:
Dominic Leelodharry [dominic at authorsoftware d0t com] also sent me a binary for 64bit Windows:
+ nullfile-1.01_64bit.exe
This program is provided here without warranty.

Windows "Erasor":
This freeware program can erase your disk in a safe way, but it can also be told to just write a pattern of all-0-bits to the disk. Grab it at www.heidi.ie/eraser. Thanks to Stephen Krans [s040 at krans dot org] for the hint!

Windows "onboard" solution:
Aparently Windows XP comes with a tool to do some harddisk encryption that can also be used to write 0-bytes to the disk. To do so, run the following command: cipher /W:C: for drive C:. You will need to abort (Control+C) after the first round, else it will write random data after filling the disk nicely with 0-bytes.

Run any of these right before shutting down the operating system to create an image with g4u, and see the size difference.

5.11 Setting IP-number manually
Sometimes you may not want or be able to use DHCP. In that case doing the network configuration manually is possible with g4u, too:

1. Find out if your network device is recognized, and by what name, using the command

ifconfig -a

Your network device is something like "ex0", "tlp0", etc. (Note that unlike in Linux, NetBSD doesn't call all ethernet cards "eth0"!)

2. Next configure the network device's IP number and netmask. It is assumed that your network device is xx0 here, and that the machine should run with IP number 1.2.3.4 and netmask 255.255.255.0:

ifconfig xx0 1.2.3.4 netmask 255.255.255.0

3. Last, you may want to make the default router known unless your FTP server is in the same IP subnet as the machine you want to use g4u on. Let's say the default router's IP address is 2.3.4.5, then the command to enter is:

route add default 2.3.4.5

That's all - simple, huh? Just remember that g4u is still Unix! After these steps, you should be able to use g4u just as if it used DHCP.

5.12 Extracting the g4u kernel
I've been asked how to boot g4u from harddisk (using e.g. grub). The idea is to extract the kernel from the boot floppy, and hand that to grub (or whatever bootloader you want - maybe use PXE to netboot g4u). Here's how to extract the kernel, named "netbsd":

% ( cat g4u-2.1-1.fs | dd bs=512 skip=16 ; ? cat g4u-2.1-2.fs | dd bs=512 skip=16 ? ) | tar vxf -
-r--r--r-- 1 feyrer netbsd 53948 Nov 3 23:08 boot
-rw-rw-r-- 1 feyrer netbsd 1479905 Nov 3 23:08 netbsd

Note that the kernel ("netbsd") is actually still compressed, which is fine for the NetBSD bootloader and probably GRUB, but just in case, you may want to uncompress it:

% file netbsd
netbsd: gzip compressed data, was "netbsd-INSTALL_G4U", from Unix
% mv netbsd netbsd.gz
% gunzip netbsd.gz
% ls -la netbsd
-rw-rw-r-- 1 feyrer wheel 5523084 Dec 7 18:08 netbsd
% file netbsd
netbsd: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, stripped

5.13 Netbooting g4u via PXE

In order to netboot g4u via PXE, you first need to extract the kernel from the g4u floppies, and then follow the steps that describe a netboot (diskless boot) of NetBSD/i386, see the NetBSD documentation for various ways to do this.

A writeup of the necessary steps for g4u is available in the list archives, either in french language by Jean-Christophe Guis or in an english language translation by Steve Clement.

6. Support and reporting problems

6.1 Support
The following ways of getting support for g4u exist:

+ Mailing list: g4u-announce@feyrer.de
This list will contain announces about g4u only.

# Subscribe or change your settings via the web, or
# send a mail with "subscribe yourpassword" to g4u-announce-request@feyrer.de to subscribe. Please replace "yourpassword" with your personal, secret list password.
# Look at the g4u-announce list archive.

+ Mailing list: g4u-help@feyrer.de
This list is intended for questions and answers about g4u.

# Subscribe or change your settings via the web, or
# send a mail with "subscribe yourpassword" to g4u-help-request@feyrer.de to subscribe. Please replace "yourpassword" with your personal, secret list password.
# Look at the g4u-help list archive.
+ Orkut "g4u" community:

There's a "g4u" community at Orkut which I've created as a forum for g4u. As Orkut is too slow these days (Dec 2004), I won't follow discussion on Orkut a lot, and recommend using the above-mentioned mailing lists for getting timely response.

+ Optional(!) commercial license are available, see the "Commercial license" page for more information.

6.2 Reporting problems
If you have trouble with g4u and want to report a problem, PLEASE add the following information. It's impossible to help you without knowing details on your systems & setup:

+ What g4u version do you use
+ What exact command(s) did you use
+ What exact output did you get (all of it, 1:1, no interpretation on your own)?
+ If the system hangs during boot: what are the last three lines printed on the screen (again all three of them, verbatime)
+ What does "disks" say? "parts"?
+ If you have some problems with some network card or disk driver, please include the relevant parts from "dmesg" output (scroll down with 'space')
+ What operating system, filesystem type and FTP server software do you use on your FTP server?

7. Rebuilding from source

G4u is based on NetBSD boot floppy disk set. It consists of a custom kernel and custom bootfloppy, both stripped down to an absolute minimum to fit on two 1.44MB floppy disk or a CDROM. There are also the "copydisk", "uploaddisk" and "slurpdisk" scripts.

To rebuild the images:
o Note that for rebuilding, no root permissions are needed any more! Below, the "#" prompt shows action needed as root, "%" is the prompt for commands executed as user.

o Get a i386/PC machine running NetBSD 2.0. Cross-building from other Unix-like operating systems may work too, but is untested as far as g4u is concerned.

o Install mkisofs (part of the cdrecord package), e.g. either as precompiled binary:

# PKG_PATH=ftp://ftp.NetBSD.org/pub/NetBSD/packages/2.0/i386/All
# export PKG_PATH
# pkg_add -v cdrecord

or, if you have pkgsrc installed:

# cd /usr/pkgsrc/sysutils/cdrecord
# make install

o Get NetBSD-current source from ~May 17th 2005 as /usr/src:

% su
# mkdir /usr/cvs
# chown $USER /usr/cvs
# ln -s cvs/src /usr/src
# exit
% cd /usr/cvs
% env CVS_RSH=ssh cvs -d anoncvs@anoncvs.netbsd.org:/cvsroot co -D 20050517 src

o To be on the safe side, rebuild a full release (regression testing and more):

% ./build.sh -N 1 -U release

o Remember where to find the new compiler - adjust if necessary:

% setenv T /usr/src/obj.i386/tooldir.NetBSD-2.0-i386/bin

o Get the g4u 2.1 sources

o Unpack the g4u sources:

% cd /usr/src
% tar plzvxf .../g4u-2.1.tgz

o Apply patches:

% cd /usr/src/sys/dev/ata
% patch < wd.c.patch-g4u
% cd /usr/src/sys/dev/scsipi
% patch < sd.c.patch-g4u
% cd /usr/src/sys/kern
% patch < subr_prf.c.patch-g4u


o Build the G4U floppy & ISO-image:

% cd /usr/src/distrib/i386/floppies/g4u
% $T/nbmake-i386

o After that, you'll find "g4u1.fs", "g4u1.fs" and "g4u.iso" in /usr/src/distrib/i386/floppies/g4u:

% pwd
/usr/cvs/src/distrib/i386/floppies/g4u
% ls -l g4u.*
-rw-rw-r-- 1 feyrer netbsd 3309568 Nov 3 23:09 g4u.iso
-rw-rw-r-- 1 feyrer netbsd 1474560 Nov 3 23:08 g4u1.fs
-rw-rw-r-- 1 feyrer netbsd 1474560 Nov 3 23:08 g4u2.fs