Steve Pate : ext2 & ext3

These inadequacies were addressed by both the Xia filesystem and the ext2 filesystem (which was modelled on the BSD Fast File System), both of which provided a number of enhancements, including a better on-disk layout for managing filesystem resources. The improvements resulting in ext2 far outweighed those of Xia, and in ext2 became the defacto standard on Linux.

The following sections first describe the ext2 filesystem, followed by a description of how the filesystem has evolved over time to produce the ext3 filesystem which supports journaling and therefore fast recovery.

Features of the ext2 Filesystem

4TB filesystems. This required changes within the VFS layer. Note that the maximum file and filesystem size are properties of the underlying filesystem and the kernel implementation.

255-byte filenames. Directory entries are variable in length with a maximum size of 255 bytes.

Selectable file semantics. With a mount option, the administrator can choose whether to have BSD or SVR4 file semantics. This has an effect on the group ID chosen when a file is created. With BSD semantics, files are created with the same group ID as the parent directory. For System V semantics, if a directory has the set group ID bit set, new files inherit the group ID bit of the parent directory and subdirectories inherit the group ID and set group ID bit; otherwise, files and directories inherit the primary group ID of the calling process.

Multiple filesystem block sizes. Block sizes of 1024, 2048, and 4096 bytes can be specified as an option to mkfs.

Reserved space. Up to 5 percent of the filesystem can be reserved for root-only files, allowing some recovery in the case of a full filesystem.

Per-file attributes. Attributes can be set on a file or directory to affect subsequent file access. This is described in detail in the next section.

BSD-like synchronous updates. A mount option ensures that all meta-data (inodes, bitmaps, indirects and directories) are written to disk synchronously when modified. This increases filesystem integrity although at the expense of performance.

Periodic filesystem checks. To enforce filesystem integrity, ext2 has two ways of ensuring that a full fsck is invoked on the filesystem. A count is kept of how many times the filesystem is mounted read/write. When it reaches a specified count, a full fsck is invoked. Alternatively, a time-based system can be used to ensure that the filesystem is cleaned on a regular basis.

Fast symbolic links. As with VxFS, symbolic links are stored in the inode itself rather than in a separate allocated block.

The following sections describe some of these features in more detail.

Per-File Attributes

EXT2_SECRM_FL. With this attribute set, whenever a file is truncated the data blocks are first overwritten with random data. This ensures that once a file is deleted, it is not possible for the file data to resurface at a later stage in another file.

EXT2_UNRM_FL. This attribute is used to allow a file to be undeleted.

EXT2_SYNC_FL. With this attribute, file meta-data, including indirect blocks, is always written synchronously to disk following an update. Note, though, that this does not apply to regular file data.

EXT2_COMPR_FL. The file is compressed. All subsequent access must use compression and decompression.

EXT2_APPEND_FL. With this attribute set, a file can only be opened in append mode (O_APPEND) for writing. The file cannot be deleted by anyone.

EXT2_IMMUTABLE_FL. If this attribute is set, the file can only be read and cannot deleted by anyone.

Attributes can be set on both regular files and directories. Attributes that are set on directories are inherited by files created within the directory.

The following example shows how the immutable attribute can be set on a file. The passwd file is first copied into the current directory and is shown to be writable by root. The chattr command is called to set the attribute, which can then displayed by calling lsattr. The two operations following show that it is then no longer possible to remove the file or extend it:

# cp /etc/passwd .
 # ls -l passwd
 -rw-r--r-- 1 root root 960 Jan 28 17:35 passwd
 # chattr +i passwd
 # lsattr passwd
 ---i--------passwd
 # rm passwd
 rm: cannot unlink 'passwd': Operation not permitted
 # cat >> passwd
 bash: passwd: Permission denied

The ext2 Disk Layout

struct ext2_super_block {
 unsigned long s_inodes_count; /* Inodes count (in use)*/
 Disk-Based Filesystem Case Studies 227
 unsigned long s_blocks_count; /* Blocks count (in use) */
 unsigned long s_r_blocks_count; /* Reserved blocks count */
 unsigned long s_free_blocks_count; /* Free blocks count */
 unsigned long s_free_inodes_count; /* Free inodes count */
 unsigned long s_first_data_block; /* First Data Block */
 unsigned long s_log_block_size; /* Block size */
 long s_log_frag_size; /* Fragment size */
 unsigned long s_blocks_per_group; /* # Blocks per group */
 unsigned long s_frags_per_group; /* # Fragments per group */
 unsigned long s_inodes_per_group; /* # Inodes per group */
 unsigned long s_mtime; /* Mount time */
 unsigned long s_wtime; /* Write time */
 unsigned short s_mnt_count; /* Mount count */
 short s_max_mnt_count; /* Maximal mount count */
 unsigned short s_magic; /* Magic signature */
 unsigned short s_state; /* File system state */
 unsigned short s_errors; /* Error handling */
 unsigned long s_lastcheck; /* time of last check */
 unsigned long s_checkinterval; /* max. time between checks */
 };

When writing sequentially to a file, ext2 tries to preallocate space in units of 8 contiguous blocks. Unused preallocation is released when the file is closed, so no space is wasted. This is used to help prevent fragmentation, a situation under which the majority of the blocks in the file are spread throughout the disk because contiguous blocks may be unavailable. Contiguous blocks are also good for performance because when files are accessed sequentially there is minimal disk head movement.

It is said that ext2 does not need defragmentation under normal load as long as there is 5 percent of free space on a disk. However, over time continuous addition and removal of files of various size will undoubtedly result in fragmentation to some degree. There is a defragmentation tool for ext2 called defrag but users are cautioned about its use.if a power outage occurs when running defrag, the file system can be damaged.

The block group is described by the following structure:

struct ext2_group_desc {
 unsigned long bg_block_bitmap; /* Blocks bitmap block */
 unsigned long bg_inode_bitmap; /* Inodes bitmap block */
 unsigned long bg_inode_table; /* Inodes table block */
 unsigned short bg_free_blocks_count; /* Free blocks count */
 unsigned short bg_free_inodes_count; /* Free inodes count */
 unsigned short bg_used_dirs_count; /* Directories count */
 };

To better see how the block group structures are used in practice, the following example, using a small ext2 filesystem, shows how structures are set up when a file is allocated. Firstly, a filesystem is made on a floppy disk as follows:

# mkfs /dev/fd0
 mke2fs 1.24a (02-Sep-2001)
 Filesystem label=
 OS type: Linux
 Block size=1024 (log=0)
 Fragment size=1024 (log=0)
 184 inodes, 1440 blocks
 72 blocks (5.00%) reserved for the super user
 First data block=1
 1 block group
 8192 blocks per group, 8192 fragments per group
 184 inodes per group
 Writing inode tables: 0/1done
 Writing superblocks and filesystem accounting information: done
 This filesystem will be automatically checked every 35 mounts or
 180 days, whichever comes first. Use tune2fs -c or -i to override.

# debugfs /dev/fd0
 debugfs 1.24a (02-Sep-2001)
 debugfs: show_super_stats -h
 Filesystem volume name: 
 Last mounted on: 
 Filesystem UUID: e4e5f20a-f5f3-4499-8fe0-183d9f87a5ba
 Filesystem magic number: 0xEF53
 Filesystem revision #: 1 (dynamic)
 Filesystem features: filetype sparse_super
 Filesystem state: clean
 Errors behavior: Continue
 Filesystem OS type: Linux
 Inode count: 184
 Block count: 1440
 Reserved block count: 72
 Free blocks: 1399
 Free inodes: 173
 First block: 1
 Block size: 1024
 Fragment size: 1024
 Blocks per group: 8192
 Fragments per group: 8192
 Inodes per group: 184
 Inode blocks per group: 23
 Last mount time: Wed Dec 31 16:00:00 1969
 Last write time: Fri Feb 8 16:11:59 2002
 Mount count: 0
 Maximum mount count: 35
 Last checked: Fri Feb 8 16:11:58 2002
 Check interval: 15552000 (6 months)
 Next check after: Wed Aug 7 17:11:58 2002
 Reserved blocks uid: 0 (user root)
 Reserved blocks gid: 0 (group root)
 First inode: 11
 Inode size: 128
 Group 0: block bitmap at 3, inode bitmap at 4, inode table at 5
 1399 free blocks, 173 free inodes, 2 used directories

Further information about the block group can be displayed with the dumpe2fs command as follows:

# dumpe2fs /dev/fd0
 dumpe2fs 1.24a (02-Sep-2001)
 ...
 Group 0: (Blocks 1 -1439)
 Primary Superblock at 1, Group Descriptors at 2-2
 Block bitmap at 3 (+2), Inode bitmap at 4 (+3)
 Inode table at 5-27 (+4)
 1399 free blocks, 173 free inodes, 2 directories
 Free blocks: 41-1439
 Free inodes: 12-184

# cd /mnt # i=12 # while [ $i -lt 188 ] ; do ; > $i ; i=_eexpr $i + 1_e ; done bash: 185: No space left on device bash: 186: No space left on device bash: 187: No space left on device # df -k Filesystem 1k-blocks Used Available Use% Mounted on /dev/hda3 19111092 1844084 17267008 10% / /dev/hda1 21929 3615 17182 18% /boot shmfs 127780 0 127780 0% /dev/shm /dev/fd0 1412 15 1325 2% /mnt So, although the filesystem is only 2 percent full, all of the inodes have been allocated. This represents one of the difficulties that filesystems have faced over the years where the number of inodes are statically allocated when the filesystem is made.

The following example shows the statistics of an allocated file:

# cp /etc/passwd /mnt ; umount /mnt
 # debugfs /dev/fd0
 debugfs 1.24a (02-Sep-2001)
 debugfs: ls -l /
 2 40755 0 0 1024 13-Feb-2002 20:20 .
 2 40755 0 0 1024 13-Feb-2002 20:20 ..
 11 40755 0 0 12288 13-Feb-2002 20:18 lost+found
 12 100644 0 0 2064 13-Feb-2002 20:20 passwd
 debugfs: stat <12>
 Inode: 12 Type: regular Mode: 0644 Flags: 0x0 Generation: 59537
 User: 0 Group: 0 Size: 2064
 File ACL: 0 Directory ACL: 0
 Links: 1 Blockcount: 6
 Fragment: Address: 0 Number: 0 Size: 0
 ctime: 0x3c6b3af9 -Wed Feb 13 20:20:09 2002
 atime: 0x3c6b3af8 -Wed Feb 13 20:20:08 2002
 mtime: 0x3c6b3af8 -Wed Feb 13 20:20:08 2002
 BLOCKS:
 (0-2):41-43
 TOTAL: 3

In this case, the file is displayed by inode number. The size of the file is 2064 bytes which results in three blocks being allocated: blocks 41 to 43. Recall from displaying the block group information shown previously that the first data block started at block 41.

ext2 On-Disk Inodes

struct ext2_inode {
 __u16 i_mode; /* File mode */
 __u16 i_uid; /* Low 16 bits of Owner Uid */
 __u32 i_size; /* Size in bytes */
 __u32 i_atime; /* Access time */
 __u32 i_ctime; /* Creation time */
 __u32 i_mtime; /* Modification time */
 __u32 i_dtime; /* Deletion Time */
 __u16 i_gid; /* Low 16 bits of Group Id */
 __u16 i_links_count; /* Links count */
 __u32 i_blocks; /* Blocks count */
 __u32 i_flags; /* File flags */
 __u32 i_block[EXT2_N_BLOCKS];/* Pointers to blocks */
 __u32 i_generation; /* File version (for NFS) */
 __u32 i_file_acl; /* File ACL */
 __u32 i_dir_acl; /* Directory ACL */
 __u32 i_faddr; /* Fragment address */
 struct {
 __u8 l_i_frag; /* Fragment number */
 __u8 l_i_fsize; /* Fragment size */
 } linux2;
 };

The inode i_block[] array has EXT2_N_BLOCKS (15) pointers to blocks of data. The first EXT2_NDIR_BLOCKS (12) entries in the array are direct pointers to data blocks. The i_block[12] element points to an indirect block of pointers to data blocks. The i_block[13] element points to a double indirect block for which each element points to an indirect block. The i_block[14] element points to a triple indirect block of pointers to double indirects.

Various inode numbers are reserved which explains why the first inode allocated has an inode number of 12 (lost+found is 11). Some reserved inodes are:

 EXT2_BAD_INO (1). This file contains a list of bad blocks on the file system.
 EXT2_ROOT_INO (2). This is the root directory of the file system.
 EXT2_ACL_IDX_INO (3). ACL inode.
 EXT2_ACL_DATA_INO (4). ACL inode.
 EXT2_BOOT_LOADER_INO (5). The file contains the boot loader.
 EXT2_UNDEL_DIR_INO (6). This file is used for file undelete.
 EXT2_FIRST_INO (11). This is the first inode that does not have a special
 meaning and can be used for other purposes.
 

The i_faddr field is used in the case where the fragment size and block size are not equal. If the file does not require an exact number of filesystem-sized blocks, the last portion of the file data is stored in a fragment. The location of the fragment is stored in this field.

Repairing Damaged ext2 Filesystems

Pass 1. This phase takes the longest time to execute, because all of the inodes have to be read into memory and checked. In this phase, e2fsck checks each inode in the filesystem to ensure the file mode is valid and that all of the blocks in the inode are valid block numbers. During pass 1, bitmaps indicating which blocks and inodes are in use are compiled, to be used later.

If e2fsck notices data blocks that are mapped by more than one inode, it can either clone the duplicated blocks so that each inode has its own copy, or remove the blocks from one or more of the inodes.

To reduce the I/O time necessary in future passes, critical filesystem information is cached in memory, including the location on disk of all of the directory blocks on the filesystem. This removes the need to re-read the directory inodes during pass 2.

Pass 2. In this phase directories are validated. Because directory entries do not span disk blocks, each directory block can be checked individually without reference to other directory blocks. The directory blocks are checked to make sure that the directory entries are valid and contain references to inode numbers that are in use (as determined by pass 1).

For the first directory block in each directory inode, the _g._h and _h.._h entries are checked to make sure they exist, and that the inode number for the _g._h entry matches the current directory. Pass 2 also caches information concerning the parent directory in which each directory is linked. If a directory is referenced by more than one directory, the second reference of the directory is treated as an illegal hard link and is removed.

Note that at the end of pass 2, nearly all disk I/O that e2fsck needs to perform is complete. Information required by passes 3, 4, and 5 are cached in memory; hence, the remaining passes of e2fsck are largely CPU bound and take less than 5 to 10 percent of the total running time.

Pass 3. In this phase, the directory connectivity is checked by tracing the path of each directory back to the root using information that was cached during pass 2. At this time, the _g.._h entry for each directory is also checked to make sure it is valid. Any directories that can not be traced back to the root are linked to the lost+found directory.

Pass 4. In this phase, e2fsck checks the reference counts for all inodes by iterating over all the inodes and comparing the link counts (which were cached in pass 1) against internal counters calculated during passes 2 and 3. Any undeleted files with a zero link count are placed in lost+found during this pass.

Pass 5. In this last phase e2fsck checks the validity of the filesystem summary information. It compares the block and inode bitmaps which were constructed during the previous passes against the actual bitmaps on the filesystem and corrects the on-disk copies if necessary.

The e2fsck program is designed to run as quickly as possible. Because filesystem checking programs tend to be disk-bound, this was done by optimizing the algorithms used by e2fsck so that filesystem structures are not repeatedly accessed from the disk. In addition, the order in which inodes and directories are checked are sorted by block number, to reduce the amount of time in disk seeks.

Tuning a ext2 Filesystem

-c max-mount-counts. This option adjusts the count of read/write mounts between two filesystem checks.

-e error-behavior. When errors are detected, the behavior of the ext2 kernel code can be altered with this option. The value of error-behavior can be continue in that the kernel continues with normal execution, remount-ro, which forces the kernel to remount the filesystem read-only, or panic in which case the kernel will panic.

-u user. This option sets the user who can benefit from the reserved blocks when the filesystem becomes full. The value of user can be a numerical user ID or a user name.

For further information on tune2fs see the tune2fs(8) manual page.

Resizing ext2 Filesystems

If an ext2 filesystem resides on an LVM (Logical Volume Manager) volume, the e2fsadm command can be used to resize both the filesystem and the underlying logical volume.

The ext3 Filesystem

As described in the section VxFS Journaling, earlier in this chapter, these times can be significant, measured in many hours, if the filesystem is very large in size. Note that large in this case is actually a property of the amount of structural data (inodes) and not specifically the size of the filesystem.

Another goal behind ext3 was to make as few changes to the underlying ext2 code base as possible because ext2 is small in size, easy to maintain, robust, and well understood.

The use of ext3 was positioned in such a way that it is easy to transition between ext2 and ext3 filesystems and vice versa.

The actual journaling layer is separate from ext3. The filesystem understands the concepts of transaction (when one starts, when it finishes) but it is not actually responsible for the journaling.

How to Use an ext3 Filesystem

# mkfs -j /dev/sda5

To convert an existing ext2 filesystem to an ext3 filesystem, the tune2fs command can be invoked as follows:

# tune2fs -j /dev/sda5

Note that the command can be invoked on either a mounted or unmounted filesystem. If invoked on a mounted filesystem, the journal will appear as a visible file (.journal). If invoked on an unmounted filesystem or if mkfs -j is run when making the filesystem, the journal will not be visible.

To actually mount the filesystem, the ext3 filesystem type must be specified:

# mount -t ext3 /dev/sda5 /mnt1

Conversion back to ext2 can be achieved by using the tune2fs command as follows:

# tune2fs -O ^has_journal /dev/sda5

or simply by replaying the log to make the filesystem clean and then simply mounting it as an ext2 filesystem.

Data Integrity Models in ext3

data=writeback. This option limits data integrity guarantees so that file data itself is not journaled. The filesystem, is however, guaranteed to be structurally sound at all times.

data=ordered. This mode, which is the default, ensures that data is consistent at all times. The data is actually written to the file before the transaction is logged. This ensures that there is no stale data in any filesystem block after a crash.

data=journal. This option writes all file data through the journal. This means that the data is actually written to disk twice. This option provides the best guarantees in terms of filesystem integrity but because data is written through the journal, performance can be significantly impacted and the time for recovery after a crash can be much greater.

How Does ext3 Work?

Journal blocks. An update to an inode, for example, will write the entire filesystem block to which the inode belongs in to the journal. In [TWEE98], Stephen Tweedie claims that this is a relatively cheap method due to the sequential nature in which data is written to the journal, and that by following this simple approach, there is little complexity in the kernel and therefore less CPU overhead.

Descriptor blocks. These blocks describe other journal blocks and are written to the journal before the journal blocks are written. Because the journal blocks are the actual meta-data blocks that must be written, the descriptor blocks are used to record information about the journal blocks, such as the disk block on which they reside.

Header blocks. The header blocks are written throughout the journal. They record the start and end of the journal together with a sequence number that is used during recovery to locate the order in which the blocks were written.

As with VxFS, transactions are delayed in memory to aid performance. With ext3, a set of transactions is batched into a compound transaction and committed to the journal on disk. This process is called checkpointing. While checkpointing is in progress, a new compound transaction is started, that will record any further changes to the filesystem while the previous compound transaction is being written to disk.

Crash recovery is performed by walking through the journal and writing any journal blocks to their correct location on disk. Because this is an idempotent operation, a crash in the middle of recovery does not matter because the process can be repeated any number of times with exactly the same effect.

Summary

There are many different UNIX filesystems and to scratch the surface on all of them would easily fill a book of this size. The three filesystems described in the chapter represent a good cross section of filesystems from the UNIX and Linux operating systems and cover the commercial filesystem market (VxFS), the most widely documented and ported filesystem (UFS), and the most popular open source filesystems (ext2 and ext3). Only a few other filesystems have been documented in any detail. [HANC01] describes the AdvFS filesystem developed by Digital which is the main filesystem of their True64 operating system. [KELL96] describes IBM_fs JFS filesystem.

To understand filesystem internals it is always best to start with one of the simple filesystems such as the original System V filesystem as documented in [LION96]. If studying Linux, the ext2 filesystem on one of the earlier kernels is a good place to start before looking at the more elaborate, and therefore more complex, filesystems.

Pseudo Filesystems

This chapter describes some of the more well known pseudo filesystem types and provides a basic implementation of the ps command using the Solaris /proc filesystem.

The /proc Filesystem

The /proc filesystem was to replace the ptrace() system call, with the advantage that the full process address space was visible and could be manipulated with read() and write() system calls. This contrasts with the interfaces offered by ptrace(), the system call traditionally used by debuggers, that only provides a word-at-a-time interface.

Roger Faulkner and Ron Gomes ported the research version of /proc to SVR4 and presented their work in another USENIX paper: _gThe Process File System and Process Model in UNIX System V_h [FAUL91]. At that time, Faulkner was with Sun Microsystems and Gomes with AT&T Bell Laboratories. As described in the paper, future work was intended to restructure /proc from a flat file system into a directory hierarchy describing a process. That work was undertaken at both Sun and USL and will be described later.

In the early /proc implementation, whose name is derived from the directory on which it is mounted, there is an entry in the directory for each process in the system. The name of the file displayed corresponds to the process ID, while the size of the file represents the size of the process address space. The file permissions correspond to the user who owns the process.

Figure 11.1 shows at a high level how the /proc filesystem is implemented. Standard file-related system calls such as open(), read(), and write() are handled at the filesystem-independent layer in the same manner as for other filesystem types. Much of the information about a process is held in the process table (traditionally in the array proc[]). To open a specific process file, the /proc filesystem must scan the process table looking for an entry whose p_pid field matches the pathname component passed.

One of the most widely used commands that access /proc is ps. Its role is to open each file in the /proc directory and then access the process status through an ioctl() system call. This was originally represented by the prstatus structure, which could be obtained by opening the file and issuing the PIOCSTATUS ioctl command. With the SVR4 implementation of /proc, there were over 40 different ioctl commands that could be issued, many of which dealt with debugging.

Note that the /proc filesystem does not have to be mounted on the /proc directory. It can in fact be mounted multiple times, which allows it to be used in chroot() environments.

The Solaris /proc Implementation

As an example, consider the following process, which is run in the background. Using the process ID that is returned, the contents of the /proc/3707 are displayed:

$ sleep 10000&
 [1] 3707
 $ cd /proc/3707
 $ ls -l
 total 1618
 -rw------- 1 spate fcf 1630208 May 28 21:24 as
 -r-------- 1 spate fcf 152 May 28 21:24 auxv
 Pseudo Filesystems 251
 -r-------- 1 spate fcf 36 May 28 21:24 cred
 --w------- 1 spate fcf 0 May 28 21:24 ctl
 lr-x------ 1 spate fcf 0 May 28 21:24 cwd ->
 dr-x------ 2 spate fcf 8208 May 28 21:24 fd
 -r--r--r-- 1 spate fcf 120 May 28 21:24 lpsinfo
 -r-------- 1 spate fcf 912 May 28 21:24 lstatus
 -r--r--r-- 1 spate fcf 536 May 28 21:24 lusage
 dr-xr-xr-x 3 spate fcf 48 May 28 21:24 lwp
 -r-------- 1 spate fcf 1728 May 28 21:24 map
 dr-x------ 2 spate fcf 544 May 28 21:24 object
 -r-------- 1 spate fcf 2048 May 28 21:24 pagedata
 -r--r--r-- 1 spate fcf 336 May 28 21:24 psinfo
 -r-------- 1 spate fcf 1728 May 28 21:24 rmap
 lr-x------ 1 spate fcf 0 May 28 21:24 root ->
 -r-------- 1 spate fcf 1440 May 28 21:24 sigact
 -r-------- 1 spate fcf 1232 May 28 21:24 status
 -r--r--r-- 1 spate fcf 256 May 28 21:24 usage
 -r-------- 1 spate fcf 0 May 28 21:24 watch
 -r-------- 1 spate fcf 2736 May 28 21:24 xmap
 

Some of the files make reference to an LWP, a light weight process. The LWP model is used to provide support for multiple threads of control within a process. Grouping threads into an LWP alters the scheduling properties of the different threads.

The various files contained within /proc on a per-process basis are:

Opening this file gives access to the address space of the process. This allows the caller to find a specific address using lseek() and then either read from or write to the address using read() and write().

This file contains dynamic linker information.

cred. The process credentials, defined by the pcred structure, can be found here. This includes information such as the real and effective user IDs, real and effective group IDs, group, and supplementary group information.

ctl. This write-only file is used for process control and accounting. A request may be made to stop or start a process or enable process event tracing. cwd. This file is a symbolic link to the process_f current working directory.

fd. This directory contains files that correspond to the files that the process has open. There is one entry per open file.

lpsinfo, lstatus, lusage. These files give information about each of the process LWPs. Note that there can be multiple LWPs per process; each contains one or more threads.

map. This file contains an array of pmap structures, each of which describes a segment within the virtual address range of the process.

object. Each address space segment maps an underlying file. This directory contains read-only files that are referenced by the map and pagedata files. Opening one of these files gives a file descriptor for the specific mapped file.

pagedata. Opening this file allows the caller to track address space references and modifications on a per-page basis.

psinfo. This file gives general information about the state of the process that is used by the ps command. The psinfo structure, defined in procfs.h, can simply be read from this file.

rmap. Similar to the map file, this file contains an array of prmap structures. These segments are reserved by the operating system for structures such as the stack.

root.This file is a symbolic link to the process_f root directory.

sigact. This file contains an array of sigaction structures which define the disposition of signals associated with the traced process.

status. The information stored in this file, underpinned by the pstatus structure, gives a fairly detailed account about the state of the process. This includes a set of flags that indicate whether the process is runnable, stopped, being single-stepped, and so on. Process group and session information, memory size, and tracing data are some of the other types of information that can be found in this file.

usage. This file, underpinned by the prusage structure, gives a wealth of timing-related information about the process.

watch. This file contains an array of pwatch structures, which enable a process to be debugged. The controlling process can set breakpoints in the process by writing a PCWATCH message through the ctl file.

The lwp directory contains further information about each light weight process.

Accessing Files in the Solaris /proc Filesystem

#include < fcntl.h>
 #include < procfs.h>
 main(int argc, char *argv[])
 {
 struct psinfo ps;
 char fname[256];
 int fd;
 sprintf(fname, "/proc/%s/psinfo", argv[1]);
 fd = open(fname, O_RDONLY);
 read(fd, (char *)&ps, sizeof(struct psinfo));
 printf("UID\tPID\tPPID\tCMD\n");
 printf("%d\t%d\t%d\t%s\n",
 ps.pr_uid, ps.pr_pid, ps.pr_ppid, ps.pr_psargs);
 }

$ ./mps 3707
 UID PID PPID CMD
 824 3707 1 sleep 100000

Tracing and Debugging with /proc

Access to the ctl file, which is write only, is achieved by writing an operational code to the file together with any additional data required for the operation in question. The controlling process tracks three different types of events, namely: Signals. A stop based on a signal is handled in all cases where the signal is detected, whether on return from a system call or trap, or during process wakeup.

System calls. The process is stopped either when the kernel is entered to process a system call or is just about to exit from the kernel back to user space after the system call has been processed.

Faults. There are a number of different fault types that can be managed, some of which depend on the type of architecture on which the operating system is running. Fault types include illegal instructions, breakpoints, memory access, and trace traps (used for single stepping).

The truss command is a prime example of a utility that controls another process. Its role is to display the system calls made by another process including the system call arguments and return values. The PCSENTRY and PCSEXIT control functions determine whether a process stops on entry to or exit from a system call. The system calls to be traced are held in the sysset_t structure, which is passed along with the PCSENTRY and PCSEXIT control functions. The prfillset() function can be used to build the complete set of system calls, because truss will monitor all system calls. For a more controlled trace, the set of system calls monitored can be altered using the praddset() and prdelset() library functions.

There are a number of different control messages that both stop and start a process. As an example of those functions that are relevant to truss, the PCSTOP function directs the process to stop on an event of interest and waits for it to stop. An event of interest is defined by invoking PCSTRACE (signals to be traced), PCSFAULT (faults to be traced), PCSENTRY (system call entry), or PCSEXIT (system call exit). The PCRUN control function makes the process runnable again. The following pseudo code gives a high-level view of how the truss utility can be implemented:

prfillset(&syscalls)
 PCSENTRY(syscalls)
 PCSEXIT(syscalls)
 do {
 PCSTOP()
 extract system call arguments
 PCSTART()
 PCSTOP()
 extract system call return value
 display system call type, arguments and return value
 PCSTART()
 } while (syscall type != exit);

There are a large number of control functions that make a debugger writer_fs life much easier. If the debugger is interested in fault types, the following are relevant:

FLTBPT. A breakpoint trap.
 FLTTRACE. A trace trap (used for single stepping).
 FLTWATCH. A watchpoint trap (used to trap on memory access).

Anyone wishing to study how a real debugger makes use of /proc should look at the Solaris implementation of gdb, the GNU debugger whose source is freely available.

The Specfs Filesystem

Having multiple special files in the namespace caused an additional problem in that there could be multiple buffers in the buffer cache corresponding to the same block on disk. Considering how files are accessed, returning a filesystem vnode for a device file is incorrect. For example, consider the case where the device file resides on a UFS filesystem. Returning a vnode that has the v_op field of the vnode set to the list of UFS vnode operations will lead to problems. First, the open vnode operation on UFS or any other filesystem really has no function to perform for regular files. Second, many of the operations that are applicable to regular files are not applicable to device files. To make matters worse, if the vnode goes inactive, the filesystem may attempt to close the device even though it is open through access to another special file that references the same device.

All of these problems can be solved by adding additional logic inside the filesystem. However, consideration must be given on how to handle device access for each vnode operation. Furthermore, reference counting to determine when the last close on a device occurs is left up to the device driver. All in all, this leads to a situation that has a lot of duplication and is prone to errors.

To solve these problems, a new filesystem type, specfs, was introduced in SVR4. The specfs filesystem is not visible to users in that it cannot be mounted or seen from within the namespace.

During a VOP_LOOKUP() operation, instead of returning a vnode which corresponds to the special file, the filesystem makes a call to specvp() which returns a new specfs vnode, that the filesystem must return from the lookup operation. This vnode points to a specfs node (snode), a private specfs data structure that references the real vnode of the filesystem.

In the case where one device has more than one entry in the namespace, the snode also points to a common specfs vnode. It is through this common vnode that device access actually takes place.

The following example shows the linkage between two device special files and the common specfs vnode that represents both. This is also shown in Figure 11.2. First of all consider the following simple program, which simply opens a file and pauses awaiting a signal:

#include < fcntl.h>
 main(int argc, char *argv[])
 {
 int fd;
 fd = open(argv[1], O_RDONLY);
 pause();
 }

# ls -l /dev/null
 crw-r--r-- 1 root other 13, 2 May 30 09:17 mynull
 # mknod mynull c 13 2
 # ls -l mynull
 crw-r--r-- 1 root other 13, 2 May 30 09:17 mynull

# ./dopen /dev/null &
 [1] 3715
 # ./dopen mynull &
 [2] 3719

# crash
 dumpfile = /dev/mem, namelist = /dev/ksyms, outfile = stdout
 > p ! grep dopen
 336 s 3719 3713 3719 3713 0 46 dopen load
 363 s 3715 3713 3715 3713 0 46 dopen load

> user 336
 ...
 OPEN FILES, POFILE FLAGS, AND THREAD REFCNT:
 [0]: F 300106fc690, 0, 0 [1]: F 300106fc690, 0, 0
 [2]: F 300106fc690, 0, 0 [3]: F 300106fca10, 0, 0
 ...

The file structure and its corresponding vnode are then displayed as shown:

> file 300106fca10
 ADDRESS RCNT TYPE/ADDR OFFSET FLAGS
 300106fca10 1 SPEC/300180a1bd0 0 read
 > vnode 300180a1bd0
 VCNT VFSMNTED VFSP STREAMP VTYPE RDEV VDATA VFILOCKS VFLAG
 1 0 300222d8578 0 c 13,2 300180a1bc8 0 -
 > snode 300180a1bc8
 SNODE TABLE SIZE = 256
 HASH-SLOT MAJ/MIN REALVP COMMONVP NEXTR SIZE COUNT FLAGS
 - 13,2 3001bdcdf50 30001b5d5b0 0 0 0
 The REALVP field references the vnode for the special file within the filesystem
 that references mynull.
 For the process that opens the /dev/null special file, the same sequence of
 operations is followed as shown:
 > user 363
 ...

Note that for the snode displayed here, the COMMONVP field is identical to the COMMONVP field shown for the process that referenced mynull.

To some readers, much of what has been described may sound like overkill. However, device access has changed substantially since the inception of specfs. By consolidating all device access, only specfs needs to be changed. Filesystems still make the same specvp() call that they were making 15 years ago and therefore have not had to make any changes as device access has evolved.

The BSD Memory-Based Filesystem (MFS)

A RAM disk is typically a contiguous section of memory that has been set aside to emulate a disk slice. A RAM disk-based device driver is the interface between this area of memory and the rest of the kernel. Filesystems access the RAM disk just as they would any other physical device. The main difference is that the driver employs memory to memory copies rather than copying between memory and disk.

The paper describes the problems inherent with RAM disk-based filesystems. First of all, they occupy dedicated memory. A large RAM disk therefore locks down memory that could be used for other purposes. If many of the files in the RAM disk are not being used, this is particularly wasteful of memory. One of the other negative properties of RAM disks, which the BSD team did not initially attempt to solve, was the triple copies of data. When a file is read, it is copied from the file_fs location on the RAM disk into a buffer cache buffer and then out to the user_fs buffer. Although this is faster than accessing the data on disk, it is incredibly wasteful of memory.

The BSD MFS Architecture

1. A call to newfs is made indicating that the filesystem will be memory-based.

2. The newfs process allocates an area of memory within its own address space in which to store the filesystem. This area of memory is then initialized with the new filesystem structure.

3. The newfs command call is made into the kernel to mount the filesystem. This is handled by the mfs filesystem type that creates a device vnode to reference the RAM disk together with the process ID of the caller.

4. The UFS mount entry point is called, which performs standard UFS mount time processing. However, instead of calling spec_strategy() to access the device, as it would for a disk-based filesystem, it calls mfs_strategy(), which interfaces with the memory-based RAM disk.

One unusual aspect of the design is that the newfs process does not exit. Instead, it stays in the kernel acting as an intermediary between UFS and the RAM disk.

As requests for read and write operations enter the kernel, UFS is invoked as with any other disk-based UFS filesystem. The difference appears at the filesystem/driver interface. As highlighted above, UFS calls mfs_strategy() in place of the typical spec_strategy(). This involves waking up the newfs process, which performs a copy between the appropriate area of the RAM disk and the I/O buffer in the kernel. After I/O is completed, the newfs process goes back to sleep in the kernel awaiting the next request.

After the filesystem is unmounted the device close routine is invoked. After flushing any pending I/O requests, the mfs_mount() call exits causing the newfs process to exit, resulting in the RAM disk being discarded.

Performance and Observations

There is a disadvantage with this approach; a large RAM disk will consume a large amount of swap space and therefore could reduce the overall amount of memory available to other processes. However, swap space can be increased, so MFS still offers advantages over the traditional RAM disk-based approach.

The Sun tmpfs Filesystem

Chapter 7 described the SVR4 filesystem architecture on which tmpfs is based. In particular, the section An Overview of the SVR4 VM Subsystem in Chapter 7, described the SVR4/Solaris VM architecture. Familiarity with these sections is essential to understanding how tmpfs is implemented. Because tmpfs is heavily tied to the VM subsystem, it is not portable between different versions of UNIX. However, this does not preclude development of a similar filesystem on the other architectures.

Architecture of the tmpfs Filesystem

Tmpfs uses anonymous memory to store file data and therefore competes with memory used by all processes in the system (for example, for stack and data segments). Because anonymous memory can be paged to a swap device, tmpfs data is also susceptible to paging.

Figure 11.4 shows how the tmpfs filesystem is implemented. The vnode representing the open tmpfs file references a tmpfs tmpnode structure, which is similar to an inode in other filesystems. Information within this structure indicates whether the file is a regular file, directory, or symbolic link. In the case of a regular file, the tmpnode references an anonymous memory header that contains the data backing the file.

File Access through tmpfs

Performance and Other Observations

[SNYD90] also shows a test under which the UNIX kernel was recompiled. The overall time for a UFS filesystem was 32 minutes and for tmpfs, 27 minutes. Filesystems such as VxFS, which provide a temporary filesystem mode under which nearly all transactions are delayed in memory, could close this gap significantly.

One aspect that is difficult to measure occurs when tmpfs file data competes for virtual memory with the applications that are running on the system. The amount of memory on the system available for applications is a combination of physical memory and swap space. Because tmpfs file data uses the same memory, the overall memory available for applications can be largely reduced.

Overall, the deployment of tmpfs is highly dependent on the type of workload that is running on a machine together with the amount of memory available. 262 UNIX Filesystems.Evolution, Design, and Implementation

CHAPTER 13 Clustered and Distributed Filesystems

The 1980s saw a number of different distributed filesystems make an appearance in the UNIX community, including Sun_fs Network Filesystem (NFS), AT&T_fs Remote File Sharing (RFS), and CMU_fs Andrew File System (AFS) which evolved into the DCE Distributed File Service (DFS). Some of the distributed filesystems faded as quickly as they appeared. By far, NFS has been the most successful, being used on tens of thousands of UNIX and non-UNIX operating systems throughout the world.

Distributed filesystems operate around a client/server model, with one of the machines owning an underlying disk-based filesystem and serving files to clients through some well-defined protocol. The protocol and means of transferring files to the user is transparent, with UNIX file semantics being a key goal.

Clustered filesystems by contrast treat a collection of machines and disks as a single entity and provide a fully coherent view of the filesystem from any of the nodes. To the user, clustered filesystems present a single coherent view of the filesystem and may or may not offer full UNIX file semantics. Clustered filesystems as well as local filesystems can be exported for use with NFS. 286 UNIX Filesystems.Evolution, Design, and Implementation

Distributed Filesystems

Regardless of the type of distributed filesystem, one goal that is absolutely essential to all of these filesystems is the need to provide UNIX file semantics when accessing remote files from the client.

There have been numerous distributed filesystems developed for UNIX over the last 20 years. Many of them have come and gone. The most successful distributed filesystem by far is the Sun Network Filesystem (NFS) which appeared in the mid 1980s. Although not as feature-rich as filesystems such as the DCE Distributed File Service (DFS), the simplicity of the NFS protocol, together with the fact that the NFS protocol is in public domain, resulted in it being ported to many different operating systems, UNIX and non-UNIX alike.

The following sections describe some of the main UNIX distributed filesystems with particular emphasis on NFS.

The Network File System (NFS)

Hard disks were still relatively expensive in the late 1970s and early 1980s. By providing a client/server file protocol such as NFS, hardware designers were free to build diskless workstations or least workstations with a minimal amount of local storage.

NFS Background and History

Provide machine and OS independence. This goal was to ensure that NFS could work on UNIX and non-UNIX operating systems. The client/server protocols were to be simple enough that they could be implemented on PCs.at the time, a DOS-based environment.

Provide resilience to server crashes. If the server crashes, the clients who are currently accessing the server must be able to recover. Furthermore, the client should be unable to tell the difference between a server that crashed and restarted and one that was just slow in responding.

Provide transparent file access. In order for the filesystem to succeed, it was important that applications could access files through NFS in the same manner in which they could access files on a local disk.

Maintain UNIX semantics on the client. To satisfy the above goal in UNIX environments, it was imperative that NFS provide UNIX file semantics to the applications on the client.

Have acceptable performance. As stated in [SAND85] _gPeople will not want to use NFS if it is no faster than the existing network utilities, such as rcp, even if it is easier to use._h The performance targets were set at 80 percent of local disk access.

There were three pieces that comprised NFS, the protocol, the client, and the server. All three will be described throughout the following sections.

The original NFS implementation, as described in [SAND85], encompassed both version 1 and 2 of the protocol when it first became available to the public in SunOS 2.0 in 1985. The first version of the protocol was only used internally within Sun.

At the time of writing, NFS implementations adhering to version 3 of the protocol have been in common use for several years and version 4 implementations are starting to appear. NFS is very well understood by tens of thousands of people throughout the world, which is a great testament of its success and an indicator as to why it will still be one of the most dominant of the distributed filesystems for many years to come.

The Version 1 and 2 NFS Protocols

To communicate across the network, NFS used the User Datagram Protocol (UDP) on top of the Internet Protocol (IP). Because the protocol was designed to be independent of machine architectures and operating systems, the encoding of the protocol messages and their responses was sensitive to issues such as endianess (the order in which bytes are packed into a machine word). This resulted in the use of the External Data Representation (XDR) specification.

The use of RPC and XDR are described in the following section. Before describing how they are used, it first helps to describe the actual procedure calls NFS introduced for communication across the network. The version 2 protocol was documented in [RFC1094], which describes the procedure calls shown in

Table 13.1. Most of the operations are self explanatory. The null procedure is used to ping the server and can also be used as a means of measuring the round trip time. The statfs procedure returns information that can be displayed when making a call to the df command.

The file referenced by these procedures is called a file handle. This is an opaque data structure provided by the server in response to a lookup request. The client should never try to interpret the contents of the file handle. File handles are constructed using a combination of operating-specific information and information provided by the filesystem. For the latter, the information must provide a means to locate the file, so the file handle is typically a combination of filesystem specific information together with the inode number of the file and its generation count.

Many of the procedures deal with file attributes. Not surprisingly, the attributes correspond to the various fields of the stat structure as described in the section Basic File Properties in Chapter 2.

The NFS server is stateless in that there is no information kept on the server about past requests. This avoids any complicated crash recovery mechanism. If the client does not receive a response from the server within a specific period of time, the request is retried until it succeeds. This tolerates a server crash and reboot, ensuring that the client cannot detect the difference between a server crash and a server that is simply slow in responding.

The version 2 protocol also requires that any file writes are synchronous. This meets the objective of achieving UNIX file semantics.

Within the file handle, the inode generation count is typically encoded. Consider the case when a file is opened and a file handle is returned. If the file is removed on the server and the inode is reused later for a new file, the file handle is no longer valid because it refers to the old file. To distinguish between the old and new files, UNIX filesystems contain a generation count that is incremented each time the inode is reused. By using the generation count within the file handle, the stale file handle will be detected by the server when the deleted file is referenced.

One of the main goals of NFS was to make it portable across different operating systems. [ROSE86] demonstrated early ports to both an SVR2.2-based version of UNIX and the Sequent Dynix operating system, a System V/BSD hybrid. There were a number of different PC (DOS-based) implementations of NFS. [CALL00] describes the various issues encountered with porting NFS to these platforms.

NFS Client/Server Communications

The XDR format assumes that bytes (8-bit) are portable in that the encoding of bytes does not change from one architecture or hardware device to another. Building on top of bytes, the XDR specification requires data types to be constructed from multiples of four bytes of data. If data types require a number of bytes that is not exactly divisible by 4, any unused bytes will be zero-filled. The ordering of the bytes is such that, if read from a byte stream, the high order byte is always read first. This is called big-endian (the biggest digit in a number comes first). XDR also uses a standard representation for floating point numbers (following the IEEE standard).

To give a simple example of how XDR is used, consider the encoding of an integer that is defined as a 32-bit data type. This is encoded as follows:

So if this data type were being read from a regular file as a series of bytes, byte 0, the most significant byte, would be read first.

The XDR specification defines a number of primitive data types including signed and unsigned integers, booleans, hyper integers (64 bits), fixed and variable length opaque data types, and strings. It also defines a wide range of structured data types including arrays, unions, linked lists, and structures. In basic terms, any data type that can be defined in the most popular high-level languages can be encoded within XDR.

The RPC mechanism used for NFS was derived from Sun RPC, a simple way to provide a remote procedure call mechanism between two processes on different machines. To the caller of such a procedure, there is no difference between calling an RPC function and calling a local function.

The RPC protocol [RFC1057] can be implemented on top of several different transport protocols. In the case of the early NFS implementations, this was based on UDP/IP. Within the last ten years, a move to using TCP/IP has been made in many environments (typically to avoid packet loss when going through routers). Description of the RPC protocol is beyond the scope of this book. The NFS specification [RFC1094] defines the NFS protocol as an RPC program and [CALL00] provides a more detailed description of the protocol itself.

The overall architecture of NFS in a VFS/vnode architecture is shown in Figure 13.1. This shows the placement of NFS, XDR, and RPC within the kernel. To the rest of the kernel, NFS appears as any other filesystem type.

Exporting, Mounting, and Accessing NFS Filesystems

There were two reasons for separating the mount protocol from the NFS protocol itself (note that both are described in [RFC1094]). First, the means by which access checking is performed on the server is typically implemented in user space, which can make use of a number of different security mechanisms. Because the NFS protocol is implemented within the kernel for performance reasons, it was felt that it was easiest to allow for this separation. The second reason that the protocols were separated was that the designers thought a single pathname to file handle procedure would tie the implementation of NFS too closely with UNIX. It was envisaged that the pathname to file handle translation may be implemented by a protocol that differs from the mount protocol.

The initial mount protocol consisted of six different procedures:

MOUNTPROC_NULL. This procedure performs no specific function. It is used to ping the server to measure the round-trip time, which can be used to determine how to time out NFS procedure calls to the server.

MOUNTPROC_MNT. This procedure takes a pathname and returns a file handle that corresponds to the pathname.

MOUNTPROC_DUMP. This function returns a list of clients and the exported filesystems that they have mounted. This is used by the UNIX commands showmount and dfmounts that list the clients that have NFS mounted filesystems together with the filesystems that they have mounted.

MOUNTPROC_UMNT. This procedure is used to inform the server that the NFS filesystem on the client has been unmounted.

MOUNTPROC_UMNTALL. This procedure is sent by a client following a reboot (or after a crash). This prevents the server from maintaining stale mount entries in the event that the client crashed before sending corresponding MOUNTPROC_UMNT messages.

MOUNTPROC_EXPORT. This procedure returns a list of exported filesystems.

The mount protocol remained unchanged for a number of years. Since then, only one additional procedure has been added, MOUNTPROC_PATHCONF, which retrieves additional information from the server allowing NFS filesystems to comply with the POSIX pathconf() system call.

Using NFS

As an example, consider the case in Solaris where a server called srv wishes to export a filesystem mounted on /fs1 to any client. The easiest way to achieve this is to place an entry in /etc/dfs/dfstab that specifies the filesystem to be shared and/or exported. This ensures that the filesystem will be exported for use on each reboot. If no options are needed for this filesystem, the following line is sufficient:

share -F nfs /fs1

On the client side, a mount call can then be made to mount the filesystem as follows:

# mount -F nfs srv:/fs1 /mnt
 # mount | grep mnt
 /mnt on srv:/fs1 remote/read/write/setuid/dev=2fc004
 on Wed Jun 19 07:25:18 2002

The Version 3 NFS Protocol

Only files up to 4GB in size could be accessed. This limitation was exposed early on when running NFS on large machines but was rapidly becoming a problem in most environments.

Because all writes were required to be written synchronously, write intensive applications suffered a performance bottleneck. There were numerous workarounds for this problem, including some that violated the NFS protocol by performing writes asynchronously.

The goals of those involved in designing NFS version 3 were to solve the two problems described above, tidy up the existing version 2 protocol, and add some minor features, but at the same time limit the scope of the version 3 protocol to avoid it becoming too bloated to be implemented in a timely manner.

[PAWL94] provides an overview of the process the designers of the version 3 protocol went through, the problems inherent in the version 2 protocol, the goals behind the version 3 protocol, the changes introduced with the version 3 protocol, and various implementation and performance-related information. In this paper, they identify twelve main areas in which the protocol was enhanced:

All arguments within the protocol such as file sizes and file offsets were widened from 32 to 64-bits. This solved the 4GB file restriction.

The write model was changed to introduce a write/commit phase that allowed for asynchronous writes. (This will be described further).

A new ACCESS procedure was introduced to solve permission checking problems when mapping the ID of the superuser. This procedure works in the presence of ACLs (Access Control Lists).

In the original protocol, some of the procedures required a subsequent call in order to retrieve file attributes. In the new protocol, all procedures returned file attributes.

In the original protocol, writes were limited to 8Kb per procedure call. This restriction was relaxed in the new protocol.

The READDIRPLUS procedure was introduced that returned both a file handle and attributes. This eliminated some lookup calls when scanning a directory.

The file handle size in the version 2 protocol was a fixed, 32-byte opaque data type. In version 3, it was changed to be of variable size up to a maximum of 64 bytes.

The CREATE procedure was modified to allow for exclusive file creates. This solved a workaround in the version 2 protocol whereby a LOOKUP was followed by a CREATE, which left a window in which another client could create the file.

The version 2 protocol limited the size of filenames and pathnames to 255 and 1024 characters respectively. In version 3, this was replaced by variable length strings which could be agreed on between the client and server.

The version 3 protocol tightened the errors that could be returned from the server. All error values are iterated, and no errors outside of the list are permitted.

For the set of file attributes, the blocksize field was removed. The blocks field was changed to used and recorded the total number of bytes used by the file.

A new error type, NFS3ERR_JUKEBOX, was introduced. In a Hierarchical Storage Management (HSM) environment, a request may be made to the server to read a file that has been migrated to tape. The time to read the data back in from tape could be quite large. This error informs the client that the operation is in progress and that the call should be retried. It also allows the client to display a message on the user_fs console if applicable.

Writes in UNIX are asynchronous by default unless the O_SYNC or O_DSYNC flags are passed to open(). Forcing asynchronous writes to be synchronous inevitably affects performance. With NFS version 3, the client can send a number of asynchronous WRITE requests that it can then commit to disk on the server at a later date by issuing a COMMIT request. Once it receives a COMMIT request, the server cannot return until all data has been flushed to disk. In some regards, the COMMIT request is similar to calling fsync(). The most noticeable difference is that the COMMIT request does not necessarily cover the data for the whole file. It does however allow the client to flush all data when a file is closed or to break up a large synchronous write request into a number of smaller writes, all of which are performed asynchronously but followed by a COMMIT request. This itself is an important enhancement because it allows the filesystem on the server or the disk driver to coalesce a number of writes in a single large write, which is more efficient. The use of asynchronous writes should not affect the crash/recovery properties of NFS because the client is required to keep a copy of all data to be written to the file until a COMMIT is issued.

The READDIRPLUS procedure, while it can be extremely effective, also presents problems. The procedure was introduced to minimize the number of over-the-wire LOOKUP requests once a READDIR procedure had been invoked. This would typically be the case when issuing an ls -F request on a directory.

Because the implementation of READDIRPLUS is significantly more expensive than READDIR, it should be used with caution. Typically, the operation should be performed only when first accessing the directory in order to populate the DNLC (or other name cache, depending on the underlying OS). The operation should then be performed again only in the case where the cache was invalidated for the directory due to a directory modification.

Because many of the goals of NFS version 3 were to improve performance, the proof of its success was therefore dependent on how well it performed. [PAWL94] documented a number of different performance-related tests that showed that the version 3 protocol did in fact meet its objectives.

The NFS Lock Manager Protocol

However, file locking was not something that could be easily overlooked and was therefore implemented in SunOS as the Network Lock Manager (NLM). Various iterations of the NLM protocol appeared, with NLM version 3 being the version that was most widely used with NFS version 2. Unfortunately, due to the complexity of implementing support for locking, the NLM protocol was not widely implemented. With the introduction of NFS version 3 [RFC1813], the definition of NLM (version 4) was included with the NFS specification but was still a separate protocol. The NLM protocol also relied on the Network Status

Monitor protocol that was required in order to notify clients and servers of a crash such that lock state could be recovered.

Crash/recovery involves coordination between both clients and server as shown here:

Server crash. When locks are handed to clients, the server maintains a list of clients and the locks that they own. If the server crashes, this information is lost. When the server reboots, a status monitor runs and sends a message to all known clients. The lock manager on each client is notified and is given an opportunity to reclaim all locks it owns for files on the server. There is a fixed amount of time (or grace period) in which the clients can respond. Note however, that this is not an ideal situation as notification of clients may be delayed, because the status monitor typically informs all clients in a serial manner. This window may be reduced by multithreading the status monitor.

Client crash. If the client crashes, any locks that the client holds on the server must be cleaned up. When the client resumes, a message is sent to the server to clean up its locks. Through use of a client state number, which is incremented on reboot, the server is able to detect that the client has been rebooted and removes any of the locks that were held by the client before it crashed/rebooted.

Since the NLM was not widely adopted, version 4 of the NFS protocol has been extended to include file locking. This is described in the next section.

The Version 4 NFS Protocol and the Future of NFS

NFS was designed for local area networks and was put in place before the widespread adoption of the World Wide Web. As time goes by there is more of a need to use NFS in wide area networks. This raises questions on security and further highlights the need for NFS to address cross-platform issues. Although one of the goals of the original NFS implementations was to support non-UNIX platforms, the protocol was still heavily geared towards UNIX environments.

The goals of the version 4 protocol are to address the problems highlighted above and to provide additional features that were omitted in the version 3 protocol (version 3 changes were kept to a minimum to ensure that it could be designed and implemented in a timely manner). Although the version 4 protocol involves some substantial changes, the goals are to allow small incremental changes that do not require a complete overhaul of the protocol. The time between the version 2 and 3 protocols was approximately 8 years, which is similar to the time between the version 3 and 4 protocols. A minor revision to the version 4 protocol allows new features to be added to the version 4 protocol in a much more timely manner, say a 1 to 2 year timeframe.

The version 4 protocol is described in [RFC3010]. Following are the main changes that are part of the new protocol. Note that the changes introduced with the version 4 protocol are substantial and only covered briefly here.

Compound procedures. Many file-related operations over NFS require a large number of procedure calls. In a local area network this is not such a great issue. However, when operating in a wide area network the effect on performance is much more noticeable. By combining a number of procedure calls into a single, compound procedure, the amount of over-the-wire communications can be reduced considerably resulting in much better performance.

Internationalization. In previous versions of the protocol, file names were handled as an opaque byte stream. Although they were typically limited to a 7-bit US ASCII representation, they were commonly encoded in 8-bit ISO-Latin-1. Problems occurred because there was no way to specify the type of encoding within XDR. This limited the use of NFS in environments where there may be mixed character sets. To provide better support for internationalization, file and directory names will be encoded with UTF-8.

Volatile file handles. The NFS version 2 and version 3 protocols provided a single file handle type with one set of semantics. This file handle is defined as having a value that is fixed for the lifetime of the filesystem to which it refers. As an example, a file handle on UNIX comprises, amongst other things, the inode number and generation count. Because inodes can be freed and reallocated, the generation count of the inode is increased when reused to ensure that a client file handle that refers to the old file cannot now refer to the new file even though the inode number stays the same. There have also been some implementations that are unable to correctly implement the traditional file handle which inhibits the adoption of NFS on some platforms.

The NFS version 4 protocol divides file handles into both persistent file handles, which describe the traditional file handle, and volatile file handles. In the case of volatile file handles, the server may not always be able to determine whether the file handle is still valid. If it detects that a file handle is in fact invalid, it returns an NFS4ERR_STALE error message. If however it is unable to determine whether the file handle is valid, it can return an NFS4ERR_FHEXPIRED error message. Clients are able to detect whether a server can handle persistent and volatile file handles and therefore act accordingly.

Attribute classes. The set of attributes that were passed over the wire with earlier versions of the protocol were very UNIX-centric in that the information returned by the server was sufficient to respond to a stat() call on the client. In some environments, these attributes are meaningless, and in some cases, servers are unable to provide valid values.

In NFS version 4, the set of file attributes is divided into three different classes, namely mandatory, recommended, and named attributes.

The mandatory set of attributes contain information such as the file type and size, information about file handle expiration times, whether hard links and symbolic links are supported, and whether the file has named data streams/attributes.

The set of recommended attributes contain information such as the type of ACLs (Access Control Lists) that the filesystem supports, the ACLs themselves, information about the owner and group, access timestamps, and quota attributes. It also contains information about the filesystem such as free space, total number of files, files available for use, and filesystem limits such as the maximum filename length and maximum number of links.

Named attributes, also called named data streams, allow a single file to have multiple streams of opaque bytes unlike the traditional UNIX model of supporting a single stream of bytes per file. To access named data streams over NFS version 4, the OPENATTR procedure can retrieve a virtual attribute directory under which READDIR and LOOKUP procedures can be used to view and access the named attributes.

Better namespace handling. Both NFS version 2 and 3 servers export a set of independent pieces of their overall namespace and do not allow NFS clients to cross mountpoints on the server, because NFS expects all lookup operations to stay within a single filesystem. In NFS v4, the server provides a single root file handle through which clients can obtain file handles for any of the accessible exports.

NFS v4 servers can be made browsable by bridging exported subtrees of the namespace with a pseudo filesystem and allowing clients to cross server mountpoints. The tree constructed by the server is a logical view of all the different exports.

File locking. As mentioned earlier, locking is not part of the NFS version 2 or version 3 protocols which rely instead on the Network Lock Manager (NLM) protocol, described in the section The NFS Lock Manager Protocol, earlier in the chapter. The NLM protocol was not, however, widely adopted. NFS version 4 provides for both UNIX file-level locking functions and Windows-based share locking functions. NFS version 4 supports both record and byte range locking functions.

Client side caching. Most NFS clients cache both file data and attributes as much as possible. When moving more towards wide area networks, the cost of a cache miss can be significant. The problem with the version 2 and version 3 protocols is that NFS does not provide a means to support cache coherency between multiple clients, which can sometimes lead to invalid file data being read.

NFS version 4 does not provide cache coherency between clients but defines a limited set of caching guarantees to allow locks and share reservations to be used without destructive interference from client-side caching. NFS v4 also provides a delegation scheme that allows clients to make decisions that were traditionally made by the server.

The delegation mechanism is an important feature in terms of performance because it limits the number of procedure calls that would typically go between client and server when accessing a file. When another client attempts to access a file for which a delegation has been granted, the server invokes an RPC to the client holding the delegation. The client is then responsible for flushing any file information, including data, that has changed before responding to the recall notice. Only after the first client has responded to the revoke request will the second client be allowed to access the file.

The NFS version 4 specification provides many details for the different types of delegations that can be granted and therefore the type of caching that can be performed on the client.

Built-in security. NFS has relied on the UNIX-centric user ID mechanisms to provide security. This has generally not been a problem because NFS has largely been used within private networks. However, because one of the goals of the version 4 protocol is to widen the use of NFS to wide area networks, this level of security is insufficient. The basic NFS security mechanisms are being extended through use of the RPCSEG_GSS framework. The RPCSEG_GSS mechanisms are implemented at the RPC layer and are capable of providing both private keys schemes, such as Kerboros version 5, and public key schemes.

The NFS version 4 protocol is a significant rework of the version 3 protocol. It provides a wide range of features aimed at continuing its success as it becomes more widely adopted in wide area networks, and it provides better support for building a distributed filesystem for heterogeneous operating systems. There was a huge amount of investment in NFS prior to version 4. Because NFS version 4 attempts to address many of the prior limitations of the earlier versions, including more attention to non-UNIX operating systems, NFS is likely to grow in popularity.

The NFS Automounter

This problem was resolved by introduction of the automounter. This aids in creation of a unified namespace while keeping the number of mounted filesystems to only those filesystems that are actually in use. The automounter is simple in nature. When a user attempts to access a file that crosses a mount point within the boundaries of the automounter, the NFS filesystem is first mounted prior to allowing the access to proceed. This is shown in Figure 13.2. The first automounters were implemented as user space daemons, which typically mount themselves on those directories that require automounter services and masquerade as NFS servers. When an attempt is made to access a file within one of these filesystems, the kernel sends an NFS LOOKUP call to the server, in this case the automounter. The automounter then NFS mounts the real filesystem onto a directory somewhere within its own mount space. The real filesystem is then referenced through symbolic links. For example, in Figure 13.2, the filesystem to be mounted on fs2 may be mounted on /auto/f2 and /mnt/fs1 will be a symbolic link to this directory.

In many environments it is usual to see a combination of standard NFS mounted filesystems and automounted filesystems. The automounter should be used for filesystems that are not accessed frequently, such as manual pages, source code repositories, and so on. User directories and bin directories are examples of directories that are typically mounted through standard NFS means.

Another common use of the automounter is to use it in conjunction with the Network Information Service (NIS) in an environment where user home directories are distributed throughout the network. In this way, NIS centrally manages all of the NFS mounts from one of the servers. Although each user_fs home directory physically resides on only one server, the same server is configured to export the home directory to all hosts on the network. Each host on the network runs the automounter as an NFS client and can therefore mount a user_fs home directory. This allows the user to log in to any host and have access to his/her home directory. In this environment, file access is enhanced transparently while the use of the automounter avoids the overhead caused by dozens of active but unused NFS mounted filesystems.

Automounter Problems and the Autofs Filesystem

Symbolic links. The preceding section described how the automounter actually NFS mounts the filesystem on a temporary directory and refers to it through a symbolic link. Because the goal of the automounter is only to mount filesystems when required, it periodically unmounts the filesystem if there is no activity for a predetermined amount of time.

However, if a process issues a getcwd() system call, the real path may be cached which references the temporary directory structure, that is, where the filesystem is actually mounted. If the path is used later, there is no guarantee that the filesystem is still mounted and the automounter is unable to detect that access is being requested. The user process may therefore see the local directory structure and thus unpredictable results.

Adding new mountpoints. The list of filesystems that the automounter manages is consulted only when the automounter first starts. A workaround is to terminate and restart the automounter.obviously not an ideal solution.

Performance. The method of sending NFS requests to the automounter when crossing its mount point, together with the management of symbolic links, is more time consuming than accessing an NFS filesystem directly.

Single threading. Because the automounter is single threaded it can only handle one request at a time. Therefore, when in the process of mounting an NFS filesystem, all subsequent access is blocked.

The autofs filesystem replaced the user-level automounter daemon with an in-kernel filesystem type. The automounter daemon is still retained. However, when it starts, it mounts autofs in place for each of the filesystems that is to be managed. In the previous example, the autofs filesystem would be mounted on /mnt. When access is detected to, say /mnt/fs2, autofs invokes an RPC request to communicate with the automounter daemon that NFS mounts the filesystem on /mnt/fs2. Once this is achieved, the autofs filesystem does not intercept any further operations. This eradicates the symbolic link problem and therefore increases the overall performance.

The Remote File Sharing Service (RFS)

The RFS Architecture

RFS also relied on an RPC protocol that provided a procedure on the server for every file-related system call. XDR was used to encode data types but only where the machine architecture of the client and server differed. On the server side, the goal was to emulate the environment of the client and provide a context similar to one on the caller to handle the remote procedure calls. This was used to provide management of the process user and group IDs, umask, and so on. This emulation was a little awkward for some operations. For example, when performing a lookup on a pathname, if an RFS mount point was to be crossed, the remainder of the pathname was sent to the server to be resolved. If a series of ".." components took the pathname out of the RFS mounted filesystem, the operation had to be aborted and completed on the client.

To provide for full UNIX semantics including file and record locking, RFS was required to provide a stateful server. This required the server to maintain reference counts for every open call from every client, file and record lock information on the server, and information about the state of named pipes. RFS also maintained a list of all client mounts. If either the server or one of the clients crashed, there was a significant amount of crash/recovery to be performed. If a client crashed, the server was required to remove all traces of the client. Amongst other things, this included decrementing reference counts, releasing any locks that client held, and so on.

Server side failure resulted in the ENOLINK error message being returned when any attempts were made to access files on the server. All inodes/vnodes on the client that accessed RFS files were marked to indicate the failure such that further attempts at access would return ENOLINK without any attempt to communicate with the server.

The overall architecture of RFS is shown in Figure 13.3. Unlike NFS, RFS requires a reliable, virtual circuit transport service. In the figure this is shown as TCP/IP. A virtual circuit is established during the mount and remains in existence for the duration of the mount. For each client/server pair, the virtual circuit is shared if a client mounts more than one RFS filesystem. Another big difference between RFS and NFS is the support for client-side caching. RFS implements a write-through cache on each server; that is, writes are always sent to the server at the time the write occurs but the data is cached for subsequent access. Obviously this presents a challenge with respect to cache coherency when a write occurs to a file and data is cached on one of more clients. RFS must invalidate cached copies of the data on clients other than the one from which the write is issued. The client-side caching of file data is subsequently disabled either until the process that issued the write closes the file or a predetermined amount of time has passed since the last write to the file.

Differences between RFS and NFS

NFS is stateless whereas RFS requires a primary name server that coordinates RFS activity.

RFS can map user and group IDs from client to server based on presence of a mapping table. NFS by contrast requires that the IDs are the same on both client and server. More specifically, NFS implementations assume the use of NIS to maintain a consistent user database across the network.

RFS allows access to device files across the network while devices are not accessible across NFS.

RFS names resources, the directories that are advertised, which are communicated to the primary server. This is not required in NFS.

RFS requires a connection-mode virtual circuit environment, while NFS runs in a connectionless state.

RFS provides support for mandatory file and record locking. This is not defined as part of the NFS protocol.

NFS can run in heterogeneous environments, while RFS is restricted to UNIX environments and in particular System V UNIX.

RFS guarantees that when files are opened in append mode (O_APPEND) the write is appended to the file. This is not guaranteed in NFS.

In an NFS environment, the administrator must know the machine name from which the filesystem is being exported. This is alleviated with RFS through use of the primary server.

When reading through this list, it appears that RFS has more features to offer and would therefore be a better offering in the distributed filesystem arena than NFS. However, the goals of both projects differed in that RFS supported full UNIX semantics whereas for NFS, the protocol was close enough for most of the environments that it was used in.

The fact that NFS was widely publicized and the specification was publicly open, together with the simplicity of its design and the fact that it was designed to be portable across operating systems, resulted in its success and the rather quick death of RFS, which was replaced by NFS in SVR4.

RFS was never open to the public in the same way that NFS was. Because it was part of the UNIX operating system and required a license from AT&T, it stayed within the SVR3 area and had little widespread usage. It would be a surprise if there were still RFS implementations in use today.

The Andrew File System (AFS)

The AFS Architecture

Client-Side Caching of AFS File Data

When retrieving data from the server, the client obtains a callback. If another client is modifying the data, the server must inform all clients that their cached data may be invalid. If only one client holds a callback, it can operate on the file without supervision of the server until a time comes for the client to notify the server of changes, for example, when the file is closed. The callback is broken if another client attempts to modify the file. With this mechanism, there is a potential for callbacks to go astray. To help alleviate this problem, clients with callbacks send probe messages to the server on a regular basis. If a callback is missed, the client and server work together to restore cache coherency.

AFS does not provide fully coherent client side caches. A client typically makes changes locally until the file is closed at which point the changes are communicated with the server. Thus, if multiple clients are modifying the same file, the client that closes the file last will write back its changes, which may overwrite another client_fs changes even with the callback mechanism in place.

Where Is AFS Now?

The DCE Distributed File Service (DFS)

The resulting Distributed File Service (DFS) technology bore a close resemblance to the AFS architecture. The RPC mechanisms of AFS were replaced with DCE RPC and the virtual filesystem architecture was replaced with VFS+ that allowed local filesystems to be used within a DFS framework, and Transarc produced the Episode filesystem that provided a wide number of features.

DCE / DFS Architecture

File server. The server that runs the services necessary for storing and exporting data. This server holds the physical filesystems that comprise the DFS namespace.

System control server. This server is responsible for updating other servers with replicas of system configuration files.

Fileset database server. The Fileset Location Database (FLDB) master and replicas are stored here. The FLDB is similar to the volume database in AFS. The FLDB holds system and user files.

Backup database server. This holds the master and replicas of the backup database which holds information used to backup and restore system and user files.

Note that a DFS server can perform one or more of these tasks. The fileset location database stores information about the locations of filesets. Each readable/writeable fileset has an entry in the FLDB that includes information about the fileset_fs replicas and clones (snapshots).

DFS Local Filesystems

Aggregates also supports fileset operations not found on standard UNIX partitions, including the ability to move a fileset from one DFS aggregate to another or from one server to another for load balancing across servers. This is comparable to the migration performed by AFS.

UNIX partitions and filesystems can also be made visible in the DFS namespace if they adhere to the VFS+ specification, a modification to the native VFS/vnode architecture with additional interfaces to support DFS. Note however that these partitions can store only a single fileset (filesystem) regardless of the amount of data actually stored in the fileset.

DFS Cache Management

When a client wishes to access a file, it requests a token for the type of operation it is about to perform, for example, a read or write token. In some circumstances, tokens of the same class allow shared access to a file; two clients reading the same file would thus obtain the same class of token. However, some tokens are incompatible with tokens of the same class, a write token being the obvious example. If a client wishes to obtain a write token for a file on which a write token has already been issued, the server is required to revoke the first client_fs write token allowing the second write to proceed. When a client receives a request to revoke a token, it must first flush all modified data before responding to the server.

The Future of DCE / DFS

Clustered Filesystems

With most critical business functions now heavily reliant on computer-based technology, this downtime is unacceptable. In some business disciplines, seconds of downtime can cost a company significant amounts of money.

By making hardware and software more reliable, clusters provide the means by which downtime can be minimized, if not removed altogether. In addition to increasing the reliability of the system, by pooling together a network of interconnected servers, the potential for improvements in both performance and manageability make cluster-based computing an essential part of any large enterprise.

The following sections describe the clustering components, both software and hardware, that are required in order to provide a clustered filesystem (CFS). There are typically a large number of components that are needed in addition to filesystem enhancements in order to provide a fully clustered filesystem. After describing the basic components of clustered environments and filesystems, the VERITAS clustered filesystem technology is used as a concrete example of how a clustered filesystem is constructed. Later sections describe some of the other clustered filesystems that are available today. The following sections only scratch the surface of clustered filesystem technology. For a more in depth look at clustered filesystems, you can refer to Dilip Ranade_fs book Shared Data Clusters [RANA02].

What Is a Clustered Filesystem?

Clustered filesystem technology is complementary to distributed filesystems. Any of the nodes in the cluster can export the filesystem, which can then be viewed across the network using NFS or another distributed filesystem technology. In fact, each node can export the filesystem, which could be mounted on several clients.

Although not all clustered filesystems provide identical functionality, the goals of clustered filesystems are usually stricter than distributed filesystems in that a single unified view of the filesystem together with full cache coherency and UNIX semantics, should be a property of all nodes within the cluster. In essence, each of the nodes in the cluster should give the appearance of a local filesystem.

There are a number of properties of clusters and clustered filesystems that enhance the capabilities of a traditional computer environment, namely:

Resilience to server failure. Unlike a distributed filesystem environment where a single server crash results loss of access, failure of one of the servers in a clustered filesystem environment does not impact access to the cluster as a whole. One of the other servers in the cluster can take over responsibility for any work that the failed server was doing.

Resilience to hardware failure. A cluster is also resilient to a number of different hardware failures, such as loss to part of the network or disks. Because access to the cluster is typically through one of a number of different routes, requests can be rerouted as and when necessary independently of what has failed. Access to disks is also typically through a shared network.

Application failover. Failure of one of the servers can result in loss of service to one or more applications. However, by having the same application set in a hot standby mode on one of the other servers, a detected problem can result in a failover to one of the other nodes in the cluster. A failover results in one machine taking the placed of the failed machine. Because a single server failure does not prevent access to the cluster filesystem on another node, the application downtime is kept to a minimum; the only work to perform is to restart the applications. Any form of system restart is largely taken out of the picture.

Increased scalability. Performance can typically be increased by simply adding another node to the cluster. In many clustered environments, this may be achieved without bringing down the cluster.

Better management. Managing a set of distributed filesystems involves managing each of the servers that export filesystems. A cluster and clustered filesystem can typically be managed as a whole, reducing the overall cost of management.

As clusters become more widespread, this increases the choice of underlying hardware. If much of the reliability and enhanced scalability can be derived from software, the hardware base of the cluster can be moved from more traditional, high-end servers to low cost, PC-based solutions.

Clustered Filesystem Components

Hardware Solutions for Clustering

While Ethernet and communication protocols such as TCP/IP are unlikely to be replaced as the communication medium between one machine and the next, the host-based storage model has been evolving over the last few years. Although SCSI attached storage will remain a strong player in a number of environments, the choice for storage subsystems has grown rapidly. Fibre channel, which allows the underlying storage to be physically separate from the server through use of a fibre channel adaptor in the server and a fibre switch, enables construction of storage area networks or SANs.

Figure 13.5 shows the contrast between traditional host-based storage and shared storage through use of a SAN.

Cluster Management

Failure detection is typically achieved through some type of heartbeat mechanism for which there are a number of methods. For example, a single master node can be responsible for pinging slaves nodes that must respond within a predefined amount of time to indicate that all is well. If a slave does not respond before this time or a specific number of heartbeats have not been acknowledged, the slave may have failed; this then triggers recovery mechanisms.

Employing a heartbeat mechanism is obviously prone to failure if the master itself dies. This can however be solved by having multiple masters along with the ability for a slave node to be promoted to a master node if one of the master nodes fails.

Cluster Volume Management

In addition to these basic volume types, volumes can also be striped (RAID 0). For a striped volume the volume must span at least two disks. The volume data is then interleaved across these disks. Data is allocated in fixed-sized units called stripes. For example, Figure 13.6 shows a logical volume where the data is striped across three disks with a stripe size of 64KB.

The first 64KB of data is written to disk 1, the second 64KB of data is written to disk 2, the third to disk 3, and so on. Because the data is spread across multiple disks, this increases both read and write performance because data can be read from or written to the disks concurrently.

Volume managers can also implement software RAID-5 whereby data is protected through use of a disk that is used to hold parity information obtained from each of the stripes from all disks in the volume.

In a SAN-based environment where all servers have shared access to the underlying storage devices, management of the storage and allocation of logical volumes must be coordinated between the different servers. This requires a clustered volume manager, a set of volume managers, one per server, which communicate to present a single unified view of the storage. This prevents one server from overwriting the configuration of another server.

Creation of a logical volume on one node in the cluster is visible by all other nodes in the cluster. This allows parallel applications to run across the cluster and see the same underlying raw volumes. As an example, Oracle RAC (Reliable Access Cluster), formerly Oracle Parallel Server (OPS), can run on each node in the cluster and access the database through the clustered volume manager.

Clustered volume managers are resilient to a server crash. If one of the servers crashes, there is no loss of configuration since the configuration information is shared across the cluster. Applications running on other nodes in the cluster see no loss of data access.

Cluster Filesystem Management

Clustered filesystems have additional problems in that they must share the resources of the filesystem across all nodes in the system. Taking a read/write lock in exclusive mode on one node is inadequate if another process on another node can do the same thing at the same time. When a node joins the cluster and when a node fails are also issues that must be taken into consideration. What happens if one of the nodes in the cluster fails? The recovery mechanisms involved are substantially different from those found in the distributed filesystem client/server model. The local filesystem must be modified substantially to take these considerations into account. Each operation that is provided by the filesystem 312 UNIX Filesystems.Evolution, Design, and Implementation must be modified to become cluster aware. For example, take the case of mounting a filesystem. One of the first operations is to read the superblock from disk, mark it dirty, and write it back to disk. If the mount command is invoked again for this filesystem, it will quickly complain that the filesystem is dirty and that fsck needs to be run. In a cluster, the mount command must know how to respond to the dirty bit in the superblock.

A transaction-based filesystem is essential for providing a robust, clustered filesystem because if a node in the cluster fails and another node needs to take ownership of the filesystem, recovery needs to be performed quickly to reduce downtime. There are two models in which clustered filesystems can be constructed, namely:

Single transaction server. In this model, only one of the servers in the cluster, the primary node, performs transactions. Although any node in the cluster can perform I/O, if any structural changes are needed to the filesystem, a request must be sent from the secondary node to the primary node in order to perform the transaction.

Multiple transaction servers. With this model, any node in the cluster can perform transactions.

Both types of clustered filesystems have their advantages and disadvantages. While the single transaction server model is easier to implement, the primary node can quickly become a bottleneck in environments where there is a lot of meta-data activity.

There are also two approaches to implementing clustered filesystems. Firstly, a clustered view of the filesystem can be constructed by layering the cluster components on top of a local filesystem. Although simpler to implement, without knowledge of the underlying filesystem implementation, difficulties can arise in supporting various filesystem features.

The second approach is for the local filesystem to be cluster aware. Any features that are provided by the filesystem must also be made cluster aware. All locks taken within the filesystem must be cluster aware and reconfiguration in the event of a system crash must recover all cluster state.

The section The VERITAS SANPoint Foundation Suite describes the various components of a clustered filesystem in more detail.

Cluster Lock Management

In a clustered environment, these locks, which are still based on primitives provided by the underlying operating system, must be enhanced to provide distributed locks, such that they can be queried and acquired by any node in the cluster. The infrastructure required to perform this service is provided by a distributed or global lock manager (GLM).

���������� �������� ������� ��� Linux Kernel

����� ����� ����������� �������� ������� uxfs. � ��� ���� ����������� ��� �������� ������������� ����������� ��������� , ������ � ������ � ���������� �����. ��� ����� ���� ������� mkfs � fsdb.

�� ���������� , ��� ������������� � ��������� ������ �������� �������. ����� ����������� ����������� ������ �������� ������� � ������� ������� printk(), � ����� ��������� kdb � gdb. ��� �������� ������� ����� ����� ��������� �� ������� �� ��������� �����.

��� ��� �������������� � ������� ��� ����� 2.4.18. ����������� ����� ���������� ��� :

 	www.wiley.com/compbooks/pate
 

���������� ����� �������� �������

�������� ������� ������� ������ �� 512-�������� ������. ��� ���������� � ������� ��������� UX_BSIZE � ������ ux_fs.h.
����� ������ � �������� ������� ����������� - ��� ����� 470. ��� ���������� � ������� ��������� UX_MAXBLOCKS.
� ������� 32 inodes (UX_MAXFILES). ���� 0 � 1 ���������������. ���� 2 ������������ ��� �������,���� 3 - ��� lost+found , 28 ��� ��� ������ � ��������� ������������.
��������� �������� � ����� 0. ������ ���� ��������� �������,���� ��� ��� � ���� ��� ������, ����������� �� �� , ������������ �� ���� ��� ���� ��� ���. � ���������� ���� ���� , ����������� �� ��������� ����� ��� � ������.

 
 struct ux_superblock {
 __u32 s_magic;
 __u32 s_mod;
 __u32 s_nifree;
 __u32 s_inode[UX_MAXFILES]
 }
 
 superblock block 0 {
 
 __u32 s_nbfree;
 __u32 s_block[UX_MAXBLOCKS]
 }
 
 
 struct ux_inode 
 {
 __u32 i_mode;
 __u32 i_nlink;
 data blocks 
 __u32 i_atime;
 __u32 i_mtime;
 __u32 i_ctime;
 __s32 i_uid;
 __s32 i_gid;
 __u32 i_size;
 __u32 i_blocks;
 __u32 i_addr[UX_DIRECT_BLOCKS]
 
 }; 
 

����� ������-������ ������ , ��������-�������������� ���������� �� ������ ������ , ����� �� ��� ��������� hard links, symbolic links, rename, � �.�. 4 �������� �������-������� : super_operations, file_operations, address_space_operations, inode_operations .

Linux Kernel Source

The Linux kernel source can be retrieved from the following Web site:

 	www.kernel.org
 

 The latest stable version of the Linux kernel is: 2.4.18 2002-07-10 00:40
 UTC F V VI Changelog
 

All of the kernels since Linux inception can be found at this site. Follow the links through to the source repositories and locate the kernel of your choice. To use the source in the book as is, you need the 2.4.18 kernel. Alternatively, as described earlier, newer versions of the filesystem can be obtained from the following Web site:

 	www.wiley.com/compbooks/pate
 

To locate the required kernel source, follow the various pointers. As an example, from the home page follow the link to Linux respository, including kernel source, then kernel and 2.4. This will take you to the following link:

 	www.kernel.org/pub/linux/kernel/v2.4/
 

 # bunzip2 linux-2.4.18.tar.bz2
 # mv linux linux.orig
 # tar xvf linux-2.4.18.tar
 # mv linux linux-2.4.18
 # ln -s linux-2.4.18 linux
 

Whats in the Kernel Source Tree

arch. This directory contains a directory for each of the different machine architectures that Linux supports including Intel, Sparc, MIPS, and IBM s390.

CREDITS. This file lists all of the major contributors to the kernel together with information about their area of expertise or contribution.

Documentation. There is a whole host of documentation distributed with the kernel source. The filesystems directory contains information about some of the different Linux filesystems in additional to generic filesystem-related information.

drivers.This directory contains all of the Linux device drivers.

fs. This is the directory that will be of most relevance to people interested in filesystems together with the mm directory that contains much of the page cache/data I/O management code. Files in the fs directory implement the dcache, buffer cache, inode cache, and file-related system call handling. Also within the fs directory is a directory for each of the Linux filesystems. Within their respective directories are the filesystem source files themselves.

include. All of the kernel header files can be accessed within this directory. This directory contains architectural-specific header files in addition to header files that are common across all architectures. The common header files can be found in the linux subdirectory. The fs.h header file is of particular importance to filesystem writers. The dcache.h header file defines the structures used by the Linux dcache.

init. This directory contains functions that are executed during kernel bootstrap.

ipc. This directory contains source applicable to System V IPC (Inter Process Communication) including semaphores, shared memory, and message queues.

 kdb.If the kdbpatch is installed, this directory contains source for the kernel debugger. Note that the kdb patch also changes other files throughout the kernel.

kernel. This directory contains core kernel routines such as process management, system call handling, module management, and so on.

lib.Some of the standard C library functions have counterparts in the kernel. The source can be found in this directory.

MAINTAINERS. This file lists the people who are responsible for various parts of the kernel.

mm. This directory contains all of the memory management code that is not specific to one architecture or another. The Linux page cache managment routines can be found in this directory.

net. All of the networking protocols (TCP, UDP, IP, etc.) are stored in this directory.

There are too many files and directories to decribe here. However, for readers interested in learning about filesystems, the include, fs, and mm directories are where most of the filesystem-related structures and routines can be found. There are also a few interesting files in the drivers/block directory for those wishing to look at the filesystem/driver interfaces in more detail.

Configuring the Kernel

There are several different methods of defining the configuration. The Linux kernel HOWTO should be consulted in addition to the notes described here. There are multiple copies of the HOWTO available across the World Wide Web. You can find it at the following Web site:

 	www.tldp.org/HOWTO/Kernel-HOWTO.html
 

 # cp /usr/src/linux-2.4.18-3/.config /usr/src/linux-2.4.18/.config
 

One method of defining the configuration is to run the following command for both the installed kernel and the new kernel source. For example, for Red Hat 7.3 run the following:

 # cd /usr/src/linux-2.4.18-
 # make menuconfig
 

 # cd /usr/src/linux-2.4.18
 # make menuconfig
 

Items are selected if noted by an asterisk. Loadable kernel modules are denoted by the letter "M." Instructions are available at the top of the screen to indicate how to select. Pressing Enter expands the menu to the next level. Pressing the Escape key takes you back up a level.

Once you have completed changing the configuration, a series of Escape key sequences will prompt you as to whether you wish to save and exit. Note that you do not need to save the configuration for the current kernel. This is particularly important if you have accidently made any changes. After saving the configuration and exiting the program, the following message appears:

 Saving your kernel configuration..
 
 *** End of Linux kernel configuration.
 *** Check the top-level Makefile for additional configuration.
 *** Next, you must run make dep
 

 # make dep
 # make clean
 

The next step, which is the longest, is to compile the kernel. This can be achieved by typing the following:


 # make bzImage
 ..
 objcopy -O binary -R .note -R .comment -S compressed/bvmlinux 
 compressed/bvmlinux.out
 tools/build -b bbootsect bsetup compressed/bvmlinux.out CURRENT 
 bzImage
 Root device is (3, 2)
 Boot sector 512 bytes.
 Setup is 2536 bytes.
 System is 1301 kB
 warning: kernel is too big for standalone boot from floppy
 make[1]: Leaving directory '/usr/src/linux-2.4.18/arch/i386/boot'
 

 # cp arch/i386/boot/bzImage /boot/linux.spate
 

Because many of the kernel components were probably selected to be kernel modules, they must be compiled and installed as follows:

 # make modules
 # make modules_install
 

 	/lib/modules/2.4.18
 

Installing and Booting the New Kernel

Consider the following lines taken from one specific /etc/lilo.conf file that was created as part of Red Hat 7.3 installation:

 image=/boot/vxlinuz-2.4.18-
 label=linux
 
 initrd=/boot/initrd-2.4.18-3.img
 read-only
 root=/dev/hda2
 

In order to bootstrap the new kernel, copy these lines to the end of the file and change both the image and label lines as follows:

 image=/boot/linux.spate
 label=linux.spate
 initrd=/boot/initrd-2.4.18-3.img
 read-only
 root=/dev/hda2
 

The final step is to run lilo to install information about the new kernel in the master boot record:

 # lilo
 

Using GRUB to Handle Bootstrap

Booting the New Kernel

 # uname -
 Linux x.y.com 2.4.18 #2 SMP Tue Jul 30 18:55:27 PDT 2002 i686 unknown
 

Installing Debugging Support

There are three main methods under which a filesystem or indeed any other part of the kernel can be debugged. The first approach involves using the kernel printk() command which is very similar to printf(). The second approach involves using a standalone debugger such as kdb whereby flow can be stopped by placing explicit breakpoints or by entering a special key sequence to enter the debugger. The third approach involves the use of two machines connected through a serial cable and over which gdb can be used for source level debugging.

The following sections describe each of these approaches. The amount of work to perform each task is considerably different with printk() being the simplest approach while the gdb approach involves more time to set up and an additional machine. For readers who wish to experiment and have access to all the available resources it is recommended that you start with printk() first, then move to kdb, and finally to gdb. The following sections assume some familiarity with debugging concepts.

The printk Approach to Debugging

Linux provides the printk() function for kernel/module writers to use. With the exception of the name change, it can be used in the same manner in which printf() can be called. One method employed when writing uxfs was to place a printk() at the start of each entry point to the filesystem. When typing various commands at the user prompt, it is then easy to see which functions in the filesystem are called.

Because Linux supports loadable modules, and the time to recompile and reload a module is in the order of seconds, this is the easiest way to watch how the filesystem works in practice and should be the method initially followed by anyone new to kernel development who wants to understand how the kernel works. To get a better idea of how the filesystem-related kernel functions work, printk() calls can be placed throughout the kernel, and various structures can be displayed.

Using the SGI kdb Debugger

The source code for kdb, which was developed by engineers at SGI (Silicon Graphics Inc), can be downloaded from the SGI Web site. The home page for kdb is as follows:

 http://oss.sgi.com/projects/kdb/
 

The README file in the download directory contains instructions on which files to download. This file should be consulted prior to downloading. Note that there may be several versions for the same kernel. The README file specifies how to interpret the version numbers of the patches.

There are two patch files to download. The first is common across all different machine architectures and the second is specific to the machine architecture on which youre running. After downloading the patches, they can be applied as follows:

 # cd /usr/src/linux-2.4.18
 # patch -p1 < ../kdb-v2.1-2.4.18-common-
 patching file kernel/sysctl.
 patching file kernel/ksyms.
 patching file kernel/Makefile
 patching file init/main.
 ..
 patching file Documentation/kdb/kdb_env.man
 patching file Documentation/kdb/kdb.mm
 
 
 
 patching file Documentation/kdb/kdb_bp.man
 patching file Documentation/kdb/slides
 # patch -p2 < ../kdb-v2.1-2.4.18-i386-
 patching file include/asm-i386/hw_irq.
 patching file include/asm-i386/keyboard.
 patching file include/asm-i386/ptrace.
 patching file arch/i386/vmlinux.lds
 ..
 patching file arch/i386/kdb/kdbasupport.
 patching file arch/i386/kdb/ansidecl.
 patching file arch/i386/kdb/bfd.
 patching file arch/i386/kdb/ChangeLog
 

Included with the kdb patch is documentation on how the debugger works, the commands that are available, and so on. The debugger can be entered by pressing the BREAKkey. The kdbprompt is then displayed as follows:

 Entering kdb (current=0xc03b0000,pid 0)on processor 0 due to Keyboard Entry
 [0]kdb> 
 

 bp.Set or display a breakpoint.
 bph.Set a hardware breakpoint.
 bc. Clear a breakpoint.
 bl. List the current breakpoints.
 bt.Display the stack backtrace for the current process.
 go. Exit the debugger and restart kernel execution.
 id.Disassemble instructions.
 md.Display the contents of the specified address.
 mds.Display memory symbolically.
 mm.Modify memory.
 reboot.Reboot the machine immediately.
 rd.Display the register contents.
 ss.Single step (instruction at a time)
 ssb.Single step the CPU until a branch is reached.
 

Source Level Debugging with gdb

 http://kgdb.sourceforge.net/
 

Connecting the Host and Target Machines

Serial ports support transmission rates from 110 baud up to 115,200 baud. The default baud rate for a serial port is 9,600. This is generally adequate for simple debugging although higher baud rates are preferred if a lot of information will be transmitted over the wire. This will certainly be the case when displaying multiple thread stacks.

Once the link is in place, the speed of the serial port on each machine must be identical. This can be verified on each machine as follows:

 # stty < /dev/ttyS0
 speed 9600 baud; line = 0;
 min = 0; time = 10;
 -brkint -icrnl -imaxbel
 -opost -onlcr
 -isig -icanon -iexten -echo -echoe -echok -echoctl -echoke
 

 # stty ispeed 9600 ospeed 9600 < /dev/ttyS0
 

 Host  					Target
 # cat /dev/ttyS0		# echo hello > /dev/ttyS0
 
 hello
 

Downloading the kgdb Patch

 # cd /usr/src/linux
 # patch -p1 < ../linux-2.4.18-kgdb-1.5.patch
 patching file Documentation/Configure.help
 patching file Documentation/i386/gdb-serial.txt
 patching file Makefile
 patching file arch/i386/Makefile
 patching file arch/i386/config.in
 patching file arch/i386/kernel/Makefile
 ..
 patching file kernel/ksyms.
 patching file kernel/sched.
 

 KGDB: Remote (serial) kernel debugging with gdb (NEW)
 

 # make dep
 # make clean
 # make bzImage
 

Installing the kgdb-Modified Kernel

 image=/boot/linux.gdb
 label=linux.gdb
 initrd=/boot/initrd-2.4.18-3.img
 read-only
 root=/dev/hda2
 append="gdb gdbttyS=0 gdbbaud=9600"
 

 Waiting for connection from remote gdb..
 

 # gdb
 GNU gdb Red Hat Linux (5.1.90CVS-5)
 Copyright 2002 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you 
 are welcome to change it and/or distribute copies of it under certain 
 conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB. Type "show warranty" for 
 details.
 This GDB was configured as "i386-redhat-linux"
 (gdb) target remote /dev/ttyS0
 Remote debugging using /dev/ttyS0
 0xc011323d in ?? (
 (gdb) 
 Continuing.
 PCI: PCI BIOS revision 2.10 entry at 0xfbfee, last bus=
 PCI: Using configuration type 
 ..
 

 Program received signal SIGTRAP, Trace/breakpoint trap.
 0xc011323d in ?? (
 (gdb) symbol-file /usr/src/linux/vmlinux
 Reading symbols from /usr/src/linux/vmlinux...done.
 

 gdb and Module Interactions 
 

Before running loadmodule, the GDBSCRIPTS variable, located at the top of the script, must be altered to point to a directory where it can install a script for use with gdb. As an example:

 GDBSCRIPTS=/home/spate/uxfs/tools/gdbscripts
 

 # loadmodule target-machine ../kern/uxfs
 Copying ../kern/uxfs to linux
 Loading module ../kern/uxfs
 Generating script /home/spate/uxfs/tools/gdbscripts/loadlinuxuxfs
 

 Program received signal SIGTRAP, Trace/breakpoint trap.
 breakpoint () at gdbstub.c:1177
 1177 
 (gdb) so /home/spate/uxfs/tools/gdbscripts/loadlinuxuxfs
 add symbol table from file "/home/spate/uxfs/kern/uxfs" at
 
 
 .text_addr = 0xd0854060
 .rodata_addr = 0xd0855c60
 __ksymtab_addr = 0xd085618c
 __archdata_addr = 0xd08562b0
 __kallsyms_addr = 0xd08562b0
 .data_addr = 0xd08568c0
 .bss_addr = 0xd0856a60
 

Building the uxfs Filesystem

 # gunzip uxfs.tar.gz
 # tar xvf uxfs.tar
 # ls
 uxfs.tar uxfs
 # ls uxfs
 cmds kern
 

 # make fsdb
 cc fsdb.c -o fsdb 
 # make fsdb 
 cc fsdb.c -o fsdb 
 
 The commands can then be used. 
 The kernel makefileis relatively straightforward as follows: 
 KERNELDIR = /usr/src/linux 
 include $(KERNELDIR)/.config 
 FLAGS = -D__KERNEL__ -DMODULE $(VERCFLAGS) 
 GLOBAL_CFLAGS = -g -I$(KERNELDIR)/include $(FLAGS) 
 M_OBJS = ux_dir.o ux_alloc.o ux_file.o ux_inode.o 
 M_TARGET = uxfs 
 SRCS = $(M_OBJS:.o=.c) 
 CFLAGS = $(GLOBAL_CFLAGS) $(EXTRA_CFLAGS) 
 $(M_TARGET) : $(M_OBJS) 
 ld -r -o $@ $(M_OBJS) 
 $(M_OBJS) : %.o : %.c 
 $(CC) -c $(CFLAGS) -o $@ $< 
 
 
 all: uxfs
 
 
 clean:
 rm -f $(M_OBJS) $(M_TARGET)
 

 # make uxfs
 cc -c -g -I/usr/src/linux/include -D__KERNEL__ -DMODULE -o ux_dir.o ux_dir.
 cc -c -g -I/usr/src/linux/include -D__KERNEL__ -DMODULE -o ux_alloc.
 ux_alloc.
 cc -c -g -I/usr/src/linux/include -D__KERNEL__ -DMODULE -o ux_file.o ux_file.
 cc -c -g -I/usr/src/linux/include -D__KERNEL__ -DMODULE -o ux_inode.
 ux_inode.
 ld -r -o uxfs ux_dir.o ux_alloc.o ux_file.o ux_inode.
 

Creating a uxfs Filesystem

The code for mkfs can be found on lines 104 to 262. For uxfs, it is a fairly simple program. As with the kernel, it uses various structure definitions and information from ux_fs.h including superblock structural information, inode formats, directory entries and various filesystem boundaries such as the maximum number of blocks and inodes.

Before the filesystem is implemented, it is important to verify the information that mkfs writes to disk. Thus, the next program to write is fsdb, which can read back and display various superblock and inode information. The fsdbcommand (lines 264 to 393) is very simple. It accepts two commands that allow the superblock or a specified inode to be displayed.

The first task is to read the superblock into memory (lines 365 to 369), validate it, and keep in in memory for the duration of the program. From here, it can access any information it needs to about inodes or data blocks.

The remainder of the main() loop involves reading commands and then calling additional routines. For now, only the superblock or an inode can be displayed. By entering 'q', the program will exit.

The following output from fsdb shows the two commands being run on a newly created filesystem:

 # ./mkfs /dev/fd0
 # ./fsdb /dev/fd0
 uxfsdb > 
 
 Superblock contents: 
 s_magic = 0x58494e55 
 s_mod = UX_FSCLEAN 
 s_nifree = 28 
 s_nbfree = 468
 
 uxfsdb > i2
 
 inode number 
 i_mode = 41ed 
 i_nlink = 
 i_atime = Wed Aug 21 09:55:16 2002 
 i_mtime = Wed Aug 21 09:55:16 2002 
 i_ctime = Wed Aug 21 09:55:16 2002 
 i_uid = 
 
 
 
 i_gid = 
 i_size = 512 
 i_blocks = 
 i_addr[ 0] = 50 i_addr[ 1] = 0 i_addr[ 2] = 0 i_addr[ 3] = 
 i_addr[ 4] = 0 i_addr[ 5] = 0 i_addr[ 6] = 0 i_addr[ 7] = 
 i_addr[ 8] = 0 i_addr[ 9] = 0 i_addr[10] = 0 i_addr[11] = 
 i_addr[12] = 0 i_addr[13] = 0 i_addr[14] = 0 i_addr[15] = 
 
 
  Directory entries: 
 inum[ 2],name[.
 inum[ 2],name[..
 inum[ 3],name[lost+found]
 
 
 uxfsdb > q
 

Module Initialization and Deinitialization

The various components that are applicable to uxfs are shown in ux_inode.c on lines 1304 to 1317. The module_init() call specifies the function to be run when the module is loaded while the module_exit() function specifies the function to be called when the module is unloaded. Both of these functions perform little work other than registering and unregistering the filesystem driver respectively. The DECLARE_FSTYPE_DEV()macro is shown below:

 #define DECLARE_FSTYPE(var,type,read,flags) 
 
 
 struct file_system_type var = { 
 name: type, 
 read_super: read, 
 fs_flags: flags, 
 owner: THIS_MODULE, 
 }
 
 #define DECLARE_FSTYPE_DEV(var,type,read) 
 DECLARE_FSTYPE(var,type,read,FS_REQUIRES_DEV)
 

The rmmod command is used to remove a kernel module. If there are still filesystems mounted, the removal will fail; otherwise the kernel calls the module exit function, which in the case of uxfs, is the exit_uxfs_fs() function. The only action to perform is to call unregister_filesystem().

Testing the New Filesystem

 # ./mkfs /dev/fd0
 # insmod ./uxfs
 # lsmod
 Module Size Used by Not tainted
 uxfs 8608 0 (unused)
 ext3 71968 2 (autoclean)
 jbd 66208 2 (autoclean) [ext3]
 # mount -t uxfs /dev/fd0 /mnt
 # mount
 /dev/hda2 on / type ext3 (rw)
 none on /proc type proc (rw)
 /dev/hda1 on /boot type ext3 (rw)
 none on /dev/pts type devpts (rw,gid=5,mode=620)
 /dev/hda5 on /home type ext3 (rw)
 none on /dev/shm type tmpfs (rw)
 /dev/fd0 on /mnt type uxfs (rw)
 # rmmod uxfs
 uxfs: Device or resource busy
 # umount /mnt
 # rmmod uxfs
 # lsmod
 Module Size Used by Not tainted
 ext3 71968 2 (autoclean)
 jbd 66208 2 (autoclean) [ext3]
 

 ux_read_super(struct super_block *s, void *data, int silent)
 

 (gdb) b ux_read_super
 Breakpoint 1 at 0xd08557ca: file ux_inode.c, line 237.
 (gdb) 
 Continuing.
 

 # mount -f uxfs /dev/fd0 /mnt
 
 Breakpoint 1, ux_read_super (s=0xcf15a400, data=0x0, silent=0)
 
 
  at ux_inode.c:237
 237 dev = s->s_dev;
 (gdb) list
 232 struct ux_fs *fs;
 233 struct buffer_head *bh;
 234 struct inode *inode;
 235 kdev_t dev;
 236
 237 dev = s->s_dev;
 238 set_blocksize(dev, UX_BSIZE)
 239 s->s_blocksize = UX_BSIZE;
 240 s->s_blocksize_bits = UX_BSIZE_BITS;
 241
 

 (gdb) bt
 #0 ux_read_super (s=0xcf15a400, data=0x0, silent=0) at ux_inode.c:237
 #1 0xc0143868 in get_sb_bdev (fs_type=0xd0856a44, 
 
 
  dev_name=0xccfe8000 "/dev/fd0", flags=0, data=0x0) at super.c:697
 #2 0xc0143d2d in do_kern_mount (type=0xccfe9000 "uxfs", flags=0, 
 name=0xccfe8000 "/dev/fd0", data=0x0) at super.c:879
 
 
 #
 0xc0156ff1 in do_add_mount (nd=0xcd011f5c, type=0xccfe9000 "uxfs"
 flags=0, mnt_flags=0, name=0xccfe8000 "/dev/fd0", data=0x0) 
 at namespace.c:630
 
 
 #4 0xc01572b7 in do_mount (dev_name=0xccfe8000 "/dev/fd0"
 dir_name=0xcf80f000 "/mnt", type_page=0xccfe9000 "uxfs"
 flags=3236757504, data_page=0x0) at namespace.c:746
 
 
 #
 0xc015737f in sys_mount (dev_name=0x805b418 "/dev/fd0"
 dir_name=0x805b428 "/mnt", type=0x805b438 "uxfs", flags=3236757504, 
 data=0x0) at namespace.c:779
 
 
 #
 0xc010730b in system_call ()
 

 
 (gdb) print *(struct super_block *)0xcf15a400
 
 
 $1 = {s_list = {next = 0xc0293840, prev = 0xcf6df400}, s_dev = 512, 
 s_blocksize = 0, s_blocksize_bits = 0 \0, s_dirt = 0 \0
 s_maxbytes = 2147483647, s_type = 0xd0856a44, s_op = 0x0, dq_op = 0x0, 
 s_flags = 0, s_magic = 0, s_root = 0x0, s_umount = {count = -65535, 
 
 
  wait_lock = {lock = 1}, wait_list = {next = 0xcf15a43c, 
 
 
 
 prev = 0xcf15a43c}}, s_lock = {count = {counter = 0}, sleepers = 0, 
 
 
  wait = {lock = {lock = 1}, task_list = {next = 0xcf15a450, 
 
 
 prev = 0xcf15a450}}}, s_count = 1073741824, s_active = {counter = 1}
 
 
  s_dirty = 0,
 
 
  ..
 

 bh = sb_bread(sb, block)
 bh = bread(sb->s_dev, block, sb->s_blocksize)
 

 usb = (struct ux_superblock *)bh->b_data;
 

 struct super_block *sb = inode->i_sb;
 struct ux_fs *fs = (struct ux_fs *)sb->s_private;
 struct ux_superblock *usb = fs->u_sb;
 

 (gdb) b ux_read_inode
 Breakpoint 2 at 0xd0855312: file ux_inode.c, line 54.
 (gdb) 
 Continuing.
 

 Breakpoint 2, ux_read_inode (inode=0xcd235460) at ux_inode.c:54
 
 unsigned long ino = inode->i_ino;
 
 (gdb) list
 
 
 49 void
 50 ux_read_inode(struct inode *inode)
 51 { 
 52 struct buffer_head *bh; 
 53 struct ux_inode *di; 
 54 unsigned long ino = inode->i_ino; 
 55 int block; 
 56 
 57 
 if (ino < UX_ROOT_INO || ino > UX_MAXFILES) {
 58 printk("uxfs: Bad inode number %lu\n", ino)
 

 (2) passed to iget(). 
 
 (gdb) bt
 #0 ux_read_inode (inode=0xcd235460) at ux_inode.c:54
 #1 0xc015411a in get_new_inode (sb=0xcf15a400, ino=2, head=0xcfda3820, 
 
 
  find_actor=0, opaque=0x0) at inode.c:871
 #2 0xc015439a in iget4 (sb=0xcf15a400, ino=2, find_actor=0, opaque=0x0) 
 at inode.c:984
 #3 0xd0855bfb in iget (sb=0xcf15a400, ino=2) 
 at /usr/src/linux/include/linux/fs.h:1328
 #4 0xd08558c3 in ux_read_super (s=0xcf15a400, data=0x0, silent=0) 
 at ux_inode.c:272
 #5 0xc0143868 in get_sb_bdev (fs_type=0xd0856a44, 
 
 
  dev_name=0xccf35000 "/dev/fd0", flags=0, data=0x0) at super.c:697
 #6 0xc0143d2d in do_kern_mount (type=0xccf36000 "uxfs", flags=0, 
 ..
 

 (gdb) print *(struct inode *)0xcd235460
 $2 = {i_hash = {next = 0xce2c7400, prev = 0xcfda3820}, i_list = 
 
 
 next = 0xcf7aeba8, prev = 0xc0293d84}, i_dentry = {next = 0xcd235470, 
 prev = 0xcd235470}, i_dirty_buffers = {next = 0xcd235478, 
 prev = 0xcd235478}, i_dirty_data_buffers = {next = 0xcd235480, 
 prev = 0xcd235480}, i_ino = 2, i_count = {counter = 1}, i_dev = 512, 
 i_mode = 49663, i_nlink = 1, i_uid = 0, i_gid = 0, 
 i_rdev = 512, i_size = 0, 
 

 mark_buffer_dirty(bh)
 

 # ./mkfs /dev/fd0
 # mount -t uxfs /dev/fd0 /mnt
 # touch /mnt/file
 

 Entering kdb (current=0xcbe20000, pid 1320) on processor 0 due to
 Keyboard Entry[0]kdb> bp ux_write_super
 Instruction(i) BP #1 at 0xd08ab788 ([uxfs]ux_write_super)
  is enabled globally adjust 
 

 Entering kdb (current=0xc1464000, pid 7) on processor 0 due to Breakpoint
 @ 0xd08ab788
 [0]kdb> bt
 
 
  EBP EIP Function(args)
 0xc1465fc4 0xd08ab788 [uxfs]ux_write_super (0xcc53b400, 0xc1464000) 
 uxfs .text 0xd08aa060 0xd08ab788 0xd08ab7c4 
 0xc014b242 sync_supers+0x142 (0x0, 0xc1464000)
 
 
 kernel .text 0xc0100000 0xc014b100 0xc014b2c0
 0xc1465fd4 0xc0149bd6 sync_old_buffers+0x66 (0xc1464000, 0x10f00, 
 0xcffe5f9c, 0xc0105000)
 
 
 kernel .text 0xc0100000 0xc0149b70 0xc0149cf0
 0xc1465fec 0xc014a223 kupdate+0x273
 kernel .text 0xc0100000 0xc0149fb0 0xc014a230
 
 
  0xc01057c6 kernel_thread+0x26
 kernel .text 
 0xc0100000 0xc01057a0 0xc01057e0
 

  if (sb->s_root && sb->s_dirt) 
 if (sb->s_op && sb->s_op->write_super) 
 sb->s_op->write_super(sb)
 

 Breakpoint 4, ux_put_super (s=0xcede4c00) at ux_inode.c:167
 
 167 struct ux_fs *fs = (struct ux_fs *)s->s_private;
 
 (gdb) bt
 #0 ux_put_super (s=0xcede4c00) at ux_inode.c:167
 #1 0xc0143b32 in kill_super (sb=0xcede4c00) at super.c:800
 #2 0xc01481db in path_release (nd=0xc9da1f80)
 at /usr/src/linux-2.4.18/include/linux/mount.h:50
 #3 0xc0156931 in sys_umount (name=0x8053d28 "/mnt", flags=0)
 	 at namespace.c:395
 #4 0xc015694e in sys_oldumount (name=0x8053d28 "/mnt"
 	at namespace.c:406
 #5 0xc010730b in system_call (
 

 struct inode_operations ux_dir_inops = {
 create: ux_create, 
 lookup: ux_lookup, 
 mkdir: ux_mkdir, 
 rmdir: ux_rmdir, 
 link: ux_link, 
 unlink: ux_unlink, 
 }; 
 
 struct file_operations ux_dir_operations = {
 read: generic_read_dir,
 readdir: ux_readdir,
 fsync: file_fsync,
 }
 

 int getdents(unsigned int fd, struct dirent *dirp, unsigned int count)
 

 struct dirent
 { 
 long d_ino; /* inode number */ 
 off_t d_off; /* offset to next dirent */ 
 unsigned short d_reclen; /* length of this dirent */ 
 char d_name [NAME_MAX+1]; /* file name (null-terminated) */ 
 } 
 

 int
 ux_readdir(struct file *filp, void *dirent, filldir_t filldir)
 

 pos = filp->f_pos;
 blk = (pos + 1) / UX+BSIZE;
 blk = uip->iaddr[blk]
 

 filp->f_pos += sizeof(struct ux_dirent)
 

 inum = ux_find_entry(dip, (char *)dentry->d_name.name)
 

 (gdb) b ux_lookup
 Breakpoint 8 at 0xd0854b32: file ux_dir.c, line 367.
 (gdb) b ux_readdir
 Breakpoint 9 at 0xd0854350
 (gdb) b ux_read_inode
 Breakpoint 10 at 0xd0855312: file ux_inode.c, line 54.
 

 # mount -f uxfs /dev/fd0 /mnt
 
 
 Breakpoint 10, ux_read_inode (inode=0xcd235280) at ux_inode.c:54
 54 
 
 
 unsigned long ino = inode->i_ino;
 (gdb) p inode->i_ino
 $19 = 2
 

 (gdb) bt
 #0 ux_read_inode (inode=0xcd235280) at ux_inode.c:54
 #1 0xc015411a in get_new_inode (sb=0xcf15a400, ino=2, head=0xcfda3820, 
 
 
  find_actor=0, opaque=0x0) at inode.c:871
 #2 0xc015439a in iget4 (sb=0xcf15a400, ino=2, find_actor=0, opaque=0x0) 
 at inode.c:984
 #3 0xd0855bfb in iget (sb=0xcf15a400, ino=2) 
 at /usr/src/linux/include/linux/fs.h:1328
 #4 0xd08558c3 in ux_read_super (s=0xcf15a400, data=0x0, silent=0) 
 at ux_inode.c:272
 ..
 

 # ls /mnt
 
 
 Breakpoint 9, 0xd0854350 in ux_readdir (filp=0xcd39cc60, 
 dirent=0xccf0dfa0, filldir=0xc014dab0 < filldir64>
 

 (gdb) p ((struct inode *)(filp->f_dentry->d_inode))->i_ino
 $20 = 
 

 (gdb) bt
 #0 0xd0854350 in ux_readdir (filp=0xcd39cc60, dirent=0xccf0dfa0, 
 
 
  filldir=0xc014dab0 
 #1 0xc014d64e in vfs_readdir (file=0xcd39cc60, filler=0xc014dab0 
 
 
 
  buf=0xccf0dfa0) at readdir.c:27
 #2 0xc014dc2d in sys_getdents64 (fd=3, dirent=0x8058730, count=512) 
 at readdir.c:311
 #3 0xc010730b in system_call (
 

 pos = filp->f_pos;
 

 Breakpoint 8, ux_lookup (dip=0xcd235280, dentry=0xcd1e9ae0) at
 ux_dir.c:367
 367 struct ux_inode *uip = (struct ux_inode *)
 

 (gdb) p dentry->d_name
 $23 = {name = 0xcd1e9b3c "lost+found", len = 10, hash = 4225228667}
 

 Breakpoint 10, ux_read_inode (inode=0xcf7aeba0) at ux_inode.c:54
 54 unsigned long ino = inode->i_ino;
 (gdb) p inode->i_ino
 $24 = 3
 

 (gdb) bt
 #0 ux_read_inode (inode=0xcf7aeba0) at ux_inode.c:54
 #1 0xc015411a in get_new_inode (sb=0xcf15a400, ino=3, head=0xcfda3828, 
 
 
  find_actor=0, opaque=0x0) at inode.c:871
 #2 0xc015439a in iget4 (sb=0xcf15a400, ino=3, find_actor=0, opaque=0x0) 
 at inode.c:984
 #3 0xd0854e73 in iget (sb=0xcf15a400, ino=3) 
 at /usr/src/linux/include/linux/fs.h:1328
 #4 0xd0854b93 in ux_lookup (dip=0xcd235280, dentry=0xcd1e9ae0) 
 at ux_dir.c:379
 #5 0xc01482c0 in real_lookup (parent=0xcd1e9160, 
 name=0xccf0df5c, flags=0) 
 at namei.c:305
 #6 0xc0148ba4 in link_path_walk (name=0xcf80f00f "", nd=0xccf0df98) 
 at namei.c:590
 #7 0xc014943a in __user_walk (name=0x0, flags=8, nd=0xccf0df98) 
 at namei.c:841
 
 
 #8 0xc0145877 in sys_lstat64 (filename=0xbffff950 "/mnt/lost+found"
 statbuf=0x805597c, flags=1108542220) at stat.c:352
 #9 0xc010730b in system_call (
 

 Breakpoint 8, ux_lookup (dip=0xcd235280,dentry=0xcd1e90e0) at ux_dir.c:367
 367 struct ux_inode *uip = (struct ux_inode *
 (gdb) p dentry->d_name
 $26 = {name = 0xcd1e913c "passwd", len = 6, hash = 3467704878}
 

 Breakpoint 10, ux_read_inode (inode=0xcd0c4c00) at ux_inode.c:54
 54 unsigned long ino = inode->i_ino;
 (gdb) p inode->i_ino
 $27 = 4
 

 void
 ux_read_inode(struct inode *inode)
 

 block = UX_INODE_BLOCK + ino;
 bh = sb_bread(inode->i_sb, block)
 

 #define i_private u_generic_ip
 

 if (di->i_mode & S_IFDIR) 
 inode->i_mode |= S_IFDIR;
 inode->i_op = &ux_dir_inops;
 inode->i_fop = &ux_dir_operations;
 } else if (di->i_mode & S_IFREG) 
 inode->i_mode |= S_IFREG;
 inode->i_op = &ux_file_inops;
 inode->i_fop = &ux_file_operations;
 inode->i_mapping->a_ops = &ux_aops;
 

 mark_inode_dirty(inode)
 

 [0]kdb> bp ux_write_inode
 

 Instruction(i) BP #0 at 0xd08cd4c8 ([uxfs]ux_write_inode)
  is enabled globally adjust 
 Entering kdb (current=0xc1464000, pid 7) on processor 0 due to Breakpoint 
 @ 0xd08cd4c8
 [0]kdb> bt
  EBP EIP Function(args)
 0xc1465fc8 0xd08cd4c8 [uxfs]ux_write_inode (0xc77f962c, 0x0, 0xcf9a8868,
 0xcf9a8800, 0xc1465fd4)
 uxfs .text 0xd08cc060 0xd08cd4c8 0xd08cd5c0
  0xc015d738 sync_unlocked_inodes+0x1d8 (0xc1464000)
  kernel .text 0xc0100000 0xc015d560 
 0xc015d8e0
 0xc1465fd4 0xc0149bc8 sync_old_buffers+0x58 (0xc1464000, 0x10f00,
 0xcffe5f9c, 0xc0105000)
  kernel .text 0xc0100000 0xc0149b70 
 0xc0149cf0
 0xc1465fec 0xc014a223 kupdate+0x273
  kernel .text 0xc0100000 0xc0149fb0 
 0xc014a230
  0xc01057c6 kernel_thread+0x26
 kernel .text 0xc0100000 0xc01057a0
 0xc01057e0
 

 # touch 
 # touch 
 # ln A 
 

 # rm 
 # rm 
 

 [0]kdb> bp ux_write_inode
 Instruction(i) BP #0 at 0xd08cd4c8 ([uxfs]ux_write_inode) 
 is enabled globally adjust 
 [0]kdb> go
 

 # touch /mnt/file
 # rm /mnt/file
 

 Entering kdb (current=0xcaae6000, pid 1398) 
 on processor 0 due to Breakpoint @ 0xd08bc5c0
 [0]kdb> bt
 EBP EIP Function(args)
 0xcab81f34 0xd08bc5c0 [uxfs]ux_delete_inode (0xcaad2824, 0xcaad2824,
 0xcac4d484, 0xcabc6e0c)
 uxfs .text 0xd08bb060 0xd08bc5c0 0xd08bc6b4 
 0xc015f1f4 iput+0x114 (0xcaad2824, 0xcac4d4e0, 0xcab81f98,
 0xcaad2824, 0xcac4d484)
 kernel .text 0xc0100000 0xc015f0e0 0xc015f3a0
 0xcab81f58 0xc015c466 d_delete+0xd6 (0xcac4d484, 0xcac4d56c, 0xcab81f98,
 0x0, 0xcabc6e0c)
 kernel .text 0xc0100000 0xc015c390 0xc015c590
 0xcab81f80 0xc01537a8 vfs_unlink+0x1e8 (0xcabc6e0c, 0xcac4d484,
 0xcac4d56c, 0xcffefcf8, 0xcea16005)
 kernel .text 0xc0100000 0xc01535c0 0xc01537e0
 0xcab81fbc 0xc0153878 sys_unlink+0x98 (0xbffffc50, 0x2, 0x0, 
 0xbffffc50, 0x0)
 kernel .text 0xc0100000 0xc01537e0 0xc01538e0 
 0xc01077cb system_call+0x33
 kernel .text 0xc0100000 0xc0107798 0xc01077d0
 

 lstat64("foo", 0xbffff8a0) = -1 ENOENT (No such file or directory)
 stat64("file", {st_mode=S_IFREG|0644, st_size=0, ...}) = 
 open("file", O_RDONLY|O_LARGEFILE) = 
 open("foo", O_WRONLY|O_CREAT|O_LARGEFILE, 0100644) = 4
 

 Breakpoint 5, ux_lookup (dip=0xcd73cba0, dentry=0xcb5ed3a0) 
 at ux_dir.c:367
 367 struct ux_inode *uip = (struct ux_inode *
 (gdb) bt
 #0 ux_lookup (dip=0xcd73cba0, dentry=0xcb5ed3a0) at ux_dir.c:367
 #1 0xc01482c0 in real_lookup (parent=0xcb5ed320, name=0xc97ebf5c,
 
 
 flags=0) 
 at namei.c:305
 #2 0xc0148ba4 in link_path_walk (name=0xcb0f700b "", nd=0xc97ebf98) 
 at namei.c:590
 #3 0xc014943a in __user_walk 
 name=0xd0856920 "\220D\205,K\205AK\205< L\205"
 flags=9, nd=0xc97ebf98) 
 at namei.c:841
 #4 0xc0145807 in sys_stat64 (filename=0x8054788 "file"
 statbuf=0xbffff720, flags=1108542220) 
 at stat.c:337
 #5 0xc010730b in system_call (
 

 Breakpoint 6, 0xd0854494 in ux_create
 
 (dip=0xcd73cba0, dentry=0xcb5ed3a0, mode=33188)
 (gdb) bt
 #0 0xd0854494 in ux_create (dip=0xcd73cba0, dentry=0xcb5ed3a0, 
 
 
 mode=33188)
 #1 0xc014958f in vfs_create (dir=0xcd73cba0, dentry=0xcb5ed3a0, 
 mode=33188) 
 at namei.c:958
 #2 0xc014973c in open_namei (pathname=0xcb0f7000 "foo"
 flag=32834, 
 mode=33188, nd=0xc97ebf74) at namei.c:1034
 #3 0xc013cd67 in filp_open (filename=0xcb0f7000 "foo"
 flags=32833, 
 mode=33188) at open.c:644
 #4 0xc013d0d0 in sys_open (filename=0x8054788 "foo"
 flags=32833, mode=33188) 
 at open.c:788
 #5 0xc010730b in system_call (
 

 int
 ux_link(struct dentry *old, struct inode *dip, struct dentry *new)
 

 $ ln filea fileb
 

 Breakpoint 11, ux_link (old=0xcf2fe740, dip=0xcf23a240, new=0xcf2fe7c0)
  at ux_dir.c:395
 395 
 (gdb) bt
 #0 ux_link (old=0xcf2fe740, dip=0xcf23a240, new=0xcf2fe7c0) 
 at ux_dir.c:395
 #1 0xc014adc4 in vfs_link (old_dentry=0xcf2fe740, dir=0xcf23a240, 
  new_dentry=0xcf2fe7c0) at namei.c:1613
 #2 0xc014aef0 in sys_link (oldname=0xbffffc20 "filea"
  newname=0xbffffc26 "fileb") at namei.c:1662
 #3 0xc010730b in system_call (
 The gdb command can be used to display the arguments passed to ux_link() 
 as follows: 
 (gdb) p new
 $9 = (struct dentry *) 0xcf2fe7c0
 (gdb) p *old
 $10 = {d_count = {counter = 1}, d_flags = 0, d_inode = 0xcd138260, 
 d_parent = 0xcb5ed920, d_hash = {next = 0xc2701750, prev = 0xcfde6168}
  d_lru = {next = 0xcf2fe758, prev = 0xcf2fe758}, d_child = 
  next = 0xcb5ed948, prev = 0xcf2fe7e0}, d_subdirs = {next 
 0xcf2fe768, 
 prev = 0xcf2fe768}, d_alias = {next = 0xcd138270, prev = 0xcd138270}
  d_mounted = 0, d_name = {name = 0xcf2fe79c "filea", len = 5, 
  hash = 291007618}, d_time = 0, d_op = 0x0, d_sb = 0xcede4c00, 
  d_vfs_flags = 8, d_fsdata = 0x0, d_iname = "filea\0g\0\0\0\0\0\0\0\0"
 (gdb) p old->d_name.name
 $11 = (unsigned char *) 0xcf2fe79c "filea"
 (gdb) p new->d_name.name
 $12 = (unsigned char *) 0xcf2fe81c "fileb"
 

 Breakpoint 5, ux_lookup (dip=0xcd73cba0, dentry=0xcb5ed420) 
 at ux_dir.c:367
 367 struct ux_inode *uip = (struct ux_inode *
 (gdb) bt
 #0 ux_lookup (dip=0xcd73cba0, dentry=0xcb5ed420) at ux_dir.c:367
 #1 0xc01492f2 in lookup_hash (name=0xc97ebf98, base=0xcb5ed320) 
 
 at namei.c:781
 #2 0xc0149cd1 in lookup_create (nd=0xc97ebf90, is_dir=1) 
 at namei.c:1206
 #3 0xc014a251 in sys_mkdir (pathname=0xbffffc1c "/mnt/dir", mode=511) 
 at namei.c:1332
 #4 0xc010730b in system_call (
 

 Breakpoint 7, 0xd08546d0 in ux_mkdir (dip=0xcd73cba0, dentry=0xcb5ed420,
  mode=493)
 (gdb) bt
 #0 0xd08546d0 in ux_mkdir (dip=0xcd73cba0, dentry=0xcb5ed420, mode=493)
 #1 0xc014a197 in vfs_mkdir (dir=0xcd73cba0, dentry=0xcb5ed420,
 mode=493) 
 at namei.c:1307
 #2 0xc014a282 in sys_mkdir (pathname=0xbffffc1c "/mnt/dir", mode=511) 
 at namei.c:1336
 #3 0xc010730b in system_call (
 

 Breakpoint 8, 0xd0854a0c in ux_rmdir (dip=0xcd73cba0, dentry=0xcb5ed420)
 (gdb) bt
 #0 0xd0854a0c in ux_rmdir (dip=0xcd73cba0, dentry=0xcb5ed420)
 #1 0xc014a551 in vfs_rmdir (dir=0xcd73cba0, dentry=0xcb5ed420) 
 at namei.c:1397
 #2 0xc014a696 in sys_rmdir (pathname=0xbffffc1c "/mnt/dir"
 at namei.c:1443
 #3 0xc010730b in system_call (
 

 struct file_operations ux_file_operations = {
  llseek: generic_file_llseek, 
 read: generic_file_read, 
 write: generic_file_write, 
 mmap: generic_file_mmap, 
 }; 
 

 struct address_space_operations ux_aops = {
 readpage: ux_readpage,
 writepage: ux_writepage, 
 sync_page: block_sync_page, 
 prepare_write: ux_prepare_write, 
 commit_write: generic_commit_write, 
 bmap: ux_bmap, 
 }; 
 

 int
 ux_readpage(struct file *file, struct page *page)
 {
  return block_read_full_page(page, ux_get_block)
 }
 

 int
 ux_get_block(struct inode *inode, long block, 
 struct buffer_head *bh_result, int create)
 

 bh_result->b_dev = inode->i_dev;
 bh_result->b_blocknr = uip->i_addr[block]
 

 # ls -l /mnt/passwd
 -rw-r--r--1 root root 1203 Jul 24 07:51 /etc/passwd
 

 Breakpoint 1, ux_get_block (inode=0xcf23a420, 
 block=0, bh_result=0xc94f4740, create=0) 
 at ux_file.c:21
 21 struct super_block *sb = inode->i_sb;
 (gdb) bt
 #0 ux_get_block (inode=0xcf23a420, block=0, bh_result=0xc94f4740,
 create=0) 
 at ux_file.c:21
 #1 0xc0140b1f in block_read_full_page (page=0xc1250fc0, 
 get_block=0xd0855094 < ux_get_block>) at buffer.c:1781
 #2 0xd08551ba in ux_readpage (file=0xcd1c9360, page=0xc1250fc0)
  at ux_file.c:67
 #3 0xc012e773 in do_generic_file_read (filp=0xcd1c9360, 
 ppos=0xcd1c9380, 
  desc=0xc96d1f5c, actor=0xc012eaf0 < file_read_actor>
 at filemap.c:1401
 #4 0xc012ec72 in generic_file_read (filp=0xcd1c9360, buf=0x804eb28 ""
 count=4096, ppos=0xcd1c9380) at filemap.c:1594
 #5 0xc013d7c8 in sys_read (fd=3, buf=0x804eb28 "", count=4096) 
 at read_write.c:162
 #6 0xc010730b in system_call (
 

 Breakpoint 1, ux_get_block (inode=0xcf23a420, 
 block=1, bh_result=0xc94f46e0, create=0) 
 at ux_file.c:21
 21 
 
 
 struct super_block *sb = inode->i_sb;
 (gdb) bt
 #0 ux_get_block (inode=0xcf23a420, block=1, 
 
 
 bh_result=0xc94f46e0, create=0) 
 at ux_file.c:21
 
 
 
 #1 0xc0140b1f in block_read_full_page (page=0xc1250fc0, 
 ..
 

 (gdb) bt
 #0 ux_get_block (inode=0xcf23a420, block=2, 
 bh_result=0xc94f4680, create=0) 
 at ux_file.c:21
 #1 0xc0140b1f in block_read_full_page (page=0xc1250fc0, 
 get_block=0xd0855094 ) at buffer.c:1781
 #2 0xd08551ba in ux_readpage (file=0xcd1c9360, page=0xc1250fc0) 
 at ux_file.c:67
 

 # ls -l /etc/passwd
 -rw-r--r--1 root root 1336 Jul 24 14:28 /etc/passwd
 # cp /etc/passwd /mnt
 

 Breakpoint 1, ux_get_block (inode=0xcd710440, 
 block=0, bh_result=0xc96b72a0, create=1) 
 at ux_file.c:21
 struct super_block *sb = inode->i_sb;
 (gdb) bt
 #0 ux_get_block (inode=0xcd710440, block=0, 
 bh_result=0xc96b72a0, create=1) 
 at ux_file.c:21
 #1 0xc014074b in __block_prepare_write (inode=0xcd710440, 
 page=0xc125e640, from=0, to=1024,
 get_block=0xd0855094 < ux_get_block>
 at buffer.c:1641
 #2 0xc0141071 in block_prepare_write (page=0xc125e640, from=0, to=1024, 
 get_block=0xd0855094 < ux_get_block>) at buffer.c:1960
 #3 0xd08551dd in ux_prepare_write (file=0xcd1c9160, page=0xc125e640,
 from=0, to=1024) 
 at ux_file.c:74
 #4 0xc013085f in generic_file_write (file=0xcd1c9160, 
 buf=0xbffff160 
 "root:x:0:0:root:/root:/bin/bash\nbin:x:1:1:bin:/bin:/sbin/nologin\ndaem
 on:x:2:2:daemon:/sbin:/sbin/nologin\nadm:x:3:4:adm:/var/adm:/sbin/nologi
 n\nlp:x:4:7:lp:/var/spool/lpd:/sbin/nologin\nsync:x:5:0:sync:/"...
 
  count=1024, ppos=0xcd1c9180) at filemap.c:3001
 #5 0xc013d8e8 in sys_write (fd=4, 
 
  buf=0xbffff160 
 "root:x:0:0:root:/root:/bin/bash\nbin:x:1:1:bin:/bin:/sbin/nologin\ndaem
 on:x:2:2:daemon:/sbin:/sbin/nologin\nadm:x:3:4:adm:/var/adm:/sbin/nologi
 n\nlp:x:4:7:lp:/var/spool/lpd:/sbin/nologin\nsync:x:5:0:sync:/"...
 
  count=1024) at read_write.c:188
 #6 0xc010730b in system_call (
 

 Breakpoint 1, ux_get_block (inode=0xcd710440, 
 block=1, bh_result=0xc96b7240, create=1) 
 
 at ux_file.c:21
 21 struct super_block *sb = inode->i_sb;
 (gdb) bt
 #0 ux_get_block (inode=0xcd710440, block=1, 
 
 bh_result=0xc96b7240, create=1) 
 at ux_file.c:21
 

 Breakpoint 1, ux_get_block (inode=0xcd710440, block=2,
 bh_result=0xc9665900, create=1) 
 
 at ux_file.c:21
 21 struct super_block *sb = inode->i_sb;
 (gdb) bt
 #0 ux_get_block (inode=0xcd710440, block=2, 
 bh_result=0xc9665900, create=1) 
 at ux_file.c:21
 

 Breakpoint 1, ux_get_block (inode=0xcf23a420, 
 block=0, bh_result=0xc94bbba0, create=0) 
 at ux_file.c:21
 struct super_block *sb = inode->i_sb;
 (gdb) bt
 #0 ux_get_block (inode=0xcf23a420, block=0, 
 bh_result=0xc94bbba0, create=0) 
 at ux_file.c:21
 #1 0xc0140b1f in block_read_full_page (page=0xc1238340, 
 get_block=0xd0855094 
 at buffer.c:1781
 #2 0xd08551ba in ux_readpage (file=0xcd1c97e0, page=0xc1238340) 
 at ux_file.c:67
 #3 0xc012dd92 in page_cache_read (file=0xcd1c97e0, offset=3441203168) 
 at filemap.c:714
 #4 0xc012ddef in read_cluster_nonblocking (file=0xcd1c97e0, 
 offset=3475219664, filesize=1) 
 at filemap.c:739
 #5 0xc012f389 in filemap_nopage (area=0xc972a300, address=1073823744,
 unused=0) 
 at filemap.c:1911
 #6 0xc012b512 in do_no_page (mm=0xcf996d00, vma=0xc972a300, 
 address=1073823744, write_access=0, page_table=0xc91e60a0) 
 at memory.c:1249
 #7 0xc012b76c in handle_mm_fault (mm=0xcf996d00, vma=0xc972a300, 
 address=1073823744, write_access=0) 
 at memory.c:1339
 #8 0xc011754a in do_page_fault (regs=0xc952dfc4, error_code=4) 
 at fault.c:263
 #9 0xc01073fc in error_code (
 

 int 
 ux_statfs(struct super_block *sb, struct statfs *buf)
 

  int statfs(const char *path, struct statfs *buf)
 

 struct statfs 
 {
 long f_type; /* type of filesystem (see below) */ 
 long f_bsize; /* optimal transfer block size */ 
 long f_blocks; /* total data blocks in file system */ 
 long f_bfree; /* free blocks in fs */ 
 long f_bavail; /* free blocks avail to non-superuser */ 
 long f_files; /* total file nodes in file system */ 
 long f_ffree; /* free file nodes in fs */ 
 fsid_t f_fsid; /* file system id */ 
 long f_namelen; /* maximum length of filenames */ 
 }; 
 

 Entering kdb (current=0xc03b0000, pid 0) on processor 0 due to Keyboard Entry
 [0]kdb> bp ux_statfs
 Instruction(i) BP #0 at 0xd08bb400 ([uxfs]ux_statfs)
  is enabled globally adjust 
 [0]kdb> bl
 Instruction(i) BP #0 at 0xd08bb400 ([uxfs]ux_statfs)
  is enabled globally adjust 
 [0]kdb> go
 

 # df -k /mnt
 Filesystem 1k-blocks Used Available Use% Mounted on
 Instruction(i) breakpoint #0 at 0xd08bb400 (adjusted)
 0xd08bb400 ux_statfs
 Entering kdb (current=0xcd31c000, pid 1509) on processor 0 due to 
 Breakpoint @ 0xd08bb400
 [0]kdb> bt
 EBP EIP Function(args)
 0xcd31df38 0xd08bb400 [uxfs]ux_statfs (0xcc2be400, 0xcd31df50,0xffffffda, 
 uxfs .text 0xd08bb060 0xd08bb400 0xd08bb460
 0xc0141ea2 vfs_statfs+0xa2 (0xcc2be400, 0xcd31df50, 0x43, ..
 kernel .text 0xc0100000 0xc0141e00 0xc0141f20
 0xcd31dfbc 0xc0141f58 sys_statfs+0x38 (0x8052bb8, 0xbffff760, ..
 kernel .text 0xc0100000 0xc0141f20 0xc0141fb0
 0xc01077cb system_call+0x33
 kernel .text 0xc0100000 0xc0107798 0xc01077d0
 [0]kdb> go
 

 # ./mkfs /dev/fd0
 # insmod ./uxfs
 # mount -t uxfs /dev/fd0 /mnt
 # df -k 
 Filesystem 1k-blocks Used Available Use% Mounted on 
 /dev/hda2 15120648 2524836 11827716 18% / 
 /dev/hda1 102454 11147 86017 12% /boot 
 /dev/hda5 497829 8240 463887 2% /home 
 none 127076 0 127076 0% /dev/shm 
 /dev/fd0 1000 1 999 1% /mnt 
 

 # mkdir /mnt/dir
 # df -
 Filesystem 1k-blocks Used Available Use% Mounted on
 /dev/hda2 15120648 2524836 11827716 18% 
 /dev/hda1 102454 11147 86017 12% /boot
 /dev/hda5 497829 8240 463887 2% /home
 none 127076 0 127076 0% /dev/shm
 /dev/fd0 1000 2 998 1% /mnt
 

 # df -i /mnt
 Filesystem Inodes IUsed IFree IUse% Mounted on
 /dev/fd0 32 4 28 13% /mnt
 # mkdir /mnt/mydir
 # df -i /mnt
 Filesystem Inodes IUsed IFree IUse% Mounted on
 /dev/fd0 32 5 27 16% /mnt