Сетевое программирование на перл

Filehandles

Filehandles are the foundation of networked applications. In this section we review the ins and outs of filehandles. Even if you're an experienced Perl programmer, you might want to scan this section to refresh your memory on some of the more obscure aspects of Perl I/O.

Standard Filehandles

A filehandle connects a Perl script to the outside world. Reading from a filehandle brings in outside data, and writing to one exports data. Depending on how it was created, a filehandle may be connected to a disk file, to a hardware device such as a serial port, to a local process such as a command-line window in a windowing system, or to a remote process such as a network server. It's also possible for a filehandle to be connected to a "bit bucket" device that just sucks up data and ignores it.

A filehandle is any valid Perl identifier that consists of uppercase and lowercase letters, digits, and the underscore character. Unlike other variables, a filehandle does not have a distinctive prefix (like "$"). So to make them distinct, Perl programmers often represent them in all capital letters, or caps.

When a Perl script starts, exactly three filehandles are open by default: STDOUT, STDIN, and STDERR. The STDOUT filehandle, for "standard output," is the default filehandle for output. Data sent to this filehandle appears on the user's preferred output device, usually the command-line window from which the script was launched. STDIN, for "standard input," is the default input filehandle. Data read from this filehandle is taken from the user's preferred input device, usually the keyboard. STDERR ("standard error") is used for error messages, diagnostics, debugging, and other such incidental output. By default STDERR uses the same output device as STDOUT, but this can be changed at the user's discretion. The reason that there are separate filehandles for normal and abnormal output is so that the user can divert them independently; for example, to send normal output to a file and error output to the screen.

This code fragment will read a line of input from STDIN, remove the terminating end-of-line character with the chomp() function, and echo it to standard output:

$input = <STDIN>;
 chomp($input);
 print STDOUT "If I heard you correctly, you said: $input\n";
 

By taking advantage of the fact that STDIN and STDOUT are the defaults for many I/O operations, and by combining chomp() with the input operation, the same code could be written more succinctly like this:

chomp($input = <>);
 print "If I heard you correctly, you said: $input\n";
 

We review the <> and print() functions in the next section. Similarly, STDERR is the default destination for the warn() and die() functions.

The user can change the attachment of the three standard filehandles before launching the script. On UNIX and Windows systems, this is done using the redirect metacharacters "<" and ">". For example, given a script named muncher.pl this command will change the script's standard input so that it comes from the file data.txt, and its standard output so that processed data ends up in crunched.txt:

% muncher.pl <data.txt >crunched.txt
 						

Standard error isn't changed, so diagnostic messages (e.g., from the built-in warn() and die() functions) appear on the screen.

On Macintosh systems, users can change the source of the three standard filehandles by selecting filenames from a dialog box within the MacPerl development environment.

Input and Output Operations

Perl gives you the option of reading from a filehandle one line at a time, suitable for text files, or reading from it in chunks of arbitrary length, suitable for binary byte streams like image files.

For input, the <> operator is used to read from a filehandle in a line-oriented fashion, and read() or sysread() to read in a byte-stream fashion. For output, print() and syswrite() are used for both text and binary data (you decide whether to make the output line-oriented by printing newlines).

$line = <FILEHANDLE> @lines = <FILEHANDLE> $line <> @lines <> The <> ("angle bracket") operator is sensitive to the context in which it is called. If it is used to assign to a scalar variable, a so-called scalar context, it reads a line of text from the indicated filehandle, returning the data along with its terminating end-of-line character. After reading the last line of the filehandle, <> will return undef, signaling the end-of-file (EOF) condition. When <> is assigned to an array or used in another place where Perl ordinarily expects a list, it reads all lines from the filehandle through to EOF, returning them as one (potentially gigantic) list. This is called a list context. If called in a "void context" (i.e., without being assigned to a variable),<> copies a line into the $_ global variable. This is commonly seen in while() loops, and often combined with pattern matches and other operations that use $_ implicitly: while (<>) { print "Found a gnu\n" if /GNU/i; } The <FILEHANDLE> form of this function explicitly gives the filehandle to read from. However, the <> form is "magical." If the script was called with a set of file names as command-line arguments, <> will attempt to open() each argument in turn and will then return lines from them as if they were concatenated into one large pseudofile. If no files are given on the command line, or if a single file named "-" is given, then <> reads from standard input and is equivalent to <STDIN>. See the perlfunc POD documentation for an explanation of how this works (pod perlfunc, as explained in the Preface). $bytes = read (FILEHANDLE,$buffer,$length [,$offset]) $bytes = sysread (FILEHANDLE,$buffer,$length [,$offset]) The read() and sysread() functions read data of arbitrary length from the indicated filehandle. Up to $length bytes of data will be read, and placed in the $buffer scalar variable. Both functions return the number of bytes actually read, numeric 0 on the end of file, or undef on an error. This code fragment will attempt to read 50 bytes of data from STDIN, placing the information in $buffer, and assigning the number of bytes read to $bytes: my $buffer; $bytes = read (STDIN,$buffer,50); By default, the read data will be placed at the beginning of $buffer, overwriting whatever was already there. You can change this behavior by providing the optional numeric $offset argument, to specify that read data should be written into the variable starting at the specified position. The main difference between read() and sysread() is that read() uses standard I/O buffering, and sysread() does not. This means that read() will not return until either it can fetch the exact number of bytes requested or it hits the end of file. The sysread() function, in contrast, can return partial reads. It is guaranteed to return at least 1 byte, but if it cannot immediately read the number of bytes requested from the filehandle, it will return what it can. This behavior is discussed in more detail later in the Buffering and Blocking section. $result = print FILEHANDLE $data1,$data2,$data3... $result = print $data1,$data2,$data3... The print() function prints a list of data items to a filehandle. In the first form, the filehandle is given explicitly. Notice that there is no comma between the filehandle name and the first data item. In the second form, print() uses the current default filehandle, usually STDOUT. The default filehandle can be changed using the one-argument form of select() (discussed below). If no data arguments are provided, then print() prints the contents of $_. If output was successful, print() returns a true value. Otherwise it returns false and leaves an error message in the variable named $!. Perl is a parentheses-optional language. Although I prefer using parentheses around function arguments, most Perl scripts drop them with print(), and this book follows that convention as well. $result = printf $format,$data1,$data2,$data3... The printf() function is a formatted print. The indicated data items are formatted and printed according to the $format format string. The formatting language is quite rich, and is explained in detail in Perl's POD documentation for the related sprintf() (string formatting) function. $bytes = syswrite (FILEHANDLE,$data [,$length [,$offset]]) The syswrite() function is an alternative way to write to a filehandle that gives you more control over the process. Its arguments are a filehandle and a scalar value (avariable or string literal). It writes the data to the filehandle, and returns the number of bytes successfully written. By default, syswrite() attempts to write the entire contents of $data, beginning at the start of the string. You can alter this behavior by providing an optional $length and $offset, in which case syswrite() will write $length bytes beginning at the position specified by $offset. Aside from familiarity, the main difference between print() and syswrite() is that the former uses standard I/O buffering, while the latter does not. We discuss this later in the Buffering and Blocking section. Don't confuse syswrite() with Perl's unfortunately named write() function. The latter is part of Perl's report formatting package, which we won't discuss further. $previous = select(FILEHANDLE) The select() function changes the default output filehandle used by print print (). It takes the name of the filehandle to set as the default, and returns the name of the previous default. There is also a version of select() that takes four arguments, which is used for I/O multiplexing. We introduce the four-argument version in Chapter 8.

When reading data as a byte stream with read() or sysread(), a common idiom is to pass length($buffer) as the offset into the buffer. This will make read() append the new data to the end of data that was already in the buffer. For example:

my $buffer;
 while (1) {
   $bytes = read (STDIN,$buffer,50,length($buffer));
   last unless $bytes > 0;
 }
 

Detecting the End of File

The end-of-file condition occurs when there's no more data to be read from a file or device. When reading from files this happens at the literal end of the file, but the EOF condition applies as well when reading from other devices. When reading from the terminal (command-line window), for example, EOF occurs when the user presses a special key: control-D on UNIX, control-Z on Windows/DOS, and command-. on Macintosh. When reading from a network-attached socket, EOF occurs when the remote machine closes its end of the connection.

The EOF condition is signaled differently depending on whether you are reading from the filehandle one line at a time or as a byte stream. For byte-stream operations with read() or sysread(), EOF is indicated when the function returns numeric 0. Other I/O errors return undef and set $! to the appropriate error message. To distinguish between an error and a normal end of file, you can test the return value with defined():

while (1) {
   my $bytes = read(STDIN,$buffer,100);
   die "read error" unless defined ($bytes);
   last unless $bytes > 0;
 }
 

In contrast, the <> operator doesn't distinguish between EOF and abnormal conditions, and returns undef in either case. To distinguish them, you can set $! to undef before performing a series of reads, and check whether it is defined afterward:

undef $!;
 while (defined(my $line = <STDIN>)) {
    $data .= $line;
 }
 die "Abnormal read error: $!" if defined ($!);
 

When you are using <> inside the conditional of a while() loop, as shown in the most recent code fragment, you can dispense with the explicit defined() test. This makes the loop easier on the eyes:

while (my $line = <STDIN>) {
    $data .= $line;
 }
 

This will work even if the line consists of a single 0 or an empty string, which Perl would ordinarily treat as false. Outside while() loops, be careful to use defined() to test the returned value for EOF.

Finally, there is the eof() function, which explicitly tests a filehandle for the EOF condition:

$eof = eof(FILEHANDLE) The eof() function returns true if the next read on FILEHANDLE will return an EOF. Called without arguments or parentheses, as in eof, the function tests the last filehandle read from. When using while(<>) to read from the command-line arguments as a single pseudofile, eof() has "magical"or at least confusingproperties. Called with empty parentheses, as in eof(), the function returns true at the end of the very last file. Called without parentheses or arguments, as in eof, the function returns true at the end of each of the individual files on the command line. See the Perl POD documentation for examples of the circumstances in which this behavior is useful.

In practice, you do not have to use eof() except in very special circumstances, and a reliance on it is often a signal that something is amiss in the structure of your program.

Anarchy at the End of the Line

When performing line-oriented I/O, you have to watch for different interpretations of the end-of-line character. No two operating system designers can seem to agree on how lines should end in text files. On UNIX systems, lines end with the linefeed character (LF, octal \012 in the ASCII table); on Macintosh systems, they end with the carriage return character (CR, octal \015); and the Windows/DOS designers decided to end each line of text with two characters, a carriage return/linefeed pair (CRLF, or octal \015\012). Most line-oriented network servers also use CRLF to terminate lines.

This leads to endless confusion when moving text files between machines. Fortunately, Perl provides a way to examine and change the end-of-line character. The global variable $/ contains the current character, or sequence of characters, used to signal the end of line. By default, it is set to \012 on Unix systems, \015 on Macintoshes, and \015\012 on Windows and DOS systems.

The line-oriented <> input function will read from the specified handle until it encounters the end-of-line character(s) contained in $/, and return the line of text with the end-of-line sequence still attached. The chomp() function looks for the end-of-line sequence at the end of a text string and removes it, respecting the current value of $/.

The string escape \n is the logical newline character, and means different things on different platforms. For example, \n is equivalent to \012 on UNIX systems, and to \015 on Macintoshes. (On Windows systems, \n is usually \012, but see the later discussion of DOS text mode.) In a similar vein, \r is the logical carriage return character, which also varies from system to system.

When communicating with a line-oriented network server that uses CRLF to terminate lines, it won't be portable to set $/ to \r\n. Use the explicit string \015\012 instead. To make this less obscure, the Socket and IO::Socket modules, which we discuss in great detail later, have an option to export globals named $CRLF and CRLF() that return the correct values.

There is an additional complication when performing line-oriented I/O on Microsoft Windows and DOS machines. For historical reasons, Windows/DOS distinguishes between filehandles in "text mode" and those in "binary mode." In binary mode, what you see is exactly what you get. When you print to a binary filehandle, the data is output exactly as you specified. Similarly, read operations return the data exactly as it was stored in the file.

In text mode, however, the standard I/O library automatically translates LF into CRLF pairs on the way out, and CRLF pairs into LF on the way in. The virtue of this is that it makes text operations on Windows and UNIX Perls look the samefrom the programmer's point of view, the DOS text files end in a single \n character, just as they do in UNIX. The problem one runs into is when reading or writing binary filessuch as images or indexed databasesand the files become mysteriously corrupted on input or output. This is due to the default line-end translation. Should this happen to you, you should turn off character translation by calling binmode() on the filehandle.

binmode (FILEHANDLE [$discipline]) The binmode() function turns on binary mode for a filehandle, disabling character translation. It should be called after the filehandle is opened, but before doing any I/O with it. The single-argument form turns on binary mode. The two-argument form, available only with Perl 5.6 or higher, allows you to turn binary mode on by providing :raw as the value of $discipline, or restore the default text mode using :crlf as the value. binmode() only has an effect on systems like Windows and VMS, where the end-of-line sequence is more than one character. On UNIX and Macintosh systems, it has no effect.

Another way to avoid confusion over text and binary mode is to use the sysread() and syswrite() functions, which bypass the character translation routines in the standard I/O library.

A whole bevy of special global variables control other aspects of line-oriented I/O, such as whether to append an end-of-line character automatically to data output with the print() statement, and whether multiple data values should be separated by a delimiter. See Appendix B for a brief summary.

Opening and Closing Files

In addition to the three standard filehandles, Perl allows you to open any number of additional filehandles. To open a file for reading or writing, use the built-in Perl function open() If successful, open() gives you a filehandle to use for the read and/or write operations themselves. Once you are finished with the filehandle, call close() to close it. This code fragment illustrates how to open the file message.txt for writing, write two lines of text to it, and close it:

open (FH,">message.txt") or die "Can't open file: $!";
 print FH "This is the first line.\n";
 print FH "And this is the second.\n";
 close (FH) or die "Can't close file: $!";
 

We call open() with two arguments: a filehandle name and the name of the file we wish to open. The filehandle name is any valid Perl identifier consisting of any combination of uppercase and lowercase letters, digits, and the underscore character. To make them distinct, most Perl programmers choose all uppercase letters for filehandles. The " > " symbol in front of the filename tells Perl to overwrite the file's contents if it already exists, or to create the file if it doesn't. The file will then be opened for writing.

If open() succeeds, it returns a true value. Otherwise, it returns false, causing Perl to evaluate the expression to the right of the or operator. This expression simply dies with an error message, using Perl's $! global variable to retrieve the last system error message encountered.

We call print() twice to write some text to the filehandle. The first argument to print() is the filehandle, and the second and subsequent arguments are strings to write to the filehandle. Again, notice that there is no comma between the filehandle and the strings to print. Whatever is printed to a filehandle shows up in its corresponding file. If the filehandle argument to print() is omitted, it defaults to STDOUT.

After we have finished printing, we call close() to close the filehandle. close() returns a true value if the filehandle was closed uneventfully, or false if some untoward event, such as a disk filling up, occurred. We check this result code using the same type of or test we used earlier.

Let's look at open() and close() in more detail.

$success = open(FILEHANDLE,$path) $success = open(FILEHANDLE,$mode,$path) The open() call opens the file given in $path, associating it with a designated FILEHANDLE. There are both two- and three-argument versions of open(). In the three-argument version, which is available in Perl versions 5.6 and higher, a $mode argument specifies how the file is to be opened. $mode is a one- or two-character string chosen to be reminiscent of the I/O redirection operators in the UNIX and DOS shells. Choices are shown here. Mode Description < Open file for reading > Truncate file to zero length and open for writing >> Open file for appending, do not truncate +> Truncate file and then open for read/write <+ Open file for read/write, do not truncate

We can open the file named darkstar.txt for reading and associate it with the filehandle DARKFH like this:

open(DARKFH,'<','darkstar.txt');
 

In the two-argument form of open(), the mode is appended directly to the filename, as in:

open(DARKFH,'<darkstar.txt');
 

For readability, you can put any amount of whitespace between the mode symbol and the filename; it will be ignored. If you leave out the mode symbol, the file will be opened for reading. Hence the above examples are all equivalent to this:

open(DARKFH,'darkstar.txt');
 

If successful, open() will return a true value. Otherwise it returns false. In the latter case, the $! global will contain a human-readable message indicating thecause of the error.

$success = close(FH); The close() function closes a previously opened file, returning true if successful, or false otherwise. In the case of an error, the error message can again be found in $!. When your program exits, any filehandles that are still open will be closed automatically.

The three-argument form of open() is used only rarely. However, it has the virtue of not scanning the filename for special characters the way that the two-argument form does. This lets you open files whose names contain leading or trailing whitespace, ">" characters, and other weird and arbitrary data. The filename "-" is special. When opened for reading, it tells Perl to open standard input. When opened for writing, it tells Perl to open standard output.

If you call open() on a filehandle that is already open, it will be automatically closed and then reopened on the file that you specify. Among other things, this call can be used to reopen one of the three standard filehandles on the file of your choice, changing the default source or destination of the <>, print(), and warn() functions. We will see an example of this shortly.

As with the print() function, many programmers drop the parentheses around open() and close(). For example, this is the most common idiom for opening a file:

open DARKSTAR,"darkstar.txt" or die "Couldn't open darkstar.txt: $!"
 

I don't like this style much because it leads to visual ambiguity (does the or associate with the string "darkstar.txt" or with the open() function?). However, I do use this style with close(), print(), and return() because of their ubiquity.

The two-argument form of open() has a lot of magic associated with it (too much magic, some would say). The full list of magic behavior can be found in the perlfunc and perlopentut POD documentation. However, one trick is worth noting because we use it in later chapters. You can duplicate a filehandle by using it as the second argument to open() with the sequence >& or <& prepended to the beginning. >& duplicates filehandles used for writing, and <& duplicates those used for reading:

open (OUTCOPY,">&STDOUT");
 open (INCOPY,"<&STDOUT");
 

This example creates a new filehandle named OUTCOPY that is attached to the same device as STDOUT. You can now write to OUTCOPY and it will have the same effect as writing to STDOUT. This is useful when you want to replace one or more of the three standard filehandles temporarily, and restore them later. For example, this code fragment will temporarily reopen STDOUT onto a file, invoke the system date command (using the system() function, which we discuss in more detail in Chapter 2), and then restore the previous value of STDOUT. When date runs, its standard output is opened on the file, and its output appears there rather than in the command window:

#!/usr/bin/perl
 # file: redirect.pl
 
 print "Redirecting STDOUT\n";
 open (SAVEOUT,">&STDOUT");
 open (STDOUT,">test.txt") or die "Can't open test.txt: $!";
 
 print "STDOUT is redirected\n";
 system "date";
 
 open (STDOUT,">&SAVEOUT");
 print "STDOUT restored\n";
 

When this script runs, its output looks like this:

% redirect.pl
 Redirecting STDOUT
 STDOUT restored
 

and the file test.txt contains these lines:

STDOUT is redirected
 Thu Aug 10 09:19:24 EDT 2000
 

Notice how the second print() statement and the output of the date system command went to the file rather than to the screen because we had reopened STDOUT at that point. When we restored STDOUT from the copy saved in SAVEOUT, our ability to print to the terminal was restored.

Perl also provides an alternative API for opening files that avoids the magic and obscure syntax of open() altogether. The sysopen() function allows you to open files using the same syntax as the C library's open() function.

$result = sysopen (FILEHANDLE,$filename,$mode [,$perms]) The sysopen() function opens the file indicated by $filename, using the I/O mode indicated by $mode. If the file doesn't exist, and $mode indicates that the file should be created, then the optional $perms value specifies the permission bits for the newly created file. We discuss I/O modes and permissions in more detail below. If successful, sysopen() returns a true result and associates the opened file with FILEHANDLE. Otherwise it returns a false result and leaves the error message in $!.

The $mode argument used in sysopen() is different from the mode used in ordinary open(). Instead of being a set of characters, it is a numeric bitmask formed by ORing together one or more constants using the bitwise OR operator " | ". For example, the following snippet opens up a file for writing using a mode that causes it to be created if it doesn't exist, and truncated to length zero if it does (equivalent to open()'s " > " mode):

sysopen(FH, "darkstar.txt",O_WRONLY|O_CREAT|O_TRUNC)
       or die "Can't open: $!"
 

The standard Fcntl module exports the constants recognized by sysopen(), all of which begin with the prefix O_. Just use Fcntl at the top of your script togain access to them.

The mode constants useful for sysopen() are listed in Table 1.1. Each call to sysopen() must have one (and only one) of O_RDONLY, O_WRONLY, and O_RDWR. The O_WRONLY and O_RDWR constants may be ORed with one or more of O_CREAT, O_EXCL, O_TRUNC, or O_APPEND.

O_CREAT causes the file to be created if it doesn't already exist. If it isn't specified, and the file doesn't exist when you try to open it for writing, then sysopen() will fail.

Combining O_CREAT with O_EXCL leads to the useful behavior of creating the file if it doesn't already exist, but failing if it does. This can be used to avoid accidentally clobbering an existing file.

If O_TRUNC is specified, then the file is truncated to zero length before the first write, effectively overwriting the previous contents. O_APPEND has the opposite effect, positioning the write pointer to the end of the file so that everything written to the file is appended to its existing contents.

Table 1.1. sysopen() Mode Constants

Constant	Description
O_RDONLY	Open read only.
O_WRONLY	Open write only.
O_RDWR	Open read/write.
O_CREAT	Create file if it doesn't exist.
O_EXCL	When combined with O_CREAT, create file if it doesn't exist and fail if it does.
O_TRUNC	If file already exists, truncate it to zero.
O_APPEND	Open file in append mode (equivalent to open()'s " >> ").
O_NOCTTY	If the file is a terminal device, open it without allowing it to become the process's controlling terminal.
O_NONBLOCK	Open file in nonblockingmode.
O_SYNC	Open file for synchronous mode, so that all writes block until the data is physically written.

The O_NOCTTY, O_NONBLOCK, and O_SYNC modes all have specialized uses that are discussed in later chapters.

If sysopen() needs to create a file, the $perm argument specifies the permissions mode of the resulting file. File permissions is a UNIX concept that maps imperfectly onto the Windows/DOS world, and not at all onto the Macintosh world. It is an octal value, such as 0644 (which happens to specify read/write permissions for the owner of the file, and read-only permissions for others).

If $perm is not provided, sysopen() defaults to 0666, which grants read/write access to everyone. However, whether you specify the permissions or accept the default, the actual permissions of the created file are determined by performing the bitwise AND between the $perm argument and the current contents of the user's umask (another UNIX concept). This is often set, at the user's discretion, to forbid access to the file from outside the user's account or group.

In most circumstances, it is best to omit the permissions argument and let the user adjust the umask. This also increases the portability of the program. See the umask() entry in the perlfunc POD documentation for information on how you can examine and set the umask programatically.

Buffering and Blocking

When you print() or syswrite() to a filehandle, the actual output operation does not occur immediately. If you are writing to a file, the system has to wait for the write head to reach the proper location on the disk drive, and for the spinning platter to bring the proper location under the head. This is usually an insignificant length of time (although it may be quite noticeable on a laptop that intermittently spins down its disk to save battery power), but other output operations can take much more time. In particular, network operations may take a considerable length of time to complete. The same applies to input.

There is a fundamental mismatch between computational speed and I/O speed. A program can execute the contents of a tight loop a million times a second, but a single I/O operation may take several seconds to complete. To overcome this mismatch, modern operating systems use the techniques of buffering and blocking.

The idea behind buffering is shown in Figure 1.3. Buffers decouple the I/O calls invoked by the program from the actual I/O operations that occur at the hardware level. A call to print(), for example, doesn't send data directly to the terminal, network card, or disk drive, but instead it results in data being written to a memory area. This occurs quickly, because writes to memory are fast. Meanwhile, in an asynchronous fashion, the operating system reads from data previously written to the buffer and performs the actions necessary to write the information to the hardware device.

Similarly, for input operations, the operating system receives data from active input devices (the keyboard, disk drive, network card) and writes that data to an input buffer somewhere in memory. The data remains in the input buffer until your program calls <> or read(), at which point the data is copied from the operating system's buffer into the memory space corresponding to a variable in your program.

The advantage of buffering is significant, particularly if your program performs I/O in a "bursty" way; that is, it performs numerous reads and writes of unpredictable size and timing. Instead of waiting for each operation to complete at the hardware level, the data is safely buffered in the operating system and "flushed"passed on to the output devicewhenever the downstream hardware can accept it.

The buffers in Figure 1.3 are conceptually circular FIFO (first in first out) data structures. When data is written beyond the end of the buffer memory area, the operating system merely begins writing new data at the beginning. The operating system maintains two pointers in each of its I/O buffers. The write pointer is the place where new data enters the buffer. The read pointer is the position from which stored data is moved out of the buffer to its next destination. For example, on write operations, each print() you perform adds some data to the output buffer and advances the write pointer forward. The operating system reads older data starting at the read pointer and copies it to the low-level hardware device.

The size of the I/O buffer minus the distance between the write pointer and the read pointer is the amount of free space remaining. If your program is writing faster than the output hardware can receive it, then the buffer will eventually fill up and the buffer's free space will be zero. What happens then?

Because there is no room for new data in the buffer, the output operation cannot succeed immediately. As a result, the write operation blocks. Your program will be suspended at the blocked print() or syswrite() for an indefinite period of time. When the backlog in the output buffer clears and there is again room to receive new data, the output operation will unblock and print() or syswrite() will return.

In a similar fashion, reads will block when the input buffer is empty; that is, it blocks when the amount of free space is equal to the size of the buffer. In this case, calls to read() or sysread() will block until some new data has entered the buffer and there is something to be read.

Blocking is often the behavior you want, but sometimes you need more control over I/O. There are several techniques to manage blocking. One technique, discussed in Chapter 2 under Timing Out System Calls, uses signals to abort an I/O operation prematurely if it takes too long. Another technique, discussed in Chapter 12, uses the four-argument select() system call to test a filehandle for its readiness to perform I/O before actually making the read or write call. A third technique, discussed in Chapter 13, is to mark the filehandle as nonblocking, which causes the read or write operation to return immediately with an error code if the operation would block.

Standard I/O Buffering

Although we have spoken of a single buffer for I/O operations on a filehandle, there may in fact be several buffers at different layers of the operating system. For example, when writing to a disk file, there is a very low-level buffer on the disk hardware itself, another one in the SCSI or IDE driver that controls the disk, a third in the driver for the filesystem, and a fourth in the standard C library used by Perl. I may have missed a few.

You cannot control or even access most of these buffers directly, but there is one class of buffer that you should be aware of. Many of Perl's I/O operations flow through "stdio," a standard C-language library which maintains its own I/O buffers independent of the operating system's.^[1]

^[1] You can think of the stdio library as a layersitting on top of the OS, making the OS look more C-like to programs; similarly, a Pascal standard I/O library makes the OS look as if it were written in Pascal.

Perl's <> operator, read(), and print() all use stdio. When you call print(), the data is transferred to an output buffer in the stdio layer before being sent to the operating system itself. Likewise, <> and read() both read data from an stdio buffer rather than directly from the OS. Each filehandle has its own set of buffers for input and output. For efficiency reasons, stdio waits until its output buffers reach a certain size before flushing their contents to the OS.

Normally, the presence of the stdio buffering is not a problem, but it can run you into trouble when doing more sophisticated types of I/O such as network operations. Consider the common case of an application that requires you to write a small amount of data to a remote server, wait for a response, and then send more data. You may think that the data has been sent across the network, but in fact it may not have been. The output data may still be sitting in its local stdio buffer, waiting for more data to come before flushing the buffer. The remote server will never receive the data, and so will never return a response. Your program will never receive a response and so will never send additional data. Deadlock ensues.

In contrast, the lower-level buffering performed by the operating system doesn't have this property. The OS will always attempt to deliver whatever data is in its output buffers as soon as the hardware can accept it.

There are two ways to work around stdio buffering. One is to turn on "autoflush" mode for the filehandle. Autoflush mode applies only to output operations. When active, Perl tells stdio to flush the filehandle's buffer every time print() is called.

To turn on autoflush mode, set the special variable $| to a true value. Autoflush mode affects the currently selected filehandle, so to change autoflush mode for a specific filehandle, one must first select() it, set $| to true, and then select() the previously selected filehandle. For example, to turn on autoflush mode for filehandle FH:

my $previous = select(FH);
 $| = 1;
 select($previous);
 

You will sometimes see this motif abbreviated using the following mind-blowing idiom:

select((select(FH),$|=1)[0]);
 

However, it is much cleaner to bring in the IO::Handle module, which adds an autoflush() method to filehandles. With IO::Handle loaded, FH can be put into autoflush mode like this:

use IO::Handle;
 FH->autoflush(1);
 

If the OO syntax confuses you, see the Objects and References section later in this chapter.

The other way to avoid stdio buffering problems is to use the sysread()i and syswrite() calls. These calls bypass the stdio library and go directly to the operating system I/O calls. An important advantage of these calls is that they interoperate well with other low-level I/O calls, such as the four-argument select() call, and with advanced techniques such as nonblocking I/O.

Another ramification of the fact that the sys*() functions bypass stdio is the difference in behavior between read() and sysread() when asked to fetch a larger chunk of data than is available. In the case of read(), the function will block indefinitely until it can fetch exactly the amount of data requested. The only exception to this is when the filehandle encounters the end of file before the full request has been satisfied, in which case read() will return everything to the end of the file. In contrast, sysread() can and will return partial reads. If it can't immediately read the full amount of data requested, it will return the data that is available. If no data is available, sysread() will block until it can return at least 1 byte. This behavior makes sysread() invaluable for use in network communications, where data frequently arrives in chunks of unpredictable size.

For these reasons, sysread() and syswrite() are preferred for many network applications.

Passing and Storing Filehandles

Network applications frequently must open multiple filehandles simultaneously, pass them to subroutines, and keep track of them in hashes and other data structures. Perl allows you to treat filehandles as strings, store them into variables, and pass them around to subroutines. For example, this functional but flawed code fragment stores the MY_FH filehandle into a variable named $fh, and then passes it to a subroutine named hello() to use in printing a friendly message:

# Flawed technique
 $fh = MY_FH;
 hello($fh);
 sub hello {
   my $handle = shift;
   print $handle "Howdy folks!\n";
 }
 

This technique often works; however, it will run you into problems as soon asyou try to pass filehandles to subroutines in other packages, such as functions exported by modules. The reason is that passing filehandles as strings loses the filehandle package information. If we pass the filehandle MY_FH from the main script (package main) to a subroutine defined in the MyUtils module, the subroutine will try to access a filehandle named MyUtils::MY_FH rather than the true filehandle, which is main::MY_FH. The same problem also occurs, of course, when a subroutine from one package tries to return a filehandle to a caller from another package.

The correct way to move filehandles around is as a typeglob or a typeglob reference. Typeglobs are symbol table entries, but you don't need to know much more about them than that in order to use them (see the perlref POD documentation for the full details). To turn a filehandle into a glob put an asterisk ("*") in front of its name:

$fh = *MY_FH;
 

To turn a filehandle into a typeglob reference, put "\*" in front of its name:

$fh = \*MY_FH;
 

In either case, $fh can now be used to pass the filehandle back and forth between subroutines and to store filehandles in data structures. Of the two forms, the glob reference (\*HANDLE) is the safer, because there's less risk of accidentally writing to the variable and altering the symbol table. This is the form we use throughout this book, and the one used by Perl's I/O-related modules, such as IO::Socket.

Typeglob references can be passed directly to subroutines:

hello(\*MY_FH):
 

They can also be returned directly by subroutines:

my $fh = get_fh();
 sub get_fh {
    open (FOO,"foo.txt") or die "foo: $!";
    return \*FOO;
 }
 

Typeglob refs can be used anywhere a bare filehandle is accepted, including as the first argument to print(), read(), sysread(), syswrite(), or any of the socket-related calls that we discuss in later chapters.

Sometimes you will need to examine a scalar variable to determine whether it contains a valid filehandle. The fileno() function makes this possible:

$integer = fileno (FILEHANDLE) The fileno() function accepts a filehandle in the form of a bare string, a typeglob, or a typeglob reference. If the filehandle is valid, fileno() returns the file descriptor for the filehandle. This is a small integer that uniquely identifies the filehandle to the operating system. STDIN, STDOUT, and STDERR generally correspond to descriptors 0, 1, and 2, respectively (but this can change if you close and reopen them). Other filehandles have descriptors greater than 3. If the argument passed to fileno() does not correspond to a valid filehandle (including former filehandles that have been closed), fileno() returns undef. Here is the idiom for checking whether a scalar variable contains a filehandle: die "not a filehandle" unless defined fileno($fh);

Detecting Errors

Because of the vicissitudes of Internet communications, I/O errors are common in network applications. As a rule, each of the Perl functions that performs I/O returns undef, a false value, on failure. More specific information can be found by examining the special global variable $!.

$! has an interesting dual nature. Treated as a string, it will return a human-readable error message such as Permission denied. Treated as a number, however, it will return the numeric constant for the error, as defined by the operating system (e.g., EACCES). It is generally more reliable to use these numeric error constants to distinguish particular errors, because they are standardized across operating systems.

You can obtain the values of specific error message constants by importing them from the Errno module. In the use statement, you can import individual constants by name, or all of them at once. To bring in individual constants, list them in the use() statement, as shown here:

use Errno qw(EACCES ENOENT);
 my $result = open (FH,">/etc/passwd");
 if (!$result) { # oops, something went wrong
    if ($! == EACCESS) {
       warn "You do not have permission to open this file.";
    } elsif ($! == ENOENT) {
       warn "File or directory not found.";
    } else {
       warn "Some other error occurred: $!";
    }
 }
 

The qw() operator is used to split a text string into a list of words. The first line above is equivalent to:

use Errno ('EACCESS','ENOENT');
 

and brings in the EACCESS and ENOENT constants. Notice that we use the numeric comparison operator " == " when comparing $! to numeric constants.

To bring in all the common error constants, import the tag :POSIX. Thisbrings in the error constants that are defined by the POSIX standard, across-platform API that UNIX, Windows NT/2000, and many other operating systems are largely compliant with. For example:

use Errno qw(:POSIX);
 

Do not get into the habit of testing $! to see if an error occurred during the last operation. $! is set when an operation fails, but is not unset when an operation succeeds. The value of $! should be relied on only immediately after a function has indicated failure.

Using Object-Oriented Syntax with the IO::Handle and IO::File Modules

We use Perl5's object-oriented facilities extensively later in this book. Although you won't need to know much about creating object-oriented modules, you will need a basic understanding of how to use OO modules and their syntax. This section illustrates the basics of Perl's OO syntax with reference to the IO::Handle and IO::File module, which together form the basis of Perl's object-oriented I/O facilities.

Objects and References

In Perl, references are pointers to data structures. You can create a reference to an existing data structure using the backslash operator. For example:

$a = 'hi there';
 $a_ref = \$a; # reference to a scalar
 @b = ('this','is','an','array');
 $b_ref = \@b; # reference to an array
 %c = ( first_name => 'Fred', last_name => 'Jones');
 $c_ref = \%c; # reference to a hash
 

Once a reference has been created, you can make copies of it, as you would any regular scalar, or stuff it into arrays or hashes. When you want to get to the data contained inside a reference, you dereference it using the prefix appropriate for its contents:

$a = $$a_ref;
 @b = @$b_ref;
 %c = %$c_ref;
 

You can index into array and hash references without dereferencing the whole thing by using the -> syntax:

$b_ref->[2];          # yields "an"
 $c_ref->{last_name};  # yields "Jones"
 

It is also possible to create references to anonymous, unnamed arrays and hashes, using the following syntax:

$anonymous_array = ['this','is','an','anonymous','array'];
 $anonymous_hash  = { first_name =>'Jane', last_name => 'Doe' };
 

If you try to print out a reference, you'll get a string like HASH(0x82ab0e0), which indicates the type of reference and the memory location where it can be found (which is just short of useless).

An object is a reference with just a little bit extra. It is "blessed" into a particular module's package in such a way that it carries information about what module created it.^[2] The blessed reference will continue to work just like other references. For example, if the object named $object is a blessed hash reference, you can index into it like this:

^[2] The function responsible for turning ordinary references into blessed ones is, naturally enough, called bless().

$object->{last_name};
 

What makes objects different from plain references is that they have methods. A method call uses the -> notation, but followed by a subroutine name and optional subroutine-style arguments:

$object->print_record(); # invoke the print_record() method
 

You may sometimes see a method called with arguments, like this:

$object->print_record(encoding => 'EBCDIC');
 

The "=>" symbol is accepted by Perl as a synonym for ','. It makes the relationship between the two arguments more distinct, and has the added virtue of automatically quoting the argument on the left. This allows us to write encoding rather than "encoding". If a method takes no arguments, it's often written with the parentheses omitted, as in:

$object->print_record;
 

In many cases, print_record() will be a subroutine defined in the object's package. Assuming that the object was created by a module named BigDatabase, the above is just a fancy way of saying this:

BigDatabase::print_record($object);
 

However, Perl is more subtle than this, and the print_record(), method definition might actually reside in another module, which the current module inherits from. How this works is beyond the scope of this introduction, and can be found in the perltoot, perlobj, and perlref POD pages, as well as in [Wall 2000] and the other general Perl reference works listed in Appendix D.

To create an object, you must invoke one of its constructors. A constructor is a method call that is invoked from the module's name. For example, to create a new BigDatabase object:

$object = BigDatabase->new(); # call the new() constructor
 

Constructors, which are a special case of a class method, are frequently named new(). However, any subroutine name is possible. Again, this syntax is part trickery. In most cases an equivalent call would be:

$object = BigDatabase::new('BigDatabase');
 

This is not quite the same thing, however, because class methods can also be inherited.

The IO::Handle and IO::File Modules

The IO::Handle and IO::File modules, standard components of Perl, together provide object-oriented interface to filehandles. IO::Handle provides generic methods that are shared by all filehandle objects, including pipes and sockets. The more specialized class, IO::File, adds functionality for opening and manipulating files. Together, these classes smooth out some of the bumps and irregularities in Perl's built-in filehandles, and make larger programs easier to understand and maintain.

IO::File's elegance does not by itself provide any very compelling reason to choose the object-oriented syntax over native filehandles. Its main significance is that IO::Socket, IO::Pipe, and other I/O-related modules also inherit their behavior from IO::Handle. This means that programs that read and write from local files and those that read and write to remote network servers share a common, easy-to-use interface.

We'll get a feel for the module by looking at a tiny example of a program that opens a file, counts the number of lines, and reports its findings (Figure 1.4).

Lines 13: Load modules We turn on strict syntax checking, and load the IO::File module.

Lines 45: Initialize variables We recover from the command line the name of the file to perform the line count on, and initialize the $counter variable to zero.

Line 6: Create a new IO::File object We call the IO::File::new() method, using the syntax IO::File->new(). The argument is the name of the file to open. If successful, new() returns a new IO::File object that we can use for I/O. Otherwise it returns undef, and we die with an error message.

Lines 79: Main loop We call the IO::File object's getline() method in the test portion of a while() loop. This method returns the next line of text, or undef on end of filejust like <>.

Each time through the loop we bump up $counter. The loop continues until getline() returns undef.

Line 10: Print results We print out our results by calling STDOUT->print(). We'll discuss why this surprising syntax works in a moment.

When I ran count_lines.pl on the unfinished manuscript for this chapter, I got the following result:

% count_lines.pl ch1.pod
 Counted 2428 lines
 

IO::File objects are actually blessed typeglob references (see the Passing and Storing Filehandles section earlier in this chapter). This means that you can use them in an object-oriented fashion, as in:

$fh->print("Function calls are for the birds.\n");
 

or with the familiar built-in function calls:

print $fh "Object methods are too effete.\n";
 

Many of IO::File's methods are simple wrappers around Perl's built-in functions. In addition to print() and getline() methods, there are read(), syswrite(), and close() methods, among others. We discuss the pros and cons of using object-oriented method calls and function calls in Chapter 5, where we introduce IO:: Socket.

When you load IO::File (technically, when IO::File loads IO::Handle, which it inherits from), it adds methods to ordinary filehandles. This means that any of the methods in IO::File can also be used with STDIN, STDOUT, STDERR, or even with any conventional filehandles that you happen to create. This is why line 10 of Figure 1.4 allows us to print to standard output by calling print().

Of the method listings that follow, only the new() and new_tmpfile() methods are actually defined by IO::File. The rest are inherited from IO::Handle and can be used with other descendents of IO::Handle, such as IO::Socket. This list is not complete. I've omitted some of the more obscure methods, including those that allow you to move around inside a file in a record-oriented fashion, because we won't need them for network communications.

$fh = IO::File->new($filename [,$mode [,$perms]]) The new() method is the main constructor for IO::File. It is a unified replacement for both open() and sysopen(). Called with a single argument, new() acts like the two-argument form of open(), taking a filename optionally preceded by a mode string. For example, this will open the indicated file for appending: $fh = IO::File->new(">darkstar.txt"); If called with two or three arguments, IO::File treats the second argument as the open mode, and the third argument as the file creation permissions. $mode may be a Perl-style mode string, such as " +< ", or an octal numeric mode, such as those used by sysopen(). As a convenience, IO::File automatically imports the Fcntl O_* constants when it loads. In addition, open() allows for an alternative type of symbolic mode string that is used in the C fopen() call; for example, it allows " w " to open the file for writing. We won't discuss those modes further here, because they do not add functionality. The permission agreement given by $perms is an octal number, and has the same significance as the corresponding parameter passed to sysopen(). If new() cannot open the indicated file, it will return undef and set $! to the appropriate system error message. $fh = IO::File->new_tmpfile The new_tmpfile() constructor, which is called without arguments, creates a temporary file opened for reading and writing. On UNIX systems, this file is anonymous, meaning that it cannot be seen on the file system. When the IO::File object is destroyed, the file and all its contents will be deleted automatically. This constructor is useful for storing large amounts of temporary data. $result = $fh->close The close() method closes the IO::File object, returning a true result if successful. If you do not call close() explicitly, it will be called automatically when the object is destroyed. This happens when the script exits, if you happen to undef() the object, or if the object goes out of scope such as when a my variable reaches the end of its enclosing block. $result = $fh->open($filename [,$mode [,$perms]]) You can reopen a filehandle object on the indicated file by using its open() method. The input arguments are identical to new(). The method result indicates whether the open was successful. This is chiefly used for reopening the standard filehandles STDOUT, STDIN, and STDERR. For example: STDOUT->open("log.txt") or die "Can't reopen STDOUT: $!"; Calls to print() will now write to the file log.txt . $result = $fh->print(@args) $result=$fh->printf($fmt,@args) $bytes=$fh->write($data [,$length [,$offset]]) $bytes = $fh->syswrite($data [,$length [,$offset]]) The print(), printf(), and syswrite() methods work exactly like their built-in counterparts. For example, print() takes a list of data items, writes them to the filehandle object, and returns true if successful. The write() method is the opposite of read(), writing a stream of bytes to the filehandle object and returning the number successfully written. It is similar to syswrite(), except that it uses stdio buffering. This method corrects the inconsistent naming of the built-in write() function, which creates formatted reports. The IO::File object method that corresponds to the built-in write() goes by the name of format_write(). $line = $fh->getline @lines=$fh->getlines $bytes = $fh->read($buffer,$length ",$offset") $bytes = $fh->ysread($buffer,$length ",$offsetr") The getline() and getlines() methods together replace the <> operator. getline() reads one line from the filehandle object and returns it, behaving in the same way in both scalar and list contexts. The getlines() method acts like <> in a list context, returning a list of all the available lines. getline() will return undef at the end of file. The read() and sysread() methods act just like their built-in function counterparts. $previous = $fh->autoflush([$boolean]) The autoflush() method gets or sets the autoflush() mode for the filehandle object. Called without arguments, it turns on autoflush. Called with a single boolean argument, it sets autoflush to the indicated status. In either case, autoflush()i returns the previous value of the autoflush state. $boolean = $fh->opened The opened() method returns true if the filehandle object is currently valid. It is equivalent to: defined fileno($fh); $boolean = $fh->eof Returns true if the next read on the filehandle object will return EOF. $fh->flush The flush() method immediately flushes any data that is buffered in the filehandle object. If the filehandle is being used for writing, then its buffered data is written to disk (or to the pipe, or network, as we'll see when we get to IO::Socket objects). If the filehandle is being used for reading, any data in the buffer is discarded, forcing the next read to come from disk. $boolean = $fh->blocking("$boolean") The blocking() method turns on and off blocking mode for the filehandle. We discuss how to use this at length in Chapter 13. $fh->clearerr $boolean = $fh->error These two methods are handy if you wish to perform a series of I/O operations and check the error status only after you're finished. The error()i method will return true if any errors have occurred on the filehandle since it was created, or since the last call to clearerr(). The clearerr() method clears this flag.

In addition to the methods listed here, IO::File has a constructor named new_from_fd(), and a method named fdopen(), both inherited from IO::Handle. These methods can be used to save and restore objects in much the way that the >&FILEHANDLE does with standard filehandles.

$fh = IO::File->new_from_fd($fd,$mode) The new_from_fd() method opens up a copy of the filehandle object indicated by $fd using the read/write mode given by $mode. The object may be an IO::Handle object, an IO::File object, a regular filehandle, or a numeric file descriptor returned byfileno(). $mode must match the mode with which $fd was originally opened. For example: $saveout = IO::File->new_from_fd(STDOUT,">"); $result = $fh->fdopen($fd,$mode) The fdopen() method is used to reopen an existing filehandle object, making it a copy of another one. The $fd argument may be an IO::Handle object or a regular filehandle, or a numeric file descriptor $mode must match the mode with which $fd was originally opened. This is typically used in conjunction with new_from_fd() to restore a saved filehandle: $saveout = IO::File->new_from_fd(STDOUT,">"); # save STDOUT STDOUT->open('>log.txt'); # reopen on a file STDOUT->print("Yippy yie yay!\n"); # print something STDOUT->fdopen($saveout,">"); # reopen on saved value

See the POD documentation for IO::Handle and IO::File for information about the more obscure features that these modules provide.

Chapter 2. Processes, Pipes, and Signals

This chapter discusses three key Perl features: processes, pipes, and signals. By creating new processes, a Perl program can run another program or even clone copies of itself in order to divide the work. Pipes allow Perl scripts to exchange data with other processes, and signals make it possible for Perl scripts to monitor and control other processes.

Processes

UNIX, VMS, Windows NT/2000, and most other modern operating systems are multitasking. They can run multiple programs simultaneously, each one running in a separate thread of execution known as a process. On machines with multiple CPUs, the processes running on different CPUs really are executing simultaneously. Processes that are running on single-CPU machines only appear to be running simultaneously because the operating system switches rapidly between them, giving each a small slice of time in which to execute.

Network applications often need to do two or more things at once. For example, a server may need to process requests from several clients at once, or handle a request at the same time that it is watching for new requests. Multitasking simplifies writing such programs tremendously, because it allows you to launch new processes to deal with each of the things that the application needs to do. Each process is more or less independent, allowing one process to get on with its work without worrying about what the others are up to.

Perl supports two types of multitasking. One type, based on the traditional UNIX multiprocessing model, allows the current process to clone itself by making a call to the fork() function. After fork() executes, there are two processes, each identical to the other in almost every respect. One goes off to do one task, and the other does another task.

Another type, based on the more modern concept of a lightweight "thread," keeps all the tasks within a single process. However, a single program can have multiple threads of execution running through it, each of which runs independently of the others.

In this section, we introduce fork() and the variables and functions that are relevant to processes. We discuss multithreading in Chapter 11.

The fork() Function

The fork() function is available on all UNIX versions of Perl, as well as the VMS and OS/2 ports. Version 5.6 of Perl supports fork() on Microsoft Windows platforms, but not, unfortunately, on the Macintosh.

The Perl fork() function takes no arguments and returns a numeric result code. When fork() is called, it spawns an exact duplicate of the current process. The duplicate, called the child, shares the current values of all variables, filehandles (including data in the standard I/O buffers), and other data structures. In fact, the duplicate process even has the memory of calling fork(). It is like a man walking into the cloning booth of a science fiction movie. The copy wakes up in the other booth having all the memories of the original up to and including walking into the cloning booth, but thinking wait, didn't I start out over there? And who is the handsome gentleman in that other booth?

To ensure peaceful coexistence, it is vital that the parent and child processes know which one is which. Each process on the system is associated with a unique positive integer, known as its process ID, or PID.

After a call to fork(), the parent and child examine the function's return value. In the parent process, fork() returns the PID of the child. In the child process, fork() returns numeric 0. The code will go off and do one thing if it discovers it's the parent, and do another if it's the child.

$pid = fork() Forks a new process. Returns the child's PID in the parent process, and 0 in the child process. In case of error (such as insufficient memory to fork), returns undef, and sets $! to the appropriate error message.

If the parent and child wish to communicate with each other following the fork, they can do so with a pipe (discussed later in this chapter in the Pipes section), or via shared memory (discussed in Chapter 14 in the An Adaptive Preforking Server Using Shared Memory section). For simple messages, parent and child can send signals to each others' PIDs using the kill() function. The parent gets the child's PID from fork()'s result code, and the child can get the parent's PID by calling getppid(). A process can get its own PID by examining the $$ special variable.

$pid = getppi() Returns the PID of the parent process. Every Perl script has a parent, even those launched directly from the command line (their parent is the shell process). $$ The $$ variable holds the current PID for the process. It can be read, but not changed.

We discuss the kill() function later in this chapter, in the Signals section.

If it wishes, a child process can itself fork(), creating a grandchild. The original parent can also fork() again, as can its children and grandchildren. In this way, Perl scripts can create a whole tribe of (friendly, we hope) processes. Unless specific action is taken, each member of this tribe belongs to the same process group.

Each process group has a unique ID, which is usually the same as the process ID of the shared ancestor. This value can be obtained by calling getpgrp():

$processid = getpgrp([$pid]) For the process specified by $pid, the getpgrp() function returns its process group ID. If no PID is specified, then the process group of the current process is returned.

Each member of a process group shares whatever filehandles were open at the time its parent forked. In particular, they share the same STDIN, STDOUT, and STDERR. This can be modified by any of the children by closing a filehandle, or reopening it on some other source. However, the system keeps track of which children have filehandles open, and will not close the file until the last child has closed its copy of the filehandle.

Figure 2.1 illustrates the typical idiom for forking. Before forking, we print out the PID stored in $$. We then call fork() and store the result in a variable named $child. If the result is undefined, then fork() has failed and we die with an error message.

We now examine $child to see whether we are running in the parent or the child. If $child is nonzero, we are in the parent process. We print out our PID and the contents of $child, which contains the child process's PID.

If $child is zero, then we are running in the child process. We recover the parent's PID by calling ppid() and print it and our own PID.

Here's what happens when you run fork.pl:

% fork.pl
 PID=372
 Parent process: PID=372, child=373
 Child process:  PID=373, parent=372
 

The system() and exec() Functions

Another way for Perl to launch a subprocess is with system(). system() runs another program as a subprocess, waits for it to complete, and then returns. If successful, system() returns a result code of 0 (notice that this is different from the usual Perl convention!). Otherwise it returns -1 if the program couldn't be started, or the program's exit status if it exited with some error. See the perlvar POD entry for the $? variable for details on how to interpret the exit status fully.

$status = system ('command and arguments') $status = system ('command', 'and', 'arguments') The system() function executes a command as a subprocess and waits for it to exit. The command and its arguments can be specified as a single string, or as a list containing the command and its arguments as separate elements. In the former case, the string will be passed intact to the shell for interpretation. This allows you to execute commands that contain shell metacharacters (such as input/output re-directs), but opens up the possibility of executing shell commands you didn't anticipate. The latter form allows you to execute commands with arguments that contain whitespace, shell metacharacters, and other special characters, but it doesn't interpret metacharacters at all.

The exec() function is like system(), but replaces the current process with the indicated command. If successful, it never returns because the process is gone. The new process will have the same PID as the old one, and will share the same STDIN, STDOUT, and STDERR filehandles. However, other opened filehandles will be closed automatically.^[1]

^[1] You can arrange for some filehandles to remain open across exec() by changing the value of the $~F special variable. See the perlvar POD document for details.

$status = exec ('command and arguments') $status = exec ('command', 'and', 'arguments') exec() executes a command, replacing the current process. It will return a status code only on failure. Otherwise it does not return. The single-value and list forms have the same significance as system().

exec() is often used in combination with fork() to run a command as a subprocess after doing some special setup. For example, after this code fragment forks, the child reopens STDOUT onto a file and then calls exec() to run the ls -l command. On UNIX systems, this command generates a long directory listing. The effect is to run ls -l in the background, and to write its output to the indicated file.

my $child = fork();
 die "Can't fork: $!" unless defined $child;
 if ($child == 0) { # we are in the child now
   open (STDOUT,">log.txt") or die "open() error: $!";
   exec ('ls','-l');
   die "exec error(): $!"; # shouldn't get here
 }
 

We use exec() in this way in Chapter 10, in the section titled The Inetd Super Daemon.

Pipes

Network programming is all about interprocess communication (IPC). One process exchanges data with another. Depending on the application, the two processes may be running on the same machine, may be running on two machines on the same segment of a local area network, or may be halfway across the world from each other. The two processes may be related to each otherfor example, one may have been launched under the control of the otheror they may have been written decades apart by different authors for different operating systems.

The simplest form of IPC that Perl offers is the pipe. A pipe is a filehandle that connects the current script to the standard input or standard output of another process. Pipes are fully implemented on UNIX, VMS, and Microsoft Windows ports of Perl, and implemented on the Macintosh only in the MPW environment.

Opening a Pipe

The two-argument form of open() is used to open pipes. As before, the first argument is the name of a filehandle chosen by you. The second argument, however, is a program and all its arguments, either preceded or followed by the pipe " | " symbol. The command should be entered exactly as you would type it in the operating system's default shell, which for UNIX machines is the Bourne shell ("sh") and the DOS/NT command shell on Microsoft Windows systems. You may specify the full path to the command, for example /usr/bin/ls, or rely on the PATH environment variable to find the command for you.

If the pipe symbol precedes the program name, then the filehandle is opened for writing and everything written to the filehandle is sent to the standard input of the program. If the pipe symbol follows the program, then the filehandle is opened for reading, and everything read from the filehandle is taken from the program's standard output.

For example, in UNIX the command ls -l will return a listing of the files in the current directory. By passing an argument of " ls -l | " to open(), we can open a pipe to read from the command:

open (LSFH,"ls -l |") or die "Can't open ls -l: $!";
 while (my $line = <LSFH>) {
   print "I saw: $line\n";
 }
 close LSFH;
 

This fragment simply echoes each line produced by the ls -l command. In a real application, you'd want to do something more interesting with the information.

As an example of an output pipe, the UNIX wc -lw command will count the lines (option " -l ") and words (option " -w ") of a text file sent to it on standard input. This code fragment opens a pipe to the command, writes a few lines of text to it, and then closes the pipe. When the program runs, the word and line counts produced by wc are printed in the command window:

open (WC,"| wc -lw") or die "Can't open wordcount: $!";
 print WC "This is the first line.\n";
 print WC "This is the another line.\n";
 print WC "This is the last line.\n";
 print WC "Oops. I lied.\n";
 close WC;
 

IO::Filehandle supports pipes through its open() method:

$wc = IO::Filehandle->open("| wc - lw") or die "Can't open wordcount:
 $!";
 

Using Pipes

Let's look at a complete functional example (Figure 2.2). The program whos_there.pl opens up a pipe to the UNIX who command and counts the number of times each user is logged in. It produces a report like this one:

% whos_there.pl
   jsmith 9
      abu 5
   lstein 1
  palumbo 1
 

This indicates that users "jsmith" and "abu" are logged in 9 and 5 times, respectively, while "lstein" and "palumbo" are each logged in once. The users are sorted in descending order of the number of times they are logged in. This is the sort of script that might be used by an administrator of a busy system to watch usage.

Lines 13: Initialize script We turn on strict syntax checking with use strict. This catches mistyped variables, inappropriate use of globals, failure to quote strings, and other potential errors. We create a local hash %who to hold the set of logged-in users and the number of times they are logged in.

Line 4: Open pipe to who command We call open() on a filehandle named WHOFH, using who | as the second argument. If the open() call fails, die with an error message.

Lines 58: Read the output of the who command We read and process the output of who one line at a time. Each line of who looks like this:

jsmith pts/23 Aug 12 10:26 (cranshaw.cshl.org)
 

The fields are the username, the name of the terminal he's using, the date he logged in, and the address of the remote machine he logged in from (this format will vary slightly from one dialect of UNIX to another). We use a pattern match to extract the username, and we tally the names into the %who hash in such a way that the usernames become the keys, and the number of times each user is logged in becomes the value.

The <WHOFH> loop will terminate at the EOF, which in the case of pipes occurs when the program at the other end of the pipe exits or closes its standard output.

Lines 911: Print out the results We sort the keys of %who based on the number of times each user has logged in, and print out each username and login count. The printf() format used here, " %10s %d\n ", tells printf() to format its first argument as a string that is right justified on a field 10 spaces long, to print a space, and then to print the second argument as a decimal integer.

Line 12: Close the pipe We are done with the pipe now, so we close() it. If an error is detected during close, we print out a warning.

With pipes, the open() and close() functions are enhanced slightly to provide additional information about the subprocess. When opening a pipe, open() returns the process ID (PID) of the command at the other end of the pipe. This is a unique nonzero integer that can be used to monitor and control the subprocess with signals (which we discuss in detail later in the Handling Signals section). You can store this PID, or you can ignore its special meaning and treat the return value from open() as a Boolean flag.

When closing a pipe, the close() call is enhanced to place the exit code from the subprocess in the special global variable $?. Contrary to most Perl conventions, $? is zero if the command succeeded, and nonzero on an error. The perlvar POD page has more to say about the exit code, as does the section Handling Child Termination in Chapter 10.

Another aspect of close() is that when closing a write pipe, the close() call will block until the process at the other end has finished all its work and exited. If you close a read pipe before reading to the EOF, the program at the other end will get a PIPE signal (see The PIPE Signal) the next time it tries to write to standard output.

Pipes Made Easy: The Backtick Operator

Perl's backtick operator, (`), is an easy way to create a one-shot pipe for reading a program's output. The backtick acts like the double-quote operator, except that whatever is contained between the backticks is interpreted as a command to run. For example:

$ls_output = `ls`;
 

This will run the ls (directory listing) command, capture its output, and assign the output to the $ls_output scalar.

Internally, Perl opens a pipe to the indicated command, reads everything it prints to standard output, closes the pipe, and returns the command output as the operator result. Typically at the end of the result there is a new line, which can be removed with chomp().

Just like double quotes, backticks interpolate scalar variables and arrays. For example, we can create a variable containing the arguments to pass to ls like this:

$arguments = '-l -F';
 $ls_output = `ls $arguments`;
 

The command's standard error is not redirected by backticks. If the subprocess writes any diagnostic or error messages, they will be intermingled with your program's diagnostics. On UNIX systems, you can use the Bourne shell's output redirection system to combine the subprocess's standard error with its standard output like this:

$ls_output = `ls 2>&1`;
 

Now $ls_output will contain both the standard error and the standard output of the command.

Pipes Made Powerful: The pipe() Function

A powerful but slightly involved way to create a pipe is with Perl's built-in pipe() function. pipe() creates a pair of filehandles: one for reading and one for writing. Everything written to the one filehandle can be read from the other.

$result = pipe (READHANDLE,WRITEHANDLE) Open a pair of filehandles connected by a pipe. The first argument is the name of a filehandle to read from, and the second is a filehandle to write to. If successful, pipe() returns a true result code.

Why is pipe() useful? It is commonly used in conjunction with the fork() function in order to create a parent-child pair that can exchange data. The parent process keeps one filehandle and closes the other, while the child does the opposite. The parent and child process can now communicate across the pipe as they work in parallel.

A short example will illustrate the power of this technique. Given a positive integer, the facfib.pl script calculates its factorial and the value of its position in the Fibonacci series. To take advantage of modern multiprocessing machines, these calculations are performed in two subprocesses so that both calculations proceed in parallel. The script uses pipe() to create filehandles that the child processes can use to communicate their findings to the parent process that launched them. When we run this program, we may see results like this:

% facfib.pl 8
 factorial(1) => 1
 factorial(2) => 2
 factorial(3) => 6
 factorial(4) => 24
 factorial(5) => 120
 fibonacci(1) => 1
 factorial(6) => 720
 fibonacci(2) => 1
 factorial(7) => 5040
 fibonacci(3) => 2
 factorial(8) => 40320
 fibonacci(4) => 3
 fibonacci(5) => 5
 fibonacci(6) => 8
 fibonacci(7) => 13
 fibonacci(8) => 21
 

The results from the factorial and Fibonacci calculation overlap because they are occurring in parallel.

Figure 2.3 shows how this program works.

Lines 13: Initialize module We turn on strict syntax checking and recover the command-line argument. If no argument is given, we default to 10.

Line 4: Create linked pipes We create linked pipes with pipe(). READER will be used by the main (parent) process to read results from the children, which will use WRITER to write their results.

Lines 510: Create first child process We call fork() to clone the current process. In the parent process, fork() returns the nonzero PID of the child process. In the child process, fork() returns numeric 0. If we see that the result of fork() is 0, we know we are the child process. We close the READER filehandle because we don't need it. We select() WRITER, making it the default filehandle for output, and turn on autoflush mode by setting $| to a true value. This is necessary to ensure that the parent process gets our messages as soon as we write them.

We now call the factorial() subroutine with the integer argument from the command line. After this, the child process is done with its work, so we exit(). Our copy of WRITER is closed automatically.

Lines 1116: Create the second child process Back in the parent process, we invoke fork() again to create a second child process. This one, however, calls the fibonacci() subroutine rather than factorial().

Lines 1719: Process messages from children In the parent process, we close WRITER because we no longer need it. We read from READER one line at a time, and print out the results. This will contain lines issued by both children. READER returns undef when the last child has finished and closed its WRITER filehandle, sending us an EOF. We could close() READER and check the result code, or let Perl close the filehandle when we exit, as we do here.

Lines 2025: The factorial() subroutine We calculate the factorial of the subroutine argument in a straightforward iterative way. For each step of the calculation, we print out the intermediate result. Because WRITER has been made the default filehandle with select(), each print() statement enters the pipe, where it is ultimately read by the parent process.

Lines 2634: The fibonacci() subroutine This is identical to factorial() except for the calculation itself.

Instead of merely echoing its children's output, we could have the parent do something more useful with the information. We use a variant of this technique in Chapter 14 to implement a preforked Web server. The parent Web server manages possibly hundreds of children, each of which is responsible for processing incoming Web requests. To tune the number of child processes to the incoming load, the parent monitors the status of the children via messages that they send via a pipe launching more children under conditions of high load, and killing excess children when the load is low.

The pipe() function can also be used to create a filehandle connected to another program in much the way that piped open() does. We don't use this technique elsewhere, but the general idea is for the parent process to fork(), and for the child process to reopen either STDIN or STDOUT onto one of the paired filehandles, and then exec() the desired program with arguments. Here's the idiom:

pipe(READER,WRITER) or die "pipe no good: $!";
 my $child = fork();
 die "Can't fork: $!" unless defined $child;
 if ($child == 0) { # child process
    close READER;              # child doesn't need this
    open (STDOUT,">&WRITER");  # STDOUT now goes to writer
    exec $cmd,$args;
    die "exec failed: $!";
 }
 close WRITER;  # parent doesn't need this
 

At the end of this code, READER will be attached to the standard output of the command named $cmd, and the effect is almost exactly identical to this code:

open (READER,"$cmd $args |") or die "pipe no good: $!";
 

Bidirectional Pipes

Both piped open() and pipe() create unidirectional filehandles. If you want to both read and write to another process, you're out of luck. In particular, this sensible-looking syntax does not work:

open(FH,"| $cmd |");
 

One way around this is to call pipe() twice, creating two pairs of linked filehandles. One pair is used for writing from parent to child, and the other for child to parent, rather like a two-lane highway. We won't go into this technique, but it's what the standard IPC::Open2 and IPC::Open3 modules do to create a set of filehandles attached to the STDIN, STDOUT, and STDERR of a subprocess.

A more elegant way to create a bidirectional pipe is with the socketpair() function. This creates two linked filehandles like pipe() does, but instead of being a one-way connection, both filehandles are read/write. Data written into one filehandle comes out the other one, and vice versa. Because the socketpair() function involves the same concepts as the socket() function used for network communications, we defer our discussion of it until Chapter 4.

Distinguishing Pipes from Plain Filehandles

You will occasionally need to test a filehandle to see if it is opened on a file or a pipe. Perl's filehandle tests make this possible (Table 2.1).

Table 2.1. Perl's Filehandle Tests

Test	Description
-p	Filehandle is a pipe.
-t	Filehandle is opened on a terminal.
-s	Filehandle is a socket.

If a filehandle is opened on a pipe, the -p test will return true:

print "I've got a pipe!\n" if -p FILEHANDLE;
 

The -t and -S file tests can distinguish other special types of filehandle. If a filehandle is opened on a terminal (the command-line window), then -t will return true. Programs can use this to test STDIN to see if the program is being run interactively or has its standard input redirected from a file:

print "Running in batch mode, confirmation prompts disabled.\n"
       unless -t STDIN;
 

The -S test detects whether a filehandle is opened on a network socket (introduced in Chapter 3):

print "Network active.\n" if -S FH
 

There are more than a dozen other file test functions that can give you a file's size, modification date, ownership, and other information. See the perlfunc POD page for details.

The Dreaded PIPE Error

When your script is reading from a filehandle opened on a pipe, and the program at the other end either exits or simply closes its end of the pipe, your program will receive an EOF on the filehandle. What happens in the opposite case, when your script is writing to a pipe and the program at the other end terminates prematurely or closes its end of the connection?

To find out, we can write two short Perl scripts. One, named write_ten.pl, opens up a pipe to the second program and attempts to write ten lines of text to it. The script checks the result code from print(), and bumps up a variable named $count whenever print() returns a true result. When write_ten.pl is done, it displays the contents of $count, indicating the number of lines that were successfully written to the pipe. The second program, named read_three.pl, reads three lines of text from standard input and then exits.

The two scripts are shown in Figures 2.4 and 2.5. Of note is that write_ten.pl puts the pipe into autoflush mode so that each line of text is sent down the pipe immediately, rather than being buffered locally. write_ten.pl also sleep()s for one second after writing each line of text, giving read_three.pl a chance to report that the text was received. Together, these steps make it easier for us to see what is happening. When we run write_ten.pl we see the following:

% write_ten.pl
 Writing line 1
 Read_three got: This is line number 1
 Writing line 2
 Read_three got: This is line number 2
 Writing line 3
 Read_three got: This is line number 3
 Writing line 4
 Broken pipe
 %
 

Everything works as expected through line three, at which point read_three.pl exits. When write_ten.pl attempts to write the fourth line of text, the script crashes with a Broken pipe error. The statement that prints out the number of lines successfully passed to the pipe is never executed.

When a program attempts to write to a pipe and no program is reading at the other end, this results in a PIPE exception. This exception, in turn, results in a PIPE signal being delivered to the writer. By default this signal results in the immediate termination of the offending program. The same error occurs in network applications when the sender attempts to transmit data to a remote program that has exited or has stopped receiving.

To deal effectively with PIPE, you must install a signal handler, and this brings us to the next major topic.

Signals

As with filehandles, understanding signals is fundamental to network programming. A signal is a message sent to your program by the operating system to tell it that something important has occurred. A signal can indicate an error in the program itself such as an attempt to divide by zero, an event that requires immediate attention such as an attempt by the user to interrupt the program, or a noncritical informational event such as the termination of a subprocess that your program has launched.

In addition to signals sent by the operating system, processes can signal each other. For example, when the user presses control-C (^C) on the keyboard to send an interrupt signal to the currently running program, that signal is sent not by the operating system, but by the command shell that pro cesses and interprets keystrokes. It is also possible for a process to send signals to itself.

Common Signals

The POSIX standard defines nineteen signals. Each has a small integer value and a symbolic name. We list them in Table 2.2 (the gaps in the integer sequence represent nonstandard signals used by some systems).

The third column of the table indicates what happens when a process receives the signal. Some signals do nothing. Others cause the process to terminate immediately, and still others terminate the process and cause a core dump. Most signals can be "caught." That is, the program can install a handler for the signal and take special action when the signal is received. A few signals, however, cannot be intercepted in this way.

You don't need to understand all of the signals listed in Table 2.2 because either they won't occur during the execution of a Perl script, or their generation indicates a low-level bug in Perl itself that you can't do anything about. However, a handful of signals are relatively common, and we'll look at them in more detail now.

HUP signals a hangup event. This typically occurs when a user is running a program from the command line, and then closes the command-line window or exits the interpreter shell. The default action for this signal is to terminate the program.

INT signals a user-initiated interruption. It is generated when the user presses the interrupt key, typically ^C. The default behavior of this signal is to terminate the program. QUIT is similar to INT, but also causes the program to generate a core file (on UNIX systems). This signal is issued when the user presses the "quit" key, ordinarily ^\.

Table 2.2. POSIX Signals

Signal Name	Value	Notes	Comment
HUP	1	A	Hangup detected
INT	2	A	Interrupt from keyboard
QUIT	3	A	Quit from keyboard
ILL	4	A	Illegal Instruction
ABRT	6	C	Abort
FPE	8	C	Floating point exception
KILL	9	AF	Termination signal
USR1	10	A	User-defined signal 1
SEGV	11	C	Invalid memory reference
USR2	12	A	User-defined signal 2
PIPE	13	A	Write to pipe with no readers
ALRM	14	A	Timer signal from alarm clock
TERM	15	A	Termination signal
CHLD	17	B	Child terminated
CONT	18	E	Continue if stopped
STOP	19	DF	Stop process
TSTP	20	D	Stop typed at tty
TTIN	21	D	tty input for background process
TTOU	22	D	tty output for background process
Notes: ADefault action is to terminate process. BDefault action is to ignore the signal. CDefault action is to terminate process and dump core. DDefault action is to stop the process. EDefault action is to resume the process. FSignal cannot be caught or ignored.

By convention, TERM and KILL are used by one process to terminate another. By default, TERM causes immediate termination of the program, but a program can install a signal handler for TERM to intercept the terminate request and possibly perform some cleanup actions before quitting. The KILL signal, in contrast, is uncatchable. It causes an immediate shutdown of the process without chance of appeal. For example, when a UNIX system is shutting down, the script that handles the shutdown process first sends a TERM to each running process in turn, giving it a chance to clean up. If the process is still running a few tens of seconds later, then the shutdown script sends a KILL.

The PIPE signal is sent when a program writes to a pipe or socket but the process at the remote end has either exited or closed the pipe. This signal is so common in networking applications that we will look at it closely in the Handling PIPE Exceptions section.

ALRM is used in conjunction with the alarm() function to send the program a prearranged signal after a certain amount of time has elapsed. Among other things, ALRM can be used to time out blocked I/O calls. We will see examples of this in the Timing Out Long-Running Operations section.

CHLD occurs when your process has launched a subprocess, and the status of the child has changed in some way. Typically the change in status is that the child has exited, but CHLD is also generated whenever the child is stopped or continued. We discuss how to deal with CHLD in much greater detail in Chapters 4 and 9.

STOP and TSTP both have the effect of stopping the current process. The process is put into suspended animation indefinitely; it can be resumed by sending it a CONT signal. STOP is generally used by one program to stop another. TSTP is issued by the interpreter shell when the user presses the stop key (^Z on UNIX systems). The other difference between the two is that TSTP can be caught, but STOP cannot be caught or ignored.

Catching Signals

You can catch a signal by adding a signal handler to the %SIG global hash. Use the name of the signal you wish to catch as the hash key. For example, use $SIG{INT} to get or set the INT signal handler. As the value, use a code reference: either an anonymous subroutine or a reference to a named subroutine. For example, Figure 2.6 shows a tiny script that installs an INT handler. Instead of terminating when we press the interrupt key, it prints out a short message and bumps up a counter. This goes on until the script counts three interruptions, at which point it finally terminates. In the transcript that follows, the "Don't interrupt me!" message was triggered each time I typed ^C:

% interrupt.pl
 I'm sleeping.
 I'm sleeping.
 Don't interrupt me! You've already interrupted me 1x.
 I'm sleeping.
 I'm sleeping.
 Don't interrupt me! You've already interrupted me 2x.
 I'm sleeping.
 Don't interrupt me! You've already interrupted me 3x.
 

Let's look at the script in detail.

Lines 13: Initialize script We turn on strict syntax checking, and declare a global counter named $interruptions. This counter will keep track of the number of times the script has received INT.

Line 4: Install INT handler We install a handler for the INT signal by setting $SIG{INT} to a reference to the subroutine handle_interruptions().

Lines 58: Main loop The main loop of the program simply prints a message and calls sleep with an argument of 5. This puts the program to sleep for 5 seconds, or until a signal is received. This continues until the $interruptions counter becomes 3 or greater.

Lines 912: The handle_interruptions() subroutine The handle_interruptions() subroutine is called whenever the INT signal arrives, even if the program is busy doing something else at that moment. In this case, our signal handler bumps up $interruptions and prints out a warning.

For short signal handlers, you can use an anonymous subroutine as the handler. For example, this code fragment is equivalent to that in Figure 2.6, but we don't have to formally name the handler subroutine:

$SIG{INT} = sub {
                 $interruptions++;
                 warn "Don't interrupt me! You've already interrupted
                 me ${interruptions}x.\n";
                  };
 

In addition to code references, %SIG recognizes two special cases. The string " DEFAULT " restores the default behavior of the signal. For example, setting $SIG{INT} to " DEFAULT " will cause the INT signal to terminate the script once again. The string " IGNORE " will cause the signal to be ignored altogether.

As previously mentioned, don't bother installing a handler for either KILL or STOP. These signals can be neither caught nor ignored, and their default actions will always be performed.

If you wish to use the same routine to catch several different signals, and it is important for the subroutine to distinguish one signal from another, it can do so by looking at its first argument, which will contain the name of the signal. For example, for INT signals, the handler will be called with the string " INT ":

$SIG{TERM} = $SIG{HUP} = $SIG{INT} = \&handler
 sub handler {
    my $sig = shift;
    warn "Handling a $sig signal.\n";
 }
 

Handling PIPE Exceptions

We now have what we need to deal with PIPE exceptions. Recall the write_ ten.pl and read_three.pl examples from Figures 2.4 and 2.5 in The Dreaded PIPE Error section. write_ten.pl opens a pipe to read_three.pl and tries to write ten lines of text to it, but read_three.pl is only prepared to accept three lines, after which it exits and closes its end of the pipe. write_ten.pl, not knowing that the other end of the connection has exited, attempts to write a fourth line of text, generating a PIPE signal.

We will now modify write_ten.pl so that it detects the PIPE error and handles it more gracefully. We will use variants on this technique in later chapters that deal with common issues in network communications.

The first technique is shown in Figure 2.7, the write_ten_ph.pl script. Here we set a global flag, $ok, which starts out true. We then install a PIPE handler using this code:

$SIG{PIPE} = sub { undef $ok };
 

When a PIPE signal is received, the handler will undefine the $ok flag, making it false.

The other modification is to replace the simple for() loop in the original version with a more sophisticated version that checks the status of $ok. If the flag becomes false, the loop exits. When we run the modified script, we see that the program runs to completion, and correctly reports the number of lines successfully written:

% write_ten_ph.pl
 Writing line 1
 Read_three got: This is line number 1
 Writing line 2
 Read_three got: This is line number 2
 Writing line 3
 Read_three got: This is line number 3
 Writing line 4
 Wrote 3 lines of text
 

Another general technique is to set $SIG{PIPE} to 'IGNORE', in order to ignore the PIPE signal entirely. It is now our responsibility to detect that something is amiss, which we can do by examining the result code from print(). If print() returns false, we exit the loop.

Figure 2.8 shows the code for write_ten_i.pl, which illustrates this technique. This script begins by setting $SIG{PIPE} to the string 'IGNORE', suppressing PIPE signals. In addition, we modify the print loop so that if print() is successful, we bump up $count as before, but if it fails, we issue a warning and exit the loop via last.

When we run write_ten_i.pl we get this output:

% write_ten_i.pl
 Writing line 1
 Read_three got: This is line number 1
 Writing line 2
 Read_three got: This is line number 2
 Writing line 3
 Read_three got: This is line number 3
 Writing line 4
 An error occurred during writing: Broken pipe
 Wrote 3 lines of text
 

Notice that the error message that appears in $! after the unsuccessful print is "Broken pipe." If we wanted to treat this error separately from other I/O errors, we could explicitly test its value via a pattern match, or, better still, check its numeric value against the numeric error constant EPIPE. For example:

use Errno ':POSIX';
 ...
 unless (print PIPE "This is line number $_\n") { # handle write error
    last if $! == EPIPE;   # on PIPE, just terminate the loop
    die "I/O error: $!";   # otherwise die with an error message
 }
 

Sending Signals

A Perl script can send a signal to another process using the kill() function:

$count = kill($signal,@processes) The kill() function sends signal $signal to one or more processes. You may specify the signal numerically, for example 2, or symbolically as in " INT ". @processes is a list of one or more process IDs to deliver the signal to. The number of processes successfully signaled is returned as the kill() function result.

One process can only signal another if it has sufficient privileges to do so. In general, a process running under a normal user's privileges can signal only other processes that are running under the same user's privileges. A process running with root or superuser privileges, however, can signal any other process.

The kill() function provides a few tricks. If you use the special signal number 0, then kill() will return the number of processes that could have been signaled, without actually delivering the signal. If you use a negative number for the process ID, then kill() will treat the absolute value of the number as a process group ID and deliver the signal to all members of the group.

A script can send a signal to itself by calling kill() on the variable $$, which holds the current process ID. For example, here's a fancy way for a script to commit suicide:

kill INT => $$; # same as kill('INT',$$)
 

Signal Handler Caveats

Because a signal can arrive at any time during a program's execution, it can arrive while the Perl interpreter is doing something important, like updating one of its memory structures, or even while inside a system call. If the signal handler does something that rearranges memory, such as allocating or dis posing of a big data structure, then on return from the handler Perl may find its world changed from underneath it, and get confused, which occasionally results in an ugly crash.

To avoid this possibility, signal handlers should do as little as possible. The safest course is to set a global variable and return, as we did in the PIPE handler in Figure 2.7. In addition to memory-changing operations, I/O operations within signal handlers should also be avoided. Although we liberally pepper our signal handlers with diagnostic warn() statements throughout this book, these operations should be stripped out in production programs.

It's generally OK to call die() and exit() within signal handlers. The exception is on Microsoft Windows systems, where due to limitations in the signal library, these two calls may cause "Dr. Watson" errors if invoked from within a signal handler.

Indeed, the implementation of signals on Windows systems is currently extremely limited. Simple things, such as an INT handler to catch the interrupt key, will work. More complex things, such as CHLD handlers to catch the death of a subprocess, do not work. This is an area of active development so be sure to check the release notes before trying to write or adapt any code that depends heavily on signals.

Signal handling is not implemented in MacPerl.

Timing Out Slow System Calls

A signal may occur while Perl is executing a system call. In most cases, Perl automatically restarts the call and it takes up exactly where it left off.

A few system calls, however, are exceptions to this rule. One is sleep(), which suspends the script for the indicated number of seconds. If a signal interrupts sleep(), however, it will exit early, returning the number of seconds it slept before being awakened. This property of sleep() is quite useful because it can be used to put the script to sleep until some expected event occurs.

$slept = sleep([$seconds]) Sleep for the indicated number of seconds or until a signal is received. If no argument is provided, this function will sleep forever. On return, sleep() will return the number of seconds it actually slept.

Another exception is the four-argument version of select(), which can be used to perform a timed wait until one or more of a set of filehandles are ready for I/O. This function is described in detail in Chapter 12.

Sometimes the automatic restarting of system calls is not what you want. For example, consider an application that prompts a user to type her password and tries to read the response from standard input. You might want the read to time out after some period of time in case the user has wandered off and left the terminal unattended. This fragment of code might at first seem to do the trick:

my $timed_out = 0; 
 
 $SIG{ALRM} = sub { $timed_out = 1 };
 
 print STDERR "type your password: ";
 alarm (5);    # five second timeout
 my $password = <STDIN>;
 alarm (0);
 
 print STDERR "you timed out\n" if $timed_out;
 

Here we use the alarm() function to set a timer. When the timer expires, the operating system generates an ALRM signal, which we intercept with a handler that sets the $timed_out global to true. In this code we call alarm() with a five-second timeout, and then read a line of input from standard input. After the read completes, we call alarm() again with an argument of zero, turning the timer off. The idea is that the user will have five seconds in which to type a password. If she doesn't, the alarm clock goes off and we fall through to the rest of the program.

$seconds_left = alarm($seconds) Arrange for an ALRM signal to be delivered to the process after $seconds. The function result is the number of seconds left from the previous timer, if any. An argument of zero disables the timer.

The problem is that Perl automatically restarts slow system calls, including <>. Even though the alarm clock has gone off, we remain in the <> call, waiting for the user's keyboard input.

The solution to this problem is to use eval{} and a local ALRM handler to abort the read. The general idiom is this:

print STDERR "type your password: ";
 my $password =
   eval {
     local $SIG{ALRM} = sub { die "timeout\n" };
     alarm (5);    # five second timeout
     return <STDIN>;
   };
 alarm (0);
 print STDERR "you timed out\n" if $@ =~ /timeout/;
 

Instead of having an ALRM handler in the main body of the program, we localize it within an eval{} block. The eval{} block sets the alarm, as before, and attempts to read from STDIN. If <> returns before the timer goes off, then the line of input is returned from the eval{} block, and assigned to $password.

However, if the timer goes off before the input is complete, the ALRM handler executes, dying with the error message "timeout." However, since we are dying within an eval{} block, the effect of this is for eval{} to return undef, setting the variable $@ to the last error message. We pattern match $@ for the timeout message, and print a warning if found.

In either case, we turn off the timer immediately after returning from the eval{} block in order to avoid having the timer go off at an inconvenient moment.

We will use this technique several times in later chapters when we need to time out slow network calls.

Chapter 3. Introduction to Berkeley Sockets

This chapter introduces Perl's version of the Berkeley socket API, the low-level network API that underlies all of Perl's networking modules. It explains the different kinds of sockets, demonstrates basic Berkeley-based clients and servers, and discusses some of the common traps programmers encounter when first working with the API.

Clients, Servers, and Protocols

Network communication occurs when two programs exchange data across the net. With rare exceptions, the two programs are not equal. One, the client, initiates the connection and is usually, but not always, connected to a single server at a time. The other partner in the connection, the server, is passive, waiting quietly until it is contacted by a client seeking a connection. In contrast to clients, it is common for a server to service incoming connections from multiple clients at the same time.

Although it is often true that the computer ("host") that runs the server is larger and more powerful than the client machine, this is not a rule by any means. In fact, in some popular applications, such as the X Windows System, the situation is reversed. The server is usually a personal computer or other desktop machine, while the client is run on a more powerful "server class" machine.

Most of the network protocols that we are familiar with are client-server applications. This includes the HTTP used on the Web, the SMTP used for Internet e-mail, and all the database access protocols. However, a small but growing class of network applications is peer-to-peer. In peer-to-peer scenarios, the two ends of the connection are equivalent, each making multiple connections to other copies of the same program. The controversial Napster file-sharing protocol is peer-to-peer, as are its spiritual heirs Gnutella and Freenet.

Protocols

We've thrown around the word protocol, but what is it, exactly? A protocol is simply an agreed-upon set of standards whereby two software components interoperate. There are protocols at every level of the networking stack (Figure 3.1).

At the lowest level is the hardware or datalink layer, where, for example, the drivers built into Ethernet network interface cards have a common understanding of how to interpret the pulses of electric potential on the network wire in terms of Ethernet frames, how to detect when the wire is in use by another card, and how to detect and resolve collisions between two cards trying to transmit at the same time.

One level up is the network layer. At this layer, information is grouped into packets that consist of a header containing the sender and recipient's address, and a payload that consists of the actual data to send. Payloads are typically in the range of 500 bytes to 1500 bytes. Internet routers act at the IP layer by reading packet headers and figuring out how to route them to their destinations. The main protocol at this layer is the Internet Protocol, or IP.

The transport layer is concerned with creating data packets and ensuring the integrity of their contents. The two important protocols at this layer are the Transmission Control Protocol (TCP), which provides reliable connection-oriented communications, and the User Datagram Protocol (UDP), which provides an unreliable message-oriented service. These protocols are responsible for getting the data to its destination. They don't care what is actually inside the data stream.

At the top of the stack is the application layer, where the content of the data stream does matter. There is an abundance of protocols at this level, including such familiar and unfamiliar names as HTTP, FTP, SMTP, POP3, IMAP, SNMP, XDMCP, and NNTP. These protocols specify, sometimes in excruciating detail, how a client should contact a server, what messages are allowed, and what information to exchange with each message.

The combination of the network layer and the transport layer is known as TCP/IP, named after the two major protocols that operate at those layers.

Binary versus Text-Oriented Protocols

Before they can exchange information across the network, hosts have a fundamental choice to make. They can exchange data either in binary form or as human-readable text. The choice has far-reaching ramifications.

To understand this, consider exchanging the number 1984. To exchange it as text, one host sends the other the string 1984, which, in the common ASCII character set, corresponds to the four hexadecimal bytes 0x31 0x39 0x38 0x34. These four bytes will be transferred in order across the network, and (provided the other host also speaks ASCII) will appear at the other end as "1984".

However, 1984 can also be treated as a number, in which case it can fit into the two-byte integer represented in hexadecimal as 0x7C0. If this number is already stored in the local host as a number, it seems sensible to transfer it across the network in its native two-byte form rather than convert it into its four-byte text representation, transfer it, and convert it back into a two-byte number at the other end. Not only does this save some computation, but it uses only half as much network capacity.

Unfortunately, there's a hitch. Different computer architectures have different ways of storing integers and floating point numbers. Some machines use two-byte integers, others four-byte integers, and still others use eight-byte integers. This is called word size. Furthermore, computer architectures have two different conventions for storing integers in memory. In some systems, called big-endian architectures, the most significant part of the integer is stored in the first byte of a two-byte integer. On such systems, reading from low to high, 1984 is represented in memory as the two bytes:

0x07    0xC0
 low  -> high
 

On little-endian architectures, this convention is reversed, and 1984 is stored in the opposite orientation:

0xC0    0x07
 low  -> high
 

These architectures are a matter of convention, and neither has a significant advantage over the other. The problem comes when transferring such data across the network, because this byte pair has to be transferred serially as two bytes. Data in memory is sent across the network from low to high, so for big-endian machines the number 1984 will be transferred as 0x07 0xC0, while for little-endian machines the numbers will be sent in the reverse order. As long as the machine at the other end has the same native word size and byte order, these bytes will be correctly interpreted as 1984 when they arrive. However, if the recipient uses a different byte order, then the two bytes will be interpreted in the wrong order, yielding hexadecimal 0xC007, or decimal 49,159. Even worse, if the recipient interprets these bytes as the top half of a four-byte integer, it will end up as 0xC0070000, or 3,221,684,224. Someone's anniversary party is going to be very late.

Because of the potential for such binary chaos, text-based protocols are the norm on the Internet. All the common protocols convert numeric information into text prior to transferring them, even though this can result in more data being transferred across the net. Some protocols even convert data that doesn't have a sensible text representation, such as audio files, into a form that uses the ASCII character set, because this is generally easier to work with. By the same token, a great many protocols are line-oriented, meaning that they accept commands and transmit data in the form of discrete lines, each terminated by a commonly agreed-upon newline sequence.

A few protocols, however, are binary. Examples include Sun's Remote Procedure Call (RPC) system, and the Napster peer-to-peer file exchange protocol. Such protocols have to be exceptionally careful to represent binary data in a common format. For integer numbers, there is a commonly recognized network format. In network format, a "short" integer is represented in two big-endian bytes, while a "long" integer is represented with four big-endian bytes. As we will see in Chapter 19, Perl's pack() and unpack () functions provide the ability to convert numbers into network format and back again.

Floating point numbers and more complicated things like data structures have no commonly accepted network representation. When exchanging binary data, each protocol has to work out its own way of representing such data in a platform-neutral fashion.

We will stick to text-based protocols for much of this book. However, to give you a taste for what it's like to use a binary protocol, the UDP-based real-time chat system in Chapters 19 and 20 exchanges platform-neutral binary messages.

Berkeley Sockets

Berkeley sockets are part of an application programming interface (API) that specifies the data structures and function calls that interact with the operating system's network subsystem. The name derives from the origins of the API in release 4.2 of the Berkeley Standard Distribution (4.2BSD) of UNIX. Berkeley sockets act at the transport layer: They help get the data where it's going, but have nothing to say about the content of the data.

Berkeley sockets are part of an API, not a specific protocol, which defines how the programmer interacts with an idealized network. Although strongly associated with the TCP/IP network protocol for which the API was first designed, the Berkeley sockets API is generic enough to support other types of network, such as Novell Netware, Windows NT networking, and Appletalk.

Perl provides full support for Berkeley sockets on most of the platforms it runs on. On certain platforms such as the Windows and Macintosh ports, extension modules also give you access to the non-Berkeley APIs native to those machines. However, if you're interested in writing portable applications you'll probably want to stick to the Berkeley API.

The Anatomy of a Socket

A socket is an endpoint for communications, a portal to the outside world that we can use to send outgoing messages to other processes, and to receive incoming traffic from processes interested in sending messages to us.

To create a socket, we need to provide the system with a minimum of three pieces of information.

The Socket's Domain

The domain defines the family of networking protocols and addressing schemes that the socket will support. The domain is selected from a small number of integer constants defined by the operating system and exported by Perl's Socket module. There are only two common domains (see Table 3.1).

AF_INET is used for TCP/IP networking. Sockets in this domain use IP addresses and port numbers as their addressing scheme (more on this later). AF_UNIX is used only for interprocess communication within a single host. Addresses in this domain are file pathnames. The name AF_UNIX is unfairly UNIX-specific; it's possible for non-UNIX systems to implement it. For this reason, POSIX has tried to rename this constant AF_LOCAL, although few systems have followed suit.

Table 3.1. Common Socket Domains

Constant	Description
AF_INET	The Internet protocols
AF_UNIX	Networking within a single host

In addition to these domains, there are many others including AF_APPLETALK, AF_IPX, and AF_X25, each corresponding to a particular addressing scheme. AF_INET6, corresponding to the long addresses of TCP/IP version 6, will become important in the future, but is not yet supported by Perl.

The AF_ prefix stands for "address family." In addition, there is a series of "protocol family" constants starting with the PF_ prefix. For example, there is a PF_INET constant that corresponds to AF_INET. These constants evaluate to the same value and can, in fact, be used interchangeably. The distinction between them is a historical artifact, and you'll find that published code sometimes uses the one and sometimes the other.

The Socket's Type

The socket type identifies the basic properties of socket communications. As explained more fully in the next section, sockets can be either a "stream" type, in which case data is sent through a socket in a continuous stream (like reading or writing to a file, for example), or a "datagram" type, in which case data is sent and received in discrete packages.

The type is controlled by an operating-system-defined constant that evaluates to a small integer. Common constants exported by Socket are shown in Table 3.2.

Table 3.2. Constants Exported by Socket

Constant	Description
SOCK_STREAM	A continuous stream of data
SOCK_DGRAM	Individual packets of data
SOCK_RAW	Access to internal protocols and interfaces

Perl fully supports the SOCK_STREAM and SOCK_DGRAM socket types. SOCK_RAW is supported through an add-on module named Net::Raw.

The Socket's Protocol

For a given socket domain and type, there may be one or several protocols that implement the desired behavior. Like the domain and socket type, the protocol is a small integer. However, the protocol numbers are not available as constants, but instead must be looked up at run time using the Perl getprotobyname() function. Some of the protocols are listed in Table 3.3.

Table 3.3. Some Socket Protocols

Protocol	Description
tcp	Transmission Control Protocol for stream sockets
udp	User Datagram Protocol for datagram sockets
icmp	Internet Control Message Protocol
raw	Creates IP packets manually

The TCP and UDP protocols are supported directly by the Perl sockets API. You can get access to the ICMP and raw protocols via the Net::ICMP and Net::Raw third-party modules, but we do not discuss them in this book (it would be possible, but probably not sensible, to reimplement TCP in Perl using raw packets).

Generally there exists a single protocol to support a given domain and type. When creating a socket, you must be careful to set the domain and socket type to match the protocol you've selected. The possible combinations are summarized in Table 3.4.

Table 3.4. Allowed Combinations of Socket Type and Protocol in the INET and UNIX Domains

Domain	Type	Protocol
AF_INET	SOCK_STREAM	tcp
AF_INET	SOCK_DGRAM	udp
AF_UNIX	SOCK_STREAM	PF_UNSPEC
AF_UNIX	SOCK_DGRAM	PF_UNSPEC

The allowed combinations of socket domain, type, and protocol are few. SOCK_STREAM goes with TCP, and SOCK_DGRAM goes with UDP. Also notice that the AF_UNIX address family doesn't use a named protocol, but a pseudoprotocol named PF_UNSPEC (for "unspecified").

The Perl object-oriented IO::Socket module (discussed in Chapter 5) can fill in the correct socket type and protocol automatically when provided with partial information.

Datagram Sockets

Datagram-type sockets provide for the transmission of connectionless, unreliable, unsequenced messages. The UDP is the chief datagram-style protocol used by the Internet protocol family.

As the diagram in Figure 3.2 shows, datagram services resemble the postal system. Like a letter or a telegram, each datagram in the system carries its destination address, its return address, and a certain amount of data. The Internet protocols will make the best effort to get the datagram delivered to its destination.

There is no long-term relationship between the sending socket and the recipient socket: A client can send a datagram off to one server, then immediately turn around and send a datagram to another server using the same socket. But the connectionless nature of UDP comes at a price. Like certain countries' postal systems, it is very possible for a datagram to get "lost in the mail." A client cannot know whether a server has received its message until it receives an acknowledgment in reply. Even then, it can't know for sure that a message was lost, because the server might have received the original message and the acknowledgment got lost!

Datagrams are neither synchronized nor flow controlled. If you send a set of datagrams out in a particular order, they might not arrive in that order. Because of the vagaries of the Internet, the first datagram may go by one route, and the second one may take a different path. If the second route is faster than the first one, the two datagrams may arrive in the opposite order from which they were sent. It is also possible for a datagram to get duplicated in transit, resulting in the same message being received twice.

Because of the connectionless nature of datagrams, there is no flow control between the sender and the recipient. If the sender transmits datagrams faster than the recipient can process them, the recipient has no way to signal the sender to slow down, and will eventually start to discard packets.

Although a datagram's delivery is not reliable, its contents are. Modern implementations of UDP provide each datagram with a checksum that ensures that its data portion is not corrupted in transit.

Stream Sockets

The other major paradigm is stream sockets, implemented in the Internet domain as the TCP protocol. Stream sockets provide sequenced, reliable bidirectional communications via byte-oriented streams. As depicted in Figure 3.3, stream sockets resemble a telephone conversation. Clients connect to servers using their address, the two exchange data for a period of time, and then one of the pair breaks off the connection.

Reading and writing to stream sockets is a lot like reading and writing to a file. There are no arbitrary size limits or record boundaries, although you can impose a record-oriented structure on the stream if you like. Because stream sockets are sequenced and reliable, you can write a series of bytes into a socket secure in the knowledge that they will emerge at the other end in the correct order, provided that they emerge at all ("reliable" does not mean immune to network errors).

TCP also implements flow control. Unlike UDP, where the danger of filling the data-receiving buffer is very real, TCP automatically signals the sending host to suspend transmission temporarily when the reading host is falling behind, and to resume sending data when the reading host is again ready. This flow control happens behind the scenes and is ordinarily invisible.

If you have ever used the Perl open(FH, " | command ") syntax to open up a pipe to an external program, you will find that working with stream sockets is not much different. The major visible difference is that, unlike pipes, stream sockets are bidirectional.

Although it looks and acts like a continuous byte stream, the TCP protocol is actually implemented on top of a datagram-style service, in this case the low-level IP protocol. IP packets are just as unreliable as UDP datagrams, so behind the scenes TCP is responsible for keeping track of packet sequence numbers, acknowledging received packets, and retransmitting lost packets.

Datagram versus Stream Sockets

With all its reliability problems, you might wonder why anyone uses UDP. The answer is that most client/server programs on the Internet use TCP stream sockets instead. In most cases, TCP is the right solution for you, too.

There are some circumstances, however, in which UDP might be a better choice. For example, time servers use UDP datagrams to transmit the time of day to clients who use the information for clock synchronization. If a datagram disappears in transit, it's neither necessary nor desirable to retransmit it because by the time it arrives it will no longer be relevant.

UDP is also preferred when the interaction between one host and the other is very short. The length of time to set up and take down a TCP connection is about eightfold greater than the exchange of a single byte of data via UDP (for details, see [Stevens 1996]). If relatively small amounts of data are being exchanged, the TCP setup time will dominate performance. Even after a TCP connection is established, each transmitted byte consumes more bandwidth than UDP because of the additional overhead for ensuring reliability.

Another common scenario occurs when a host must send the same data to many places; for example, it wants to transmit a video stream to multiple viewers. The overhead to set up and manage a large number of TCP connections can quickly exhaust operating system resources, because a different socket must be used for each connection. In contrast, sending a series of UDP datagrams is much more sparing of resources. The same socket can be reused to send datagrams to many hosts.

Whereas TCP is always a one-to-one connection, UDP also allows one-to-many and many-to-many transmissions. At one end of the spectrum, you can address a UDP datagram to the "broadcast address," broadcasting a message to all listening hosts on the local area network. At the other end of the spectrum, you can target a message to a predefined group of hosts using the "multicast" facility of modern IP implementations. These advanced features are covered in Chapters 20 and 21.

The Internet's DNS is a common example of a UDP-based service. It is responsible for translating hostnames into IP addresses, and vice versa, using a loose-knit network of DNS servers. If a client does not get a response from a DNS server, it just retransmits its request. The overhead of an occasional lost datagram outweighs the overhead of setting up a new TCP connection for each request. Other common examples of UDP services include Sun's Network File System (NFS) and the Trivial File Transfer Protocol (TFTP). The latter is used by diskless workstations during boot in order to load their operating system over the network. UDP was originally chosen for this purpose because its implementation is relatively small. Therefore, UDP fit more easily into the limited ROM space available to workstations at the time the protocol was designed.

The TCP/IP protocol suite is described well in [Stevens 1994, Wright and Stevens 1995, and Stevens 1996], as well as in RFC 1180, A TCP/IP Tutorial .

Socket Addressing

In order for one process to talk to another, each of them has to know the other's address. Each networking domain has a different concept of what an address is. For the UNIX domain, which can be used only between two processes on the same host machine, addresses are simply paths on the host's filesystem, such as /usr/tmp/log. For the Internet domain, each socket address has three parts: the IP address, the port, and the protocol.

IP Addresses

In the currently deployed version of TCP/IP, IPv4, the IP address is a 32-bit number used to identify a network interface on the host machine. A series of subnetworks and routing tables enables any machine on the Internet to send packets to any other machine based on its IP address.

For readability, the four bytes of a machine's IP address are usually spelled out as a series of four decimal digits separated by dots to create the "dotted quad address" that network administrators have come to know and love. For example, 143.48.7.1 is the IP address of one of the servers at my workplace. Expressed as a 32-bit number in the hexadecimal system, this address is 0x8f3071.

Many of Perl's networking calls require you to work with IP addresses in the form of packed binary strings. IP addresses can be converted manually to binary format and back again using pack() and unpack() with a template of "C4" (four unsigned characters). For example, here's how to convert 18.157.0.125 into its packed form and then reverse the process:

($a,$b,$c,$d)      = split /\./, '18.157.0.125';
 $packed_ip_address = pack 'C4',$a,$b,$c,$d;
 ($a,$b,$c,$d)      = unpack 'C4',$packed_ip_address;
 $dotted_ip_address = join '.', $a,$b,$c,$d;
 

You usually won't have to do this, however, because Perl provides convenient high-level functions to handle this conversion for you.

Most hosts have two addresses, the "loopback" address 127.0.0.1 (often known by its symbolic name "localhost") and its public Internet address. The loopback address is associated with a device that loops transmissions back onto itself, allowing a client on the host to make an outgoing connection to a server running on the same host. Although this sounds a bit pointless, it is a powerful technique for application development, because it means that you can develop and test software on the local machine without access to the network.

The public Internet address is associated with the host's network interface card, such as an Ethernet card. The address is either assigned to the host by the network administrator or, in systems with dynamic host addressing, by a Boot Protocol (BOOTP) or Dynamic Host Configuration Protocol (DHCP) server. If a host has multiple network interfaces installed, each one can have a distinct IP address. It's also possible for a single interface to be configured to use several addresses. Chapter 21 discusses IO::Interface, a third-party Perl module that allows a Perl script to examine and alter the IP addresses assigned to its interface cards.

Reserved IP Addresses, Subnets, and Netmasks

In order for a packet of information to travel from one location to another across the Internet, it must hop across a series of physical networks. For example, a packet leaving your desktop computer must travel across your LAN (local area network) to a modem or router, then across your Internet service provider's (ISP) regional network, then across a backbone to another ISP's regional network, and finally to its destination machine.

Network routers keep track of how the networks interconnect, and are responsible for determining the most efficient route to get a packet from point A to point B. However, if IP addresses were allocated ad hoc, this task would not be feasible because each router would have to maintain a map showing the locations of all IP addresses. Instead, IP addresses are allocated in contiguous chunks for use in organizational and regional networks.

For example, my employer, the Cold Spring Harbor Laboratory (CSHL), owns the block of IP addresses that range from 143.48.0.0 through 143.48.255.255 (this is a so-called class B address). When a backbone router sees a packet addressed to an IP address in this range, it needs only to determine how to get the packet into CSHL's network. It is then the responsibility of CSHL's routers to get the packet to its destination. In practice, CSHL and other large organizations split their allocated address ranges into several subnets and use routers to interconnect the parts.

A computer that is sending out an IP packet must determine whether the destination machine is directly reachable (e.g., over the Ethernet) or whether the packet must be directed to a router that interconnects the local network to more distant locations. The basic decision is whether the packet is part of the local network or part of a distant network.

To make this decision possible, IP addresses are arbitrarily split into a host part and a network part. For example, in CSHL's network, the split occurs after the second byte: the network part is 143.48. and the host part is the rest. So 143.48.0.0 is the first address in CSHL's network, and 143.48.255.255 is the last.

To describe where the network/host split occurs for routing purposes, networks use a netmask, which is a bitmask with 1s in the positions of the network part of the IP address. Like the IP address itself, the netmask is usually written in dotted-quad form. Continuing with our example, CSHL has a netmask of 255.255.0.0, which, when written in binary, is 11111111,11111111,00000000,00000000.

Historically, IP networks were divided into three classes on the basis of their netmasks (Table 3.5). Class A networks have a netmask of 255.0.0.0 and approximately 16 million hosts. Class B networks have a netmask of 255.255.0.0 and some 65,000 hosts, and class C networks use the netmask 255.255.255.0 and support 254 hosts (as we will see, the first and last host numbers in a network range are unavailable for use as a normal host address).

Table 3.5. Address Classes and Their Netmasks

Class	Netmask	Example Address	Network Park	Host Part
A	255.0.0.0	120.155.32.5	120.	155.32.5
B	255.255.0.0	128.157.32.5	128.157.	32.5
C	255.255.255.0	192.66.12.56	192.66.12.	56

As the Internet has become more crowded, however, networks have had to be split up in more flexible ways. It's common now to see netmasks that don't end at byte boundaries. For example, the netmask 255.255.255.128 (binary 11111111,11111111,11111111,10000000) splits the last byte in half, creating a set of 126-host networks. The modern Internet routes packets based on this more flexible scheme, called Classless Inter-Domain Routing (CIDR). CIDR uses a concise convention to describe networks in which the network address is followed by a slash and an integer containing the number of 1s in the mask. For example, CSHL's network is described by the CIDR address 143.48.0.0/16. CIDR is described in detail in RFCs 1517 through 1520, and in the FAQs listed in Appendix D.

Figuring out the network and broadcast addresses can be confusing when you work with netmasks that do not end at byte boundaries. The Net::Netmask module, available on CPAN, provides facilities for calculating these values in an intuitive way. You'll also find a short module that I wrote, Net::NetmaskLite, in Appendix A. You might want to peruse this code in order to learn the relationships among the network address, broadcast address, and netmask.

The first and last addresses in a subnet have special significance and cannot be used as ordinary host addresses. The first address, sometimes known as the all-zeroes address, is reserved for use in routing tables to denote the network as a whole. The last address in the range, known as the all-ones address, is reserved for use as the broadcast address. IP packets sent to this address will be received by all hosts on the subnet. For example, for the network 192.18.4.x (a class C address or 192.18.4.0/24 in CIDR format), the network address is 192.18.4.0 and the broadcast address is 192.18.4.255. We will discuss broadcasting in detail in Chapter 20.

In addition, several IP address ranges have been set aside for special purposes (Table 3.6). The class A network 10.x.x.x, the 16 class B networks 172.16.x.x through 172.31.x.x, and the 255 class C addresses 192.168.0.x through 192.168.255.x are reserved for use as internal networks. An organization may use any of these networks internally, but must not connect the network directly to the Internet. The 192.168.x.x networks are used frequently in testing, or placed behind firewall systems that translate all the internal network addresses into a single public IP address. The network addresses 224.x.x.x through 239.x.x.x are reserved for multicasting applications (Chapter 21), and everything above 240.x.x.x is reserved for future expansion.

Table 3.6. Reserved IP Addresses

Address	Description
127.0.0.x	Loopback interface
10.x.x.x	Private class A address
172.16.x.x172.32.x.x	Private class B addresses
192.168.0.x172.168.255.x	Private class C addresses

Finally, IP address 127.0.0.x is reserved for use as the loopback network. Anything sent to an address in this range is received by the local host.

IPv6

Although there are more than 4 billion possible IPv4 addresses, the presence of several large reserved ranges and the way the addresses are allocated into subnetworks reduces the effective number of addresses considerably. This, coupled with the recent explosion in network-connected devices, means that the Internet is rapidly running out of IP addresses. The crisis has been forestalled for now by various dynamic host-addressing and address-translation techniques that share a pool of IP addresses among a larger set of hosts. However, the new drive to put toaster ovens, television set-top boxes, and cell phones on the Internet is again threatening to exhaust the address space.

This is one of the major justifications for the new version of TCP/IP, known as IPv6, which expands the IP address space from 4 to 16 bytes. IPv6 is being deployed on the Internet backbones now, but this change will not immediately affect local area networks, which will continue to use addresses backwardly compatible with IPv4. Perl has not yet been updated to support IPv6, but will undoubtedly do so by the time that IPv6 is widely implemented.

More information about IPv6 can be found in [Stevens 1996] and [Hunt 1998 ].

Network Ports

Once a message reaches its destination IP address, there's still the matter of finding the correct program to deliver it to. It's common for a host to be running multiple network servers, and it would be impractical, not to say confusing, to deliver the same message to them all. That's where the port number comes in. The port number part of the socket address is an unsigned 16-bit number ranging from 1 to 65535. In addition to its IP address, each active socket on a host is identified by a unique port number; this allows messages to be delivered unambiguously to the correct program. When a program creates a socket, it may ask the operating system to associate a port with the socket. If the port is not being used, the operating system will grant this request, and will refuse other programs access to the port until the port is no longer in use. If the program doesn't specifically request a port, one will be assigned to it from the pool of unused port numbers.

There are actually two sets of port numbers, one for use by TCP sockets, and the other for use by UDP-based programs. It is perfectly all right for two programs to be using the same port number provided that one is using it for TCP and the other for UDP.

Not all port numbers are created equal. The ports in the range 0 through 1023 are reserved for the use of "well-known" services, which are assigned and maintained by ICANN, the Internet Corporation for Assigned Names and Numbers. For example, TCP port 80 is reserved for use for the HTTP used by Web servers, TCP port 25 is used for the SMTP used by e-mail transport agents, and UDP port 53 is used for the domain name service (DNS). Because these ports are well known, you can be pretty certain that a Web server running on a remote machine will be listening on port 80. On UNIX systems, only the root user (i.e., the superuser) is allowed to create a socket using a reserved port. This is partly to prevent unprivileged users on the system inadvertently running code that will interfere with the operations of the host's network services.

A list of reserved ports and their associated well-known services is given in Appendix C. Most services are either TCP- or UDP-based, but some can communicate with both protocols. In the interest of future compatibility, ICANN usually reserves both the UDP and TCP ports for each service. However, there are many exceptions to this rule. For example, TCP port 514 is used on UNIX systems for remote shell (login) services, while UDP port 514 is used for the system logging daemon.

In some versions of UNIX, the high-numbered ports in the range 49152 through 65535 are reserved by the operating system for use as "ephemeral" ports to be assigned automatically to outgoing TCP/IP connections when a port number hasn't been explicitly requested. The remaining ports, those in the range 1024 through 49151, are free for use in your own applications, provided that some other service has not already claimed them. It is a good idea to check the ports in use on your machine by using one of the network tools introduced later in this chapter (Network Analysis Tools) before claiming one.

The sockaddr_in Structure

A socket address is the combination of the host address and the port, packed together in a binary structure called a sockaddr_in. This corresponds to a C structure of the same name that is used internally to call the system networking routines. (By analogy, UNIX domain sockets use a packed structure called a sockaddr_un.) Functions provided by the standard Perl Socket module allow you to create and manipulate sockaddr_in structures easily:

$packed_address = inet_aton($dotted_quad) Given an IP address in dotted-quad form, this function packs it into binary form suitable for use by sockaddr_in(). The function will also operate on symbolic hostnames. If the hostname cannot be looked up, it returns undef. $dotted_quad = inet_ntoa($packed_address) This function takes a packed IP address and converts it into human-readable dotted-quad form. It does not attempt to translate IP addresses into hostnames. You can achieve this effect by using gethostbyaddr(), discussed later. $socket_addr = sockaddr_in($port,$address) ($port,$address) = sockaddr_in($socket_addr) When called in a scalar context, sockaddr_in() takes a port number and a binary IP address and packs them together into a socket address, suitable for use by socket(). When called in a list context, sockaddr_in() does the opposite, translating a socket address into the port and IP address. The IP address must still be passed through inet_ntoa() to obtain a human-readable string. $socket_addr = pack_sockaddr_in($port,$address) ($port,$address) = unpack_sockaddr_in($socket_addr) If you don't like the confusing behavior of sockaddr_in(), you can use these two functions to pack and unpack socket addresses in a context-insensitive manner.

We'll use several of these functions in the example that follows in the next section.

In some references, you'll see a socket's address referred to as its "name." Don't let this confuse you. A socket's address and its name are one and the same.

A Simple Network Client

To put this information into context, we'll now write a client for the daytime service. This service, which runs on many UNIX hosts, listens for incoming connections on TCP port 13. When it sees such a connection, it outputs a single line of text with the current time and date.

The script gets the dotted IP address of the daytime server from the command line. When we run it on 64.7.3.43, which is the IP address of the wuarchive.wustl.edu software archive, we get output like this:

% daytime_cli.pl 64.7.3.43
 Sat Jan 6 19:11:22 2001
 

Figure 3.4 shows the code for the daytime client.

Lines 13: Load modules We turn on strict type checking with use strict. This will avoid bugs introduced by mistyped variable names, the automatic interpretation of bare words as strings, and other common mistakes. We then load the Socket module, which imports several functions useful for socket programming.

Lines 46: Define constants We now define several constants. DEFAULT_ADDR is the IP address of a remote host to contact when an address isn't explicitly provided on the command line. We use 127.0.0.1, the loopback address.

PORT is the well-known port for the daytime service, port 13, while IPPROTO_TCP is the numeric protocol for the TCP protocol, needed to construct the socket.

It's inelegant to hard code these constants, and in the case of IPPROTO_TCP may impede portability. The next section shows how to look up these values at run time using symbolic names.

Lines 79: Construct the destination address The next part of the code constructs a destination address for the socket using the IP address of the daytime host and the port number of the daytime service. We begin by recovering the dotted-quad address from the command line or, if no address was specified, we default to the loopback address.

We now have an IP address in string form, but we need to convert it into packed binary form before passing it to the socket creation function. We do this by using the inet_aton() function, described earlier.

The last step in constructing the destination address is to create the sockaddr_in structure, which combines the IP address with the port number. We do this by calling the sockaddr_in() function with the port number and packed IP address.

Line 10: Create the socket The socket() function creates the communications endpoint. The function takes four arguments. The first argument, SOCK, is the filehandle name to use for the socket. Sockets look and act like filehandles, and like them are capitalized by convention. The remaining arguments are the address family, the socket type, and the protocol number. With the exception of the protocol, which we have hard-coded, all constants are taken from the socket module.

This call to socket() creates a stream-style Internet-domain socket that uses the TCP protocol.

Line 11: Connect to the remote host SOCK corresponds to the local communications endpoint. We now need to connect it to the remote socket. We do this using the built-in Perl function connect(), passing it the socket handle and the remote address that we built earlier. If connect() is successful, it will return a true value. Otherwise, we again die with an error message.

Line 12: Read data from remote host and print it We can now treat SOCK like a read/write filehandle, either reading data transmitted from the remote host with the read() or <> functions, or sending data to the host using print(). In this case, we use the angle bracket operator (<>) to read all lines transmitted by the remote host and immediately echo the data to standard output.

We discuss the socket() and connect () calls in more detail in Chapter 4.

Network Names and Services

The example in the last section used the remote host's dotted-quad IP address and numeric port and protocol numbers to construct the socket. Usually, however, you'll want to use symbolic names instead of numbers. Not only does this make it easier to write the program, but it makes it easier for end users to use it, allowing them to enter the host address as wuarchive.wustl.edu rather than 128.252.120.8.

The domain name system (DNS) is an Internet-wide database that translates hostnames into dotted IP addresses and back again. Various local database services translate from service names into numbers. This section introduces the functions that allow you to translate names into numbers, and vice versa.

Translating Hostnames into IP Addresses

Perl's gethostbyname() and gethostbyaddr() functions translate a symbolic hostname into a packed IP address, and vice versa. They are front ends to system library calls of the same name. Depending on your system's name-resolution setup, the call will consult one or more static text files such as /etc/hosts, local area network databases such as NIS, or the Internet-wide DNS.

($name,$aliases,$type,$len,$packed_addr) = gethostbyname($name); $packed_addr = gethostbyname($name); If the hostname doesn't exist, gethostbyname() returns undef. Otherwise, in a scalar context it returns the IP address of the host in packed binary form, while in a list context it returns a five-element list. The first element, $name, is the canonical hostnamethe official namefor the requested host. This is followed by a list of hostname aliases, the address type and address length, usually AF_INET and 4, and the address itself in packed form. The aliases field is a space-delimited list of alternative names for the host. If there are no alternative names, the list is empty. You can pass the packed address returned by gethostbyname() directly to the socket functions, or translate it back a human-readable dotted-quad form using inet_ntoa(). If you pass gethostbyname() a dotted-quad IP address, it will detect that fact and return the packed version of the address, just as inet_aton() does. $name = gethostbyaddr($packed_addr,$family); ($name,$aliases,$type,$len,$packed_addr) = gethostbyaddr($packed_addr, $family); The gethostbyaddr() call performs the reverse lookup, taking a packed IP address and returning the hostname corresponding to it. gethostbyaddr() takes two arguments, the packed address and the address family (usually AF_INET). In a scalar context, it returns the hostname of the indicated address. In a list context, it returns a five-element list consisting of the canonical hostname, a list of aliases, the address type, address length, and packed address. This is the same list returned by gethostbyname(literal). If the call is unsuccessful in looking up the address, it returns undef or an empty list.

The inet_aton() function can also translate hostnames into packed IP addresses. In general, you can replace scalar context calls to gethostbyname() with calls to inet_aton():

$packed_address = inet_aton($host);
 $packed_address = gethostbyname($host);
 

Which of these two functions you use is a matter of taste, but be aware that gethostbyname() is built into Perl, but inet_aton() is available only after you load the socket module. I prefer inet_aton() because, unlike gethostbyname(), it isn't sensitive to list context.

Hostname Translation Examples

Using these functions, we can write a simple program to translate a list of hostnames into dotted-quad IP addresses (Figure 3.5). Given a file of hostnames stored in hostnames.txt, the script's output looks like this:

% ip_trans.pl < hostnames.txt
 pesto.cshl.org => 143.48.7.220
 w3.org => 18.23.0.20
 foo.bar.com => ?
 ntp.css.gov => 140.162.3
 

Figure 3.6 shows a short program for performing the reverse operation, translating a list of dotted-quad IP addresses into hostnames. For each line of input, the program checks that it looks like a valid IP address (line 6) and, if so, packs the address using inet_aton(). The packed address is then passed to gethostbyaddr(), specifying AF_INET for the address family (line 7). If successful, the translated hostname is printed out.

Getting Information about Protocols and Services

In the same way that gethostbyname() and gethostbyaddr() interconvert hostnames and IP addresses, Perl provides functions to translate the symbolic names of protocols and services into protocol and port numbers.

$number = getprotobyname($protocol) ($name,$aliases,$number) = getprotobyname($protocol) getprotobyname() takes a symbolic protocol name, such as "udp", and converts it into its corresponding numeric value. In a scalar context, just the protocol number is returned. In a list context, the function returns the protocol name, a list of aliases, and the number. Multiple aliases are separated by spaces. If the named protocol is unknown, the function returns undef or an empty list. $name = getprotobynumber($protocol_number) ($name,$aliases,$number) = getprotobyname($protocol_number) The rarely used getprotobynumber() function reverses the previous operation, translating a protocol number into a protocol name. In a scalar context, it returns the protocol number's name as a string. In a list context, it returns the same list as getprotobyname(). If passed an invalid protocol number, this function returns undef or an empty list. $port = getservbyname($service,$protocol) ($name,$aliases,$port,$protocol) = getservbyname($service,$protocol) The getservbyname() function converts a symbolic service name, such as "echo," into a port number suitable for passing to sockaddr_in(). The function takes two arguments corresponding to the service name and the desired protocol. The reason for the additional protocol argument is that some services, echo included, come in both UDP and TCP versions, and there's no guarantee that the two versions of the service use the same port number, although this is almost always the case. In a scalar context, getservbyname() returns the port number of the service, or undef if unknown. In a list context, the function returns a four-element list consisting of the canonical name for the service, a space-delimited list of aliases, if any, the port number, and the protocol number. If the service is unknown, the function returns an empty list. $name = getservbyport($port,$protocol) ($name,$aliases,$port,$protocol) = getservbyport($port,$protocol) The getservbyport() function reverses the previous operation by translating a port number into the corresponding service name. Its behavior in scalar and list contexts is exactly the same as getservbyname().

Daytime Client, Revisited

We now have the tools to eliminate the hard-coded port and protocol numbers from the daytime client that we developed earlier. In addition, we can make the program more user friendly by accepting the name of the daytime host as either a DNS name or a dotted-IP address. Figure 3.7 gives the code for the revised client. The differences from the earlier version are as follows:

Line 5: Look up IP address of the daytime host We call gethostbyname() to translate the hostname given on the command line into a packed IP address. If no command-line argument is given, we default to using the loopback address. If the user provided a dotted-IP address instead of a hostname, then gethostbyname() simply packs it in the manner of inet_aton(). If gethostbyname() fails because the provided hostname is invalid, it returns undef and we exit with an error message.

Line 6: Look up the TCP protocol number We call getprotobyname() to retrieve the value of the TCP protocol number, rather than hard coding it as before.

Line 7: Look up the daytime service port number Instead of hard coding the port number for the daytime service, we call getservbyname() to look it up in the system service database. If for some reason this doesn't work, we exit with an error message.

Lines 811: Connect and read data The remainder of the program is as before, with the exception that we use the TCP protocol number retrieved from getprotobyname() in the call to socket().

You can test the script on the host http://www.modperl.com/

Other Sources of Network Information

In addition to gethostbyname() and its brethren, specialized add-on modules offer direct access to a number of other common databases of network information. I won't cover them in detail here, but you should know that they exist if you need them. All the modules can be obtained from CPAN.

Net::DNS

This module offers you much greater control over how hostnames are resolved using the domain name system. In addition to the functionality offered by gethostbyname() and gethostbyaddr(), Net::DNS allows you to fetch and iterate over all hosts in a domain; get the e-mail address of the network administrator responsible for the domain; and look up the machine responsible for accepting e-mail for a domain (the "mail exchanger," or MX).

Net::NIS

Many traditional UNIX networks use Sun's Network Information System (NIS) to distribute such things as hostnames, IP addresses, usernames, passwords, automount tables, and cryptographic keys. This allows a user to have the same login username, password, and home directories on all the machines on the network. Perl's built-in network information access functions, such as gethostbyname() and getpwnam(), provide transparent access to much, but not all, of NIS's functionality. The Net::NIS module provides access to more esoteric information, such as automount tables.

Net::LDAP

NIS is slowly being displaced by the Lightweight Directory Access Protocol (LDAP), which provides greater flexibility and scalability. In addition to the types of information stored by NIS, LDAP is often used to store users' e-mail addresses, telephone numbers, and other "white pages"type information. Net::LDAP gives you access to these databases.

Win32::NetResource

On Windows networks, the NT domain controller provides directory information on hosts, printers, shared file systems, and other network information. Win32::NetResources provides functions for reading and writing this information.

Network Analysis Tools

This section lists some of the basic tools for analyzing networks and diagnosing network-related problems. Some of these tools come preinstalled on many systems; others must be downloaded and installed. For more information on using these tools, please see the comprehensive discussion of network configuration and troubleshooting in [Hunt 1998].

ping

The ping utility, available as a preinstalled utility on all UNIX and Windows machines, is the single most useful network utility. It sends a series of ICMP "ping" messages to the remote IP address of your choice, and reports the number of responses the remote machine returns.

ping can be used to test if a remote machine is up and reachable across the network. It can also be used to test network conditions by looking at the length of time between the outgoing ping and the incoming response, and the number of pings that have no response (due to either loss of the outgoing message or the incoming response).

For example, this is how ping can be used to test connectivity to the machine at IP address 216.32.74.55 (which happens to be www.yahoo.com):

% ping 216.32.74.55
 PING 216.32.74.55: 56 data bytes
 64 bytes from 216.32.74.55: icmp_seq=0 ttl=245 time=41.1 ms
 64 bytes from 216.32.74.55: icmp_seq=1 ttl=245 time=16.4 ms
 64 bytes from 216.32.74.55: icmp_seq=2 ttl=245 time=16.3 ms
 ^C
 
 --- 216.32.74.55 ping statistics ---
 4 packets transmitted, 3 packets received, 25% packet loss
 round-trip min/avg/max = 16.3/24.6/41.1 ms
 

This session shows good connectivity. The average response time is a snappy 24 ms, and no packets were lost. You can also give ping a DNS name, in which case it will attempt to resolve the name before pinging the host.

One thing to watch for is that some firewall systems are configured to block ping. In this case, the destination machine may be unpingable, although you can reach it via telnet or other means.

There are many variants of ping, each with a different overlapping set of features.

nslookup

The nslookup utility, available on most UNIX systems, can be used to test and verify the DNS. It can be used interactively or as a one-shot command-line tool. To use it from the command line, call it with the DNS name of the host or domain you wish to look up. It will perform the DNS search, and return IP addresses and other DNS information corresponding to the name. For example:

% nslookup www.yahoo.com
 Server:  presto.lsjs.org
 Address:  64.7.3.44
 Non-authoritative answer:
 Name:    www.yahoo.akadns.net
 Addresses:  204.71.200.67, 204.71.200.68, 204.71.202.160, 204.71.200.74, 204.71.200.75
 Aliases:  www.yahoo.com
 

This tells us that the host www.yahoo.com has a canonical name of www.yahoo.akadns.net, and has five IP addresses assigned to it. This is typical of a heavily loaded Web server, where multiple physical machines balance incoming requests by servicing them in a round-robin fashion.

traceroute

While ping tells you only whether a packet can get from A to B, the traceroute program displays the exact path a network packet takes to get there. Call it with the IP address of the destination. Each line of the response gives the address of a router along the way. For example:

% traceroute www.yahoo.com
 traceroute to www.yahoo.akadns.net (216.32.74.52), 30 hops max, 40 byte packets
  1  gw.lsjs.org (192.168.3.1)  2.52 ms  8.78 ms  4.85 ms
  2  64.7.3.46 (64.7.3.46)  9.7 ms  9.656 ms  3.415 ms
  3  mgp-gw.nyc.megapath.net (64.7.2.1)  19.118 ms  23.619 ms  16.601 ms
  4  216.35.48.242 (216.35.48.242)  10.532 ms  10.515 ms  11.368 ms
  5  dcr03-g2-0.jrcy01.exodus.net (216.32.222.121)  9.068 ms  9.369 ms  9.08 ms
  6  bbr02-g4-0.jrcy01.exodus.net (209.67.45.126)  9.522 ms  11.091 ms  10.212 ms
  7  bbr01-p5-0.stng01.exodus.net (209.185.9.98)  15.516 ms  15.118 ms  15.227 ms
  8  dcr03-g9-0.stng01.exodus.net (216.33.96.145)  15.497 ms  15.448 ms  15.462 ms
  9  csr22-ve242.stng01.exodus.net (216.33.98.19)  16.044 ms  15.724 ms  16.454 ms
 10  216.35.210.126 (216.35.210.126)  15.954 ms  15.537 ms  15.644 ms
 11  www3.dcx.yahoo.com (216.32.74.52)  15.644 ms  15.582 ms  15.577 ms
 

traceroute can be invaluable for locating a network outage when a host can no longer be pinged. The listing will stop without reaching the desired destination, and the last item on the list indicates the point beyond which the breakage is occurring.

As with ping, some firewalls can interfere with traceroute. Traceroute is preinstalled on most UNIX systems.

netstat

The netstat utility, preinstalled on UNIX and Windows NT/2000 systems, prints a snapshot of all active network services and connections. For example, running netstat on an active Web and FTP server produces the following display (abbreviated for space):

% netstat -t
 Active Internet connections (w/o servers)
 Proto  Recv-Q  Send-Q  Local Address       Foreign Address          State
 tcp         0       0  brie.cshl.org:www   writer.loci.wisc.e:1402  ESTABLISHED
 tcp         0       0  brie.cshl.org:www   157-238-71-168.il.:1215  FIN_WAIT2
 tcp         0       0  brie.cshl.org:www   157-238-71-168.il.:1214  FIN_WAIT2
 tcp         0       0  brie.cshl.org:www   157-238-71-168.il.:1213  TIME_WAIT
 tcp         0       0  brie.cshl.org:6010  brie.cshl.org:2225       ESTABLISHED
 tcp         0       0  brie.cshl.org:2225  brie.cshl.org:6010       ESTABLISHED
 tcp         0    2660  brie.cshl.org:ssh   presto.lsjs.org:64080    ESTABLISHED
 tcp         0       0  brie.cshl.org:www   206.169.243.7:1724       TIME_WAIT
 tcp         0      20  brie.cshl.org:ftp   usr25-wok.cableine:2173  ESTABLISHED
 tcp         0     891  brie.cshl.org:www   usr25-wok.cableine:2117  FIN_WAIT1
 tcp         0      80  brie.cshl.org:ftp   soa.sanger.ac.uk:49596   CLOSE
 

The -t argument restricts the display to TCP connections. The Recv-Q and Send-Q columns show the number of bytes in the sockets' read and write buffers, respectively. The Local and Foreign Address columns show the name and port numbers of the local and remote peers, respectively, and the State column shows the current state of the connection.

netstat can also be used to show services that are waiting for incoming connections, as well as UDP and UNIX-domain sockets. The netstat syntax on Windows systems is slightly different. To get a list of TCP connections similar to the one shown above, use the command netstat -p tcp.

tcpdump

The tcpdump utility, available preinstalled on many versions of UNIX, is a packet sniffer. It can be used to dump the contents of every packet passing by your network card, including those not directed to your machine. It features a powerful filter language that can be used to detect and display just those packets you are interested in, such as those using a particular protocol or directed toward a specific port.

MacTCP Watcher

MacTCP Watcher for the Macintosh combines the functionality of ping, dnslookup, and netstat into one user-friendly application. It can be found by searching the large shareware collection located at http://www.shareware.com/

scanner.exe

For Windows 98/NT/2000 developers, the small scanner.exe utility, also available from http://www.shareware.com/ , combines the functionality of ping and dnslookup with the ability to scan a remote host for open ports. It can be used to determine the services a remote host provides.

net-toolbox.exe

This is a comprehensive set of Windows network utilities that include ping, dnslookup, tcpdump, and netstat functionality. It can be found by anonymous FTP to gatekeeper.dec.com in the directory /pub/micro/pc/winsite/win95/netutil/.

Chapter 4. The TCP Protocol

In this chapter we look at TCP, a reliable, connection-oriented byte-stream protocol. These features make working with TCP sockets similar to working with familiar filehandles and pipes. After opening a TCP socket, you can send data through it using print() or syswrite(), and read from it using <>, read(), or sysread().

A TCP Echo Client

We'll start by developing a small TCP client that is more sophisticated than the examples we looked at in the previous chapter. By and large, clients are responsible for actively initiating their connection to a remote service. We have already seen the outlines of this process in Chapter 3. To review, a TCP client takes the following steps:

Our example application is a simple client for the TCP "echo" service. This service, run by default on many UNIX hosts, is simple. It waits for an incoming connection, accepts it, and then echoes every byte that it receives, rather like an annoying child. This continues until the client closes the connection.

If you want to try the example script and have no echo server of your own handy, there is an echo server running on wuarchive.wustl.edu, a large public FTP site.

Figure 4.1 shows tcp_echo_cli1.pl. We'll step through the code first, and then explain what we did and why we did it.

Lines 16: Load modules We turn on strict syntax checking, and load the Socket and IO::Handle modules. We use Socket for its socket-related constants, and IO::Handle for its autoflush() method.

Line 7: Declare globals We create two global variables for keeping track of the number of bytes we send and receive.

Lines 89: Process command-line arguments We read the destination host and port number from the command line. If the host isn't provided, we default to "localhost". If the port number isn't provided, we use getservbyname() to look up the port number for the echo service.

Lines 1011: Look up protocol and create packed IP address We use getprotobyname() to find the TCP protocol number for use with socket(). We then use inet_aton() to turn the hostname into a packed IP address for use with sockaddr_in().

Line 12: Create the socket We call socket() to create a socket filehandle named SOCK in the same way that we did in Figure 3.4 (in Chapter 3). We pass arguments specifying the AF_INET Internet address family, a stream-based socket type of SOCK_STREAM, and the TCP protocol number looked up earlier.

Lines 1314: Create destination address and connect socket to it We use sockaddr_in() to create a packed address containing the destination's IP address and port number. This address is now used as the destination for a connect() call. If successful, connect() returns true. Otherwise, we die with an error message.

Line 15: Turn on autoflush mode for the socket We want data written to the socket to be flushed immediately, rather than hang around in a local buffer. We call the socket's autoflush() method to turn on autoflush mode. This method is available courtesy of IO::Handle.

Lines 1622: Enter main loop We now enter a small loop. Each time through the loop we read a line of text from standard input, and print it verbatim to SOCK, sending it to the remote host. We then use the <> operator to read a line of response from the server, and print it to standard output.

Each time through the loop we tally the number of bytes sent and received. This continues until we reach EOF on standard input.

Lines 2324: Close the socket and print statistics After the loop is done, we close the socket and print to standard error our summary statistics on the number of bytes sent and received.

A session with tcp_echo_cli1.pl looks like this:

% tcp_echo_cli1.pl www.modperl.com
 How now brown cow?
 How now brown cow?
 There's an echo in here.
 There's an echo in here.
 Yo-de-lay-ee-oo!
 Yo-de-lay-ee-oo!
 ^D
 bytes_sent = 61, bytes_received = 61
 

The ^D on the second-to-last line of the transcript shows where I got tired of this game and pressed the end-of-input key. (On Windows systems, this would be ^Z.)

$boolean = socket (SOCKET,$domain,$type,$protocol)

Given a filehandle name, a domain, a type, and a protocol, socket() creates a new socket and associates it with the named filehandle. On success, the function returns a true value. On error, socket() returns undef and leaves the error message in $!.

The domain, type, and protocol are all small integers. Appropriate values for the first two are constants defined in the Socket module, but the protocol value must be determined at run time by calling getprotobyname(). For creating TCP sockets, the idiom is typically

socket(SOCK,AF_INET,SOCK_STREAM, scalar getprotobyname("tcp"))

Here we force getprotobyname() into a scalar context in order to return a single function result containing the protocol number.

$boolean = connect (SOCK,$dest_addr)

The connect() function attempts to connect a connection-oriented socket to the indicated destination address. The socket must already have been created with socket(), and the packed destination address created by sockaddr_in() or equivalent. The system will automatically choose an ephemeral port to use for the socket's local address.

On success, connect() returns a true value. Otherwise, connect() returns false and $! is set to the system error code explaining the problem. It is illegal for a connection-oriented socket to call connect() more than once; if another call is attempted, it results in an EISCONN ("Transport endpoint is already connected") error.

$boolean = close (SOCK)

The close() call works with sockets just like it does with ordinary filehandles. The socket is closed for business. Once closed, the socket can no longer be read from or written to. On success, close() returns a true value. Otherwise, it returns undef and leaves an error message in $!.

The effect of close() on the other end of the connection is similar to the effect of closing a pipe. After the socket is closed, any further reads on the socket at the other end return an end of file (EOF). Any further writes result in a PIPE exception.

$boolean = shutdown (SOCK, $how)

shutdown() is a more precise version of close() that allows you to control which part of the bidirectional connection to shut down. The first argument is a connected socket. The second argument, $how, is a small integer that indicates which direction to shut down. As summarized in Table 4.1, a $how of 0 closes the socket for further reads, a value of 1 closes the socket for writes, and a value of 2 closes the socket for both reading and writing (like close()). A nonzero return value indicates that the shutdown was successful.

Table 4.1. Shutdown() Values

Value of HOW	Description
0	Closes socket for reading
1	Closes socket for writing
2	Closes socket completely

In addition to its ability to half-close a socket, shutdown() has one other advantage over close(). If the process has called fork() at any point, there may be copies of the socket filehandle in the original process's children. A conventional close() on any of the copies does not actually close the socket until all other copies are closed as well (filehandles have the same behavior). In consequence, the client at the other end of the connection won't receive an EOF until both the parent and its child process(es) have closed their copies. In contrast, shutdown() closes all copies of the socket, sending the EOF immediately. We'll take advantage of this feature several times in the course of this book.

A TCP Echo Server

Now we'll look at a simple TCP server. In contrast to a TCP client, a server does not usually call connect(). Instead, a TCP server follows the following outline:

Our example server is named tcp_echo_serv.pl. This server is a slightly warped version of the standard echo server. It echoes everything sent to it, but rather than send it back verbatim, it reverses each line right to left (but preserves the newline at the end). So if one sends it "Hello world!," the line echoed back will be "!dlrow olleH." (There's no reason to do this except that it adds some spice to an otherwise boring example.)

This server can be used by the client of Figure 4.1, or with the standard Telnet program. Figure 4.3 lists the server code.

Lines 19: Load modules, initialize constants and variables We start out as in the client by bringing in the Socket and IO::Handle modules. We also define a private echo port of 2007 that won't conflict with any existing echo server. We set up the $port and $protocol variables as before (lines 89) and initialize the counters.

Lines 1013: Install INT interrupt handler There has to be a way to interrupt the server, so we install a signal handler for the INT (interrupt) signal, which is sent from the terminal when the user presses ^C. This handler simply prints out the accumulated byte counters' statistics and exits.

Line 14: Create the socket Using arguments identical to those used by the TCP client in Figure 4.1, we call socket() to create a stream TCP socket.

Line 15: Set the socket's SO_REUSEADDR option The next step is to set the socket's SO_REUSEADDR option to true by calling setsockopt(). This option is commonly used to allow us to kill and restart the server immediately. Otherwise, there are conditions under which the system will not allow us to rebind the local address until old connections have timed out.

Lines 1617: Bind the socket to a local address We now call bind() to assign a local address to a socket. We create a local address using sockaddr_in(), passing it our private echo port for the port, and INADDR_ANY as the IP address. INADDR_ANY acts as a wildcard. It allows the operating system to accept connections on any of the host's IP addresses (including the loopback address and any network interface card addresses it might have).

Line 18: Call listen() to make socket ready to accept incoming connections We call listen() to alert the operating system that the socket will be used for incoming connections.

The listen() function takes two arguments. The first is the socket, and the second is an integer indicating the number of incoming connections that can be queued to wait for processing. It's common for multiple clients to try to connect at roughly the same time; this argument determines how large the backlog of pending connections can get. In this case, we use a constant defined in the Socket module, SOMAXCONN, to take the maximum number of queued connections that the system allows.

Lines 1934: Enter main loop The bulk of the code is the server's main loop, in which it waits for, and services, incoming connections.

Line 21: accept() an incoming connection Each time through the loop we call accept(), using the name of the listening socket as the second argument, and a name for the new socket (SESSION) as the first. (Yes, the order of arguments is a little odd.) If the call to accept() is successful, it returns the packed address of the remote socket as its function result, and returzns the connected socket in SESSION.

Lines 2223: Unpack client's address We call sockaddr_in() in a list context to unpack the client address returned by accept() into its port and IP address components, and print the address to standard error. In a real application, we might write this information to a time-stamped log file.

Lines 2433: Handle the connection This section handles communications with the client using the connected socket. We first put the SESSION socket into autoflush mode to prevent buffering problems. We now read one line at a time from the socket using the <> operator, reverse the text of the line, and send it back to the client using print(>).

This continues until <SESSION> returns undef, which indicates that the peer has closed its end of the connection. We close the SESSION socket, print a status message, and go back to accept() to wait for the next incoming connection.

Line 35: Clean up After the main loop is done we close the listening socket. This part of the code is never reached because the server is designed to be terminated by the interrupt key.

When we run the example server from the command line, it prints out the "waiting for incoming connections" message and then pauses until it receives an incoming connection. In the session shown here, there are two connections, one from a local client at the 127.0.0.1 loopback address, and the other from a client at address 192.168.3.2. After interrupting the server, we see the statistics printed out by the INT handler.

% tcp_echo_serv1.p1
 waiting for incoming connections on port 2007...
 Connection from [127.0.0.1,2865]
 Connection from [127.0.0.1,2865] finished
 Connection from [192.168.3.2,2901]
 Connection from [192.168.3.2,2901] finished
 bytes_sent = 26, bytes_received = 26
 

The INT handler in this server violates the recommendation from Chapter 2 that signal handlers not do any I/O. In addition, by calling exit() from within the handler, it risks raising a fatal exception on Windows machines as they shut down. We will see a more graceful way of shutting down a server in Chapter 10.

Socket Functions Related to Incoming Connections

Three new functions are related to the need of servers to handle incoming connections.

$boolean = bind (SOCK,$my_addr) Bind a local address to a socket, returning a true value if successful, false otherwise. The socket must already have been created with socket(), and the packed address created by sockaddr_in() or equivalent. The port part of the address can be any unused port on the system. The IP address part may be the address of one of the host's network interfaces, the loopback address, or the wildcard INADDR_ANY. On UNIX systems, it requires superuser (root) privileges to bind to the reserved ports below 1024. Such an attempt will return undef and set $! to an error of EACCES ("Permission denied"). The bind() function is usually called in servers in order to associate a newly created socket with a well-known port; however, a client can call it as well if it wishes to specify its local port and/or network interface. $boolean = listen (SOCK,$max_queue) The listen() function tells the operating system that a socket will be used to accept incoming connections. The call's two arguments are a socket filehandle, which must already have been created with socket(), and an integer value indicating the number of incoming connections that can be queued. The maximum size of the queue is system-specific. If you specify a higher value than the system allows, listen() silently truncates it to its maximum value. The Socket module exports the constant SOMAXCONN to determine this maximum value. If successful, listen() returns a true value and marks SOCK as listening. Otherwise it returns undef and sets $! to the appropriate error. $remote_addr = accept (CONNECTED_SOCKET,LISTEN_SOCKET) Once a socket has been marked listening, call accept() to accept incoming connections. The accept() function takes two arguments, CONNECTED_SOCKET, the name of a filehandle to receive the newly connected socket, and LISTEN_SOCKET, the name of the listening socket. If successful, the packed address of the remote host will be returned as the function result, and CONNECTED_SOCKET will be associated with the incoming connection. Following accept(), you will use CONNECTED_SOCKET to communicate with the peer. You do not need to create CONNECTED_SOCKET beforehand. In case you're confused by this, think of accept() as a special form of open() in which LISTEN_SOCKET replaces the file name. If no incoming connection is waiting to be accepted, accept() will block until there is one. If multiple clients connect faster than your script calls accept(), they will be queued to the limit specified in the listen() call. The accept() function returns undef if any of a number of error conditions occur, and sets $! to an appropriate error message. $my_addr = getsockname (SOCK) $remote_addr = getpeername (SOCK) Should you wish to recover the local or remote address associated with a socket, you can do so with getsockname() or getpeername(). The getsockname() function returns the packed binary address of the socket at the local side, or undef if the socket is unbound. The getpeername() function behaves in the same way, but returns the address of the socket at the remote side, or undef if the socket is unconnected. In either case, the returned address must be unpacked with sockaddr_in(), as illustrated in this short example: if ($remote_addr = getpeername(SOCK)) { my ($port,$ip) = sockaddr_in($remote_addr); my $host = gethostbyaddr($ip,AF_INET); print "Socket is connected to $host at port $port\n"; }

Limitations of tcp_echo_serv1.pl

Although tcp_echo_serv1.pl works as written, it has a number of drawbacks that are addressed in later chapters. The drawbacks include the following:

Adjusting Socket Options

Sockets have options that control various aspects of their operation. Among the things you can adjust are the sizes of the buffers used to send and receive data, the values of send and receive timeouts, and whether the socket can be used to receive broadcast transmissions.

The default options are fine for most purposes; however, occasionally you may want to adjust some options to optimize your application, or to enable optional features of the TCP/IP protocol. The most commonly used option is SO_REUSEADDR, which is frequently activated in server applications.

Socket options can be examined or changed with the Perl built-in functions getsockopt() and setsockopt().

$value = getsockopt (SOCK,$level,$option_name); $boolean = setsockopt (SOCK,$level,$option_name,$option_value); The getsockopt() and setsockopt() functions allow you to examine and change a socket's options. The first argument is the filehandle for a previously created socket. The second argument, $level, indicates the level of the networking stack you wish to operate on. You will usually use the constant SO_SOCKET, indicating that you are operating on the socket itself. However, a getsockopt() and setsockopt() are occasionally used to adjust options in the TCP or UDP protocols, in which case you use the protocol number returned by getprotobyname(). The third argument, $option_name, is an integer value selected from a large list of possible constants. The last argument, $option_value, is the value to set the option to, or it returns undef if inapplicable. On success, getsockopt() returns the value of the requested option, or undef if the call failed. On success, setsockopt() returns a true value if the option was successfully set; otherwise, it returns undef.

The value of an option is often a Boolean flag indicating whether the option should be turned on and off. In this case, no special code is needed to set or examine the value. For example, here is how to set the value of SO_BROADCAST to true (broadcasting is discussed in Chapter 20):

setsockopt(SOCK,SO_SOCKET,SO_BROADCAST,1);
 

And here is how to retrieve the current value of the flag:

my $reuse = getsockopt(SOCK,SO_SOCKET,SO_BROADCAST);
 

However, a few options act on integers or more esoteric data types, such as C timeval structures. In this case, you must pack the values into binary form before passing them to setsockopt() and unpack them after calling getsockopt(). To illustrate, here is the way to recover the maximum size of the buffer that a socket uses to store outgoing data. The SO_SNDBUF option acts on a packed integer (the "I" format):

$send_buffer_size = unpack("I",getsockopt($sock,SOL_SOCKET,SO_SNDBUF));
 

Common Socket Options

Table 4.2 lists the common socket options used in network programming. These constants are imported by default when you load the Socket module.

SO_REUSEADDR allows a TCP socket to rebind to a local address that is in use. It takes a Boolean argument indicating whether to activate address reuse. See the section The SO_REUSEADDR Socket Option later in this chapter.

SO_KEEPALIVE, when set to a true value, tells a connected socket to send messages to the peer intermittently. If the remote host fails to respond to a message, your process will receive a PIPE signal the next time it tries to write to the socket. The interval of the keepalive messages cannot be changed in a portable way and varies from OS to OS (it is 45 seconds on my Linux system).

SO_LINGER controls what happens when you try to close a TCP socket that still has unsent queued data. Normally, close() returns immediately, and the operating system continues to try to send the unsent data in the background. If you prefer, you can make the close() call block until all the data has been sent by setting SO_LINGER, allowing you to inspect the return value returned by close() for successful completion.

Unlike other socket options, SO_LINGER operates on a packed data type known as the linger structure. The linger structure consists of two integers: a flag indicating whether SO_LINGER should be active, and a timeout value indicating the maximum number of seconds that close() should linger before returning. The linger structure should be packed and unpacked using the II format:

$linger = pack("II",$flag,$timeout);
 

For example, to make a socket linger for up to 120 seconds you would call:

setsockopt(SOCK,SOL_SOCKET, SO_LINGER,pack("II",1,120))
                             or die "Can't set SO_LINGER: $!";
 

SO_BROADCAST is valid for UDP sockets only. If set to a true value, it allows send() to be used to send packets to the broadcast address for delivery to all hosts on the local subnet. We discuss broadcasting in Chapter 20.

The SO_OOBINLINE flag controls how out-of-band information is handled. This feature allows the peer to be alerted of the presence of high-priority data. We describe how this works in Chapter 17.

SO-SNDLOWAT and SO_RCVLOWAT set the low-water marks for the output and input buffers, respectively. The significance of these options is discussed in Chapter 13, Nonblocking I/O. These options are integers, and must be packed and unpacked using the I pack format.

SO_TYPE is a read-only option. It returns the type of the socket, such as SOCK_STREAM. You will need to unpack this value with the I format before using it. The IO::Socket sockopt() method discussed in Chapter 5 does the conversion automatically.

Table 4.2. Common Socket Options

Option	Description
SO_REUSEADDR	Enable reuse of the local address.
SO_KEEPALIVE	Enable the transmission of periodic "keepalive" messages.
SO_LINGER	Linger on close if data present.
SO_BROADCAST	Allow socket to send messages to the broadcast address.
SO_OOBINLINE
Keep out-of-band data inline.
SO_SNDLOWAT	Get or set the size of the output buffer "low water mark."
SO_RECVLOWAT	Get or set the size of the input buffer "low water mark."
SO_TYPE	Get the socket type (read only).
SO_ERROR	Get and clear the last error on the socket (read only).

Last, the read-only SO_ERROR option returns the error code, if any, for the last operation on the socket. It is used for certain asynchronous operations, such as nonblocking connects (Chapter 13). The error is cleared after it is read. As before, users of getsockopt() need to unpack the value with the I format before using it, but this is taken care of automatically by IO::Socket.

The SO_REUSEADDR Socket Option

Many developers will want to activate the SO_REUSEADDR flag in server applications. This flags allows a server to rebind to an address that is already in use, allowing the server to restart immediately after it has crashed or been killed. Without this option, the bind() call will fail until all old connections have timed out, a process that may take several minutes.

The idiom is to insert the following line of code after the call to socket() and before the call to bind():

setsockopt(SOCK,SOL_SOCKET,SO_REUSEADDR,1) o r die "setsockopt: $!";
 

The downside of setting SO_REUSEADDR is that it allows you to launch your server twice. Both processes will be able to bind to the same address without triggering an error, and they will then compete for incoming connections, leading to confusing results. Servers that we develop later (e.g., Chapters 10, 14, and 16) avoid this possibility by creating a file when the program starts up and deleting it on exit. The server refuses to launch if it sees that this file already exists.

The operating system does not allow a socket address bound by one user's process to be bound by another user's process, regardless of the setting of SO_REUSEADDR.

The fcntl() and ioctl() Functions

In addition to socket options, a number of attributes are adjusted using the fcntl() and ioctl() functions. We discuss fcntl() in Chapter 13, where we use it to turn on nonblocking I/O, and again in Chapter 17, where we use fcntl() to set the owner of a socket so that we receive a URG signal when the socket receives TCP urgent data.

The ioctl() function appears in Chapter 17 as well, where we use it to implement the sockatmark() function for handling urgent data, and again in Chapter 21, where we create a variety of functions for examining and modifying the IP addresses assigned to network interfaces.

$bytes = send (SOCK,$data,$flags[,$destination])

The send() function uses the socket indicated by the first argument to deliver the data indicated by $data, to the destination address indicated by $destination. If the data was successfully queued for transmission, send() returns the number of bytes sent; otherwise, it returns undef. The third argument, $flags, is the bitwise OR of zero or more of the two options listed in Table 4.3.

We discuss the MSG_OOB flag in detail in Chapter 17. MSG_DONTROUTE is used in routing and diagnostic programs and is not discussed in this book. In general, you should pass 0 as the value of $flags in order to accept the default behavior.

If the socket is a connected TCP socket, then $destination should not be specified and send() is roughly equivalent to syswrite(). With UDP sockets, the destination can be changed with every call to send().

$address = recv (SOCK,$buffer,$length,$flags)

The recv() function accepts up to $length bytes from the indicated socket and places them into the scalar variable $buffer. The variable is grown or shrunk to the length actually read. The $flags argument has the same significance as the corresponding argument in $send and should usually be set to 0.

If successful, recv() returns the packed socket address of the message sender. In case of error, the function returns undef and sets $! appropriately.

When called on a connected TCP socket, recv() acts much like sysread(), except that it returns the address of the peer. The real usefulness of recv() is for receiving datagrams for UDP transmissions.

$boolean = socketpair (SOCK_A,SOCK_B,$domain,$type,$protocol)

The socketpair() function creates two unnamed sockets connected end to end. $domain, $type, and $protocol have the same significance as in the socket() function. If successful, socketpair() returns true and opens sockets on SOCK_A and SOCK_B.

Table 4.3. send() Flags

Option	Description
MSG_OOB	Transmit a byte of urgent data on a TCP socket.
MSG_DONTROUTE	Bypass routing tables.

The socketpair() function is similar to the pipe() function that we saw in Chapter 2, except that the connection is bidirectional. Typically a script creates a pair of sockets and then fork(), with the parent closing one socket and the child closing the other. The two sockets can then be used for bidirectional communication between parent and child.

While in principle socketpair() can used for the INET protocol, in practice most systems only support socketpair() for creating UNIX-domain sockets. Here is the idiom:

socketpair(SOCK1,SOCK2,AF_UNIX,SOCK_STREAM,PF_UNSPEC) or die $!;
 

We show examples of using UNIX-domain sockets in Chapter 22.

End-of-Line Constants Exported by the Socket Module

In addition to the constants used for constructing sockets and establishing outgoing connections, the Socket module optionally exports constants and variables that are useful for dealing with text-oriented network servers.

As we saw in Chapter 2, operating systems have different ideas of what constitutes the end of line in a text file, with various OSs using carriage return (CR), linefeed (LF), or carriage return/linefeed (CRLF). Adding to the confusion is the fact that Perl's "\r" and "\n" string escapes translate into different ASCII characters depending on the local OS's idea of the newline.

Although this is not a hard and fast rule, most text-oriented network services terminate their lines with the CRLF sequence, octal " \015\012 ". When performing line-oriented reads from such servers, you should set the input record separator, global $/, to " \015\012 " (not to " \r\n ", because this is nonportable). To make this simpler, the Socket module optionally exports several constants defining the common line endings (see Table 4. 4). In addition, to make it easier to interpolate these sequences into strings, Socket exports the variables $CRLF, $CR, and $LF.

Table 4.4. Constants Exported by Socket Module

Name	Description
CRLF	A constant that contains the CRLF sequence
CR	A constant that contains the CR character
LF	A constant that contains the LF character

These symbols are not exported by default, but must be brought in with use either individually or by importing the ":crlf" tag. In the latter case, you probably also want to import the ":DEFAULT" tag in order to get the default socket-related constants as well:

use Socket qw(:DEFAULT :crlf);
 

Exceptional Conditions during TCP Communications

The TCP protocol is tremendously robust in the face of poor network conditions. It can survive slow connections, flaky routers, intermittent network outages, and a variety of misconfigurations, and still manage to deliver a consistent, error-free data stream.

TCP can't overcome all problems. This section briefly discusses the common exceptions, as well as some common programming errors.

Exceptions during connect()

Various errors are common during calls to connect().

The ENOTSOCK error can also be returned by other socket calls, including bind(), listen(), accept(), and the sockopt() calls.

Exceptions during Read and Write Operations

Once a connection is established, errors are still possible. You are almost sure to encounter the following errors during your work with networked programs.

There are several exceptions to this, however. If, instead of simply going down, one of the routers along the way starts issuing error messages, such as "host unreachable," then the connection is terminated and the effect is similar to scenario (1). Another common situation is that the remote server has its own timeout system. In this case, it times out and closes the connection as soon as network connectivity is restored.

Chapter 5. The IO::Socket API

The last chapter walked through Perl's built-in interface to Berkeley sockets, which closely mirrors the underlying C-library calls. Some of the built-in functions, however, are awkward because of the need to convert addresses and other data structures into the binary forms needed by the C library.

The advent of the object-oriented extensions to Perl5 made it possible to create a more intuitive interface based on the IO::Handle module. IO::Socket and related modules simplify code and make it easier to read by eliminating the noisy C-language-related calls and allowing you to focus on the core of the application.

This chapter introduces the IO::Socket API, and then walks through some basic TCP applications.

Using IO::Socket

Before we discuss IO::Socket in detail, let's get a feel for it by reimplementing some of the examples from Chapters 3 and 4.

A Daytime Client

The first example is a rewritten version of the daytime service client developed in Chapter 3 (Figure 3.7). As you recall, this client establishes an outgoing connection to a daytime server, reads a line, and exits.

This is a good opportunity to fix a minor bug in the original examples (left there in the interests of not unnecessarily complicating the code). Like many Internet servers, the daytime service terminates its lines with CRLF rather than a single LF as Perl does. Before reading from daytime, we set the end-of-line character to CRLF. Otherwise, the line we read will contain an extraneous CR at the end.

Figure 5.1 lists the code.

Lines 14: Initialize module We load IO::Socket and import the ":crlf" constants as well as the default constants. These constants are conveniently reexported from Socket. We recover the name of the remote host from the command line.

Line 5: Set the end-of-line separator To read lines from the daytime server, we set the $/ end-of-line global to CRLF. Note that this global option affects all filehandles, not just the socket.

Lines 67: Create socket We create a new IO::Socket object by calling IO::Socket::INET's new method, specifying the destination address in the form $host:service. We will see other ways to specify the destination address later.

Lines 89: Read the time of day and print it We read a single line from the server by calling getline(), and remove the CRLF from the end with chomp(). We print this line to STDOUT.

When we run this program, we get the expected result:

% time_of_day_tcp.pl wuarchive.wustl.edu
 Tue Aug 15 07:39:49 2000
 

TCP Echo Client

Now we'll look at an object-oriented version of the echo service client from Chapter 4. Figure 5.2 lists the code.

Lines 18: Initialize script We load IO::Socket, initialize constants and globals, and process the command-line arguments.

Line 9: Create socket We call the IO::Socket::INET->new() method using the $host:$port argument. If new() is successful, it returns a socket object connected to the remote host.

Lines 1016: Enter main loop We now enter the main loop. Each time through the loop we call getline() on the STDIN filehandle to retrieve a line of input from the user. We send this line of text to the remote host using print() on the socket, and read the server's response using the <> operator. We print the response to standard output, and update the statistics.

Lines 1718: Clean up The main loop will end when STDIN is closed by the user. We close the socket and print the accumulated statistics to STDERR.

Did you notice that line 10 uses the object-oriented getline() method with STDIN? This is a consequence of bringing in IO::Socket, which internally loads IO::Handle. As discussed in Chapter 2, a side effect of IO::Handle is to add I/O object methods to all filehandles used by your program, including the standard ones.

Unlike with the daytime client, we don't need to worry about what kind of line ends the echo service uses, because it echoes back to us exactly what we send it.

Note also that we did not need to set autoflush mode on the IO::Socket object, as we did in examples in Chapter 4. Since IO::Socket version 1.18, autoflush is turned on by default in all sockets created by the module. This is the version that comes with Perl 5.00503 and higher.

IO::Socket Methods

We'll now look at IO::Socket in greater depth.

The IO::Handle Class Hierarchy

Figure 5.3 diagrams the IO::Handle class hierarchy. The patriarch of the family tree, IO::Handle, provides object-oriented syntax for all of Perl's various input/output methods. Its immediate descendent, IO::Socket, defines additional methods that are suitable for Berkeley sockets. IO::Socket has two descendents. IO::Socket::INET defines behaviors that are specific for Internet domain sockets; IO::Socket::UNIX defines appropriate behaviors for AF_UNIX (a.k.a., AF_LOCAL) sockets.

One never creates an IO::Socket object directly, but creates either an IO::Socket::INET or an IO::Socket::UNIX object. We use the IO::Socket::INET subclass both in this chapter and in much of the rest of this book. Future versions of the I/O library may support other addressing domains.

Other important descendents of IO::Handle include IO::File, which we discussed in Chapter 2, and IO::Pipe, which provides an object-oriented interface to Perl's pipe() call. From IO::Socket::INET descends Net::Cmd, which is the parent of a whole family of third-party modules that provide interfaces to specific command-oriented network services, including FTP and Post Office Protocol. We discuss these modules beginning in Chapter 6.

Although not directly descended from IO::Handle, other modules in the IO::* namespace include IO::Dir for object-oriented methods for reading and manipulating directories, IO::Select for testing sets of filehandles for their readiness to perform I/O, and IO::Seekable for performing random access on a disk file. We introduce IO::Select in Chapter 12, where we use it to implement network servers using I/O multiplexing.

Creating IO::Socket::INET Objects

As with other members of the IO::Handle family, you create new IO::Socket::INET objects by invoking its new() method, as in:

$sock = IO::Socket::INET->new('wuarchive.wustl.edu:daytime');
 

This object is then used for all I/O related to the socket. Because IO::Socket::INET descends from IO::Handle, its objects inherit all the methods for reading, writing, and managing error conditions introduced in Chapter 2. To these inherited methods, IO::Socket::INET adds socket-oriented methods such as accept(), connect(), bind(), and sockopt().

As with IO::File, once you have created an IO::Socket option you have the option of either using the object with a method call, as in:

$sock->print('Here comes the data.');
 

or using the object as a regular filehandle:

print $sock 'Ready or not, here it comes.';
 

Which syntax you use is largely a matter of preference. For performance reasons discussed at the end of this chapter, I prefer the function-oriented style whenever there is no substantive difference between the two.

The IO::Socket::INET->new() constructor is extremely powerful, and is in fact the most compelling reason for using the object-oriented socket interface.

$socket = IO::Socket::INET->new (@args); The new() class method attempts to create an IO::Socket::INET object. It returns the new object, or if an error was encountered, undef. In the latter case, $! contains the system error, and $@ contains a more verbose description of the error generated by the module itself.

IO::Socket::INET->new() accepts two styles of argument. In the simple "shortcut" style, new() accepts a single argument consisting of the name of the host to connect to, a colon, and the port number or service name. IO::Socket::INET creates a TCP socket, looks up the host and service name, constructs the correct sockaddr_in structure, and automatically attempts to connect() to the remote host. The shortcut style is very flexible. Any of these arguments is valid:

wuarchive.wustl.edu:echo
 wuarchive.wustl.edu:7
 128.252.120.8:echo
 128.252.120.8:7
 

In addition to specifying the service by name or port number, you can combine the two so that IO::Socket::INET will attempt to look up the service name first, and if that isn't successful, fall back to using the hard-coded port number. The format is hostname:service(port). For instance, to connect to the wuarchive echo service, even on machines that for some reason don't have the echo service listed in the network information database, we can call:

my $echo = IO::Socket::INET->new('wuarchive.wustl.edu:echo(7)')
             or die "Can't connect: $!\n";
 

The new() method can also be used to construct sockets suitable for incoming connections, UDP communications, broadcasting, and so forth. For these more general uses, new() accepts a named argument style that looks like this:

my $echo = IO::Socket::INET->new(PeerAddr => 'wuarchive.wustl.edu',
                                  PeerPort => 'echo(7)',
                                  Type   => SOCK_STREAM,
                                  Proto  => 'tcp')
             or die "Can't connect: $!\n";
 

Recall from Chapter 1 that the " => " symbol is accepted by Perl as a synonym for ",". The newlines between the argument pairs are for readability only. In shorter examples, we put all the name/argument pairs on a single line.

The list of arguments that you can pass to IO::Socket::INET is extensive. They are summarized in Table 5.1

Table 5.1. Arguments to IO::Socket::INET->new()

Argument	Description	Value
PeerAddr	Remote host address	<hostname or address>[:<port>]
PeerHost	Synonym for PeerAddr
PeerPort	Remote port or service	<service name or number>
LocalAddr	Local host bind address	<hostname or address>[:port]
LocalHost	Synonym for LocalAddr
LocalPort	Local host bind port	<service name or port number>
Proto	Protocol name (or number)	<protocol name or number>
Type	Socket type	SOCK_STREAM | SOCK_DGRAM | ...
Listen	Queue size for listen	<integer>
Reuse	Set SO_REUSEADDR before binding	<boolean>
Timeout	Timeout value	<integer>
MultiHomed	Try all adresses on multihomed hosts	<boolean>

The PeerAddr and PeerHost arguments are synonyms which are used to specify a host to connect to. When IO::Socket::INET is passed either of these arguments, it will attempt to connect() to the indicated host. These arguments accept a hostname, an IP address, or a combined hostname and port number in the format that we discussed earlier for the simple form of new(). If the port number is not embedded in the argument, it must be provided by PeerPort.

PeerPort indicates the port to connect to, and is used when the port number is not embedded in the hostname. The argument can be a numeric port number, a symbolic service name, or the combined form, such as "ftp(22)."

The LocalAddr, LocalHost, and LocalPort arguments are used by programs that are acting as servers and wish to accept incoming connections. LocalAddr and LocalHost are synonymous, and specify the IP address of a local network interface. LocalPort specifies a local port number. If IO::Socket::INET sees any of these arguments, it constructs a local address and attempts to bind() to it.

The network interface can be specified as an IP address in dotted-quad form, as a DNS hostname, or as a packed IP address. The port number can be given as a port number, as a service name, or using the "service(port)" combination. Itis also possible to combine the local IP address with the port number, as in "127.0.0.1:http(80)." In this case, IO::Socket::INET will take the port number from LocalAddr, ignoring the LocalPort argument.

If you specify LocalPort but not LocalAddr, then IO::Socket::INET binds to the INADDR_ANY wildcard, allowing the socket to accept connections from any of the host's network interfaces. This is usually the behavior that you want.

Stream-oriented programs that wish to accept incoming connections should also specify the Listen and possibly Reuse arguments. Listen gives the size of the listen queue. If the argument is present, IO::Socket will call listen() after creating the new socket, using the argument as its queue length. This argument is mandatory if you wish to call accept() later.

Reuse, if a true value, tells IO::Socket::INET to set the SO_REUSEADDR option on the new socket. This is useful for connection-oriented servers that need to be restarted from time to time. Without this option, the server has to wait a few minutes between exiting and restarting in order to avoid "address in use" errors during the call to bind().

Proto and Type specify the protocol and socket type. The protocol may be symbolic (e.g., "tcp") or numeric, using the value returned by getprotoby name(). Type must be one of the SOCK_* constants, such as SOCK_STREAM. If one or more of these options is not provided, IO::Socket::INET guesses at the correct values from context. For example, if Type is absent, IO::Socket:: INET infers the correct type from the protocol. If Proto is absent but a service name was given for the port, then IO::Socket::INET attempts to infer the correct protocol to use from the service name. As a last resort, IO::Socket::INET defaults to "tcp."

Timeout sets a timeout value, in seconds, for use with certain operations. Currently, timeouts are used for the internal call to connect() and in the accept() method. This can be handy to prevent a client program from hanging indefinitely if the remote host is unreachable.

The MultiHomed option is useful in the uncommon case of a TCP client that wants to connect to a host with multiple IP addresses and doesn't know which IP address to use. If this argument is set to a true value, the new(), method uses gethostbyname() to look up all the IP addresses for the hostname specified by PeerAddr. It then attempts a connection to each of the host's IP addresses in turn until one succeeds.

To summarize, TCP clients that wish to make outgoing connections should call new() with a Proto argument of tcp, and either a PeerAddr with an appended port number, or a PeerAddr/PeerPort pair. For example:

my $sock = IO::Socket::INET->new(Proto  => 'tcp',
                                  PeerAddr => 'www.yahoo.com',
                                  PeerPort => 'http(80)');
 

TCP servers that wish to accept incoming connections should call new(), with a Proto of " tcp ", a LocalPort argument indicating the port they wish to bind with, and a Listen argument indicating the desired listen queue length:

my $listen = IO::Socket::INET->new(Proto   => 'tcp',
                                    LocalPort => 2007,
                                    Listen  => 128);
 

As we will discuss in Chapter 19, UDP applications need provide only a Proto argument of " udp " or a Type argument of SOCK_DGRAM. The idiom is the same for both clients and servers:

my $udp = IO::Socket::INET->new(Proto => 'udp');
 

IO::Socket Object Methods

Once a socket is created, you can use it as the target for all the standard I/O functions, including print(), read(), <>, sysread(), and so forth. The object-oriented method calls discussed in Chapter 2 in the context of IO::File are also available. In addition, IO::Socket::INET adds the following socket-specific methods to its objects:

$connected_socket = $listen_socket->accept() ($connected_socket,$remote_addr) = $listen_socket->accept() The accept() method performs the same task as the like-named call in the function-oriented API. Valid only when called on a listening socket object, accept() retrieves the next incoming connection from the queue, and returns a connected session socket that can be used to communicate with the remote host. The new socket inherits all the attributes of its parent, except that it is connected. When called in a scalar context, accept() returns the connected socket. When called in an array context, accept() returns a two-element list, the first of which is the connected socket, and the second of which is the packed address of the remote host. You can also recover this address at a later time by calling the connected socket's peername() method. $return_val = $sock->connect ($dest_addr) $return_val = $sock->bind ($my_addr) $return_val = $sock->listen ($max_queue) These three TCP-related methods are rarely used because they are usually called automatically by new(). However, if you wish to invoke them manually, you can do so by creating a new TCP socket without providing either a PeerAddr or a Listen argument: $sock = IO::Socket::INET->new(Proto=>'tcp'); $dest_addr = sockaddr_in(...) # etc. $sock->connect($dest_addr); $return_val = $sock->connect ($port, $host) $return_val = $sock->bind ($port, $host) For your convenience, connect() and bind() both have alternative two-argument forms that take unpacked port and host addresses rather than a packed address. The host address can be given in dotted-IP form or as a symbolic hostname. $return_val = $socket->shutdown($how) As in the function-oriented call, shutdown() is a more forceful way of closing a socket. It will close the socket even if there are open copies in forked children. $how controls which half of the bidirectional socket will be closed, using the codes shown in Table 3.1. $my_addr = $sock->sockname() $her_addr = $sock->peername() The sockname() and peername() methods are simple wrappers around their function-oriented equivalents. As with the built-in functions, they return packed socket addresses that must be unpacked using sockaddr_in(). $result = $sock->sockport() $result = $sock->peerport() $result = $sock->sockaddr() result = $sock->peeraddr() These four methods are convenience functions that unpack the values returned by sockname() and peername(). sockport() and peerport() return the port numbers of the local and remote endpoints of the socket, respectively. sockaddr(), and peeraddr() return the IP address of the local and remote endpoints of the socket as binary structures suitable for passing to gethostbyaddr(). To convert theresult into dotted-quad form, you still need to invoke inet_ntoa(). $my_name = $sock->sockhost() $her_name = $sock->peerhost() These methods go one step further, and return the local and remote IP addresses in full dotted-quad form ("aa.bb.cc.dd"). If you wish to recover the DNS name of the peer, falling back to the dotted-quad form in case of a DNS failure, here is the idiom: $peer = gethostbyaddr($sock->peeraddr,AF_INET) || $sock->peerhost; $result = $sock->connected() The connected() method returns true if the socket is connected to a remote host, false otherwise. It works by calling peername(). $protocol = $sock->protocol() $type = $sock->socktype() $domain = $sock->sockdomain() These three methods return basic information about the socket, including its numeric protocol, its type, and its domain. These methods can be used only to get the attributes of a socket object. They can't be used to change the nature of an already-created object. $value = $sock->sockopt($option [,$value]) The sockopt() method can be used to get and/or set a socket option. It is a front end for both getsockopt() and setsockopt(). Called with a single numeric argument, sockopt() retrieves the current value of the option. Called with an option and a new value, sockopt() sets the option to the indicated value, and returns a result code indicating success or failure. There is no need to specify an option level, as you do with getsockopt(), because the SOL_SOCKET argument is assumed. Unlike the built-in getsockopt(), the object method automatically converts the packed argument returned by the underlying system call into an integer, so you do not need to unpack the option values returned by sockopt(). As we discussed earlier, the most frequent exception to this is the SO_LINGER option, which operates on an 8-byte linger structure as its argument. $val = timeout([$timeout]) timeout() gets or sets the timeout value that IO::Socket uses for its connect(), and accept() methods. Called with a numeric argument, it sets the timeout value and returns the new setting. Otherwise, it returns the current setting. The timeout value is not currently used for calls that send or receive data. The eval{} trick, described in Chapter 2, can be used to achieve that result. $bytes = $sock->send ($data [, $flags ,$destination]) $address = $sock-> recv ($buffer,$length [,$flags]) These are front ends for the send() and recv() functions, and are discussed in more detail when we discuss UDP communications in Chapter 19.

An interesting side effect of the timeout implementation is that setting theIO::Socket::INET timeout makes the connect() and accept() calls interruptable by signals. This allows a signal handler to gracefully interrupt a program that is hung waiting on a connect() or accept(). We will see an example of this in the next section.

More Practical Examples

We'll now look at additional examples that illustrate important aspects of the IO::Socket API. The first is a rewritten and improved version of the reverse echo server from Chapter 4. The second is a simple Web client.

Reverse Echo Server Revisited

We'll now rewrite the reverse echo server in Figure 4.2. In addition to being more elegant than the earlier version (and easier to follow, in my opinion), this version makes several improvements on the original. It replaces the dangerous signal handler with a handler that simply sets a flag and returns. This avoids problems stemming from making I/O calls within the handler and problems with exit() on Windows platforms. The second improvement is that the server resolves the names of incoming connections, printing the remote hostname and port number to standard error. Finally, we will observe the Internet convention that servers use CRLF sequences for line endings. This means that we will set $/ to CRLF and append CRLF to the end of all our writes. Figure 5.4 lists the code for the improved server.

Lines 17: Initialize script We turn on strict syntax checking and load the IO::Socket module. We import the default constants and the newline-related constants by importing the tags :DEFAULT and :crlf.

We define our local port as a constant, and initialize the byte counters for tracking statistics. We also set the global $/ variable to CRLF in accordance with the network convention.

Lines 89: Install INT signal handler We install a signal handler for the INT signal, so that the server will shut down gracefully when the user presses the interrupt key. The improved handler simply sets the flag named $quit to true.

Lines 1015: Create the socket object We recover the port number from the command line or, if no port number is provided, we default to the hard-coded constant. We now call IO::Socket::INET->new() with arguments that cause it to create a listening socket bound to the specified local port. Other arguments set the SO_REUSEADDR option to true, and specify a 1-hour timeout (60*60 seconds) for the accept() operation.

The Timeout parameter makes each call to the accept() method return undef if an incoming connection has not been received within the specified time. However, our motivation for activating this feature is not for its own sake, but for the fact that it changes the behavior of the method so that it is not automatically restarted after being interrupted by a signal. This allows us to interrupt the server with the ^C key without bothering to wrap accept() in an eval{} block (see Chapter 2, Timing Out Slow System Calls).

Lines 1631: Enter main loop After printing a status message, we enter a loop that continues until the INT interrupt handler has set $quit to true. Each time through the loop, we call the socket's accept() method. If the accept() method completes without being interrupted by a signal or timing out on its own, it returns a new connected socket object, which we store in a variable named $session. Otherwise, accept() returns undef, in which case we go back to the beginning of the loop. This gives us a chance to test whether the interrupt handler has set $quit to true.

Lines 1921: Get remote host's name and port We call the connected socket's peeraddr() method to get the packed IP address at the other end of the connection, and attempt to translate it into a hostname using gethostbyaddr(). If this fails, it returns undef, and we call the peerhost() method to give us the remote host's address in dotted-quad form.

We get the remote host's port number using peerport(), and print the address and port number to standard error.

Lines 2230: Handle the connection We read lines from the connected socket, reverse them, and print them back out to the socket, keeping track of the number of bytes sent and received while we do so. The only change from the earlier example is that we now terminate each line with CRLF.

When the remote host is done, we get an EOF when we try to read from the connected socket. We print out a warning, close the connected socket, and go back to the top of the loop to accept() again.

When we run the script, it acts like the earlier version did, but the status messages give hostnames rather than dotted-IP addresses.

% tcp_echo_serv2.pl
 waiting for incoming connections on port 2007...
 Connection from [localhost,2895]
 Connection from [localhost,2895] finished
 Connection from [formaggio.cshl.org,12833]
 Connection from [formaggio.cshl.org,12833] finished
 ^C
 bytes_sent = 50, bytes_received = 50
 

A Web Client

In this section, we develop a tiny Web client named web_fetch.pl. It reads a Universal Resource Locator (URL) from the command line, parses it, makes the request, and prints the Web server's response to standard output. Because it returns the raw response from the Web server without processing it, it is very useful for debugging misbehaving CGI (Common Gateway Interface) scripts and other types of Web-server dynamic content.

The Hypertext Transfer Protocol (HTTP) is the main protocol for Web servers. Part of the power and appeal of the protocol is its simplicity. A client wishing to fetch a document makes a TCP connection to the desired Web server, sends a brief request, and then receives the response from the server. After the document is delivered, the Web server breaks the connection. The hardest part is parsing the URL. HTTP URLs have the following general format:

http://hostname:port/path/to/document#fragment

All HTTP URLs begin with the prefix http://. This is followed by a hostname such as www.yahoo.com, a colon, and the port number that the Web server is listening on. The colon and port may be omitted, in which case the standard server port 80 is assumed. The hostname and port are followed by the path to the desired document using UNIX-like file path conventions. The path may be followed by a "#" sign and a fragment name, which indicate a subsection in the document that the Web browser should scroll to.

Our client will parse the components of this URL into the hostname:port combination and the path. We ignore the fragment name. We then connect to the designated server using a TCP socket and send an HTTP request of this form:

GET /path/to/document HTTP/1.0 CRLF CRLF
 

The request consists of the request method "GET" followed by a single space and the designated path, copied verbatim from the URL. This is followed by another space, the protocol version number HTTP/1.0, and two CRLF pairs. After the request is sent, we wait for a response from the server. A typical response looks like this:

HTTP/1.1 200 OK
 Date: Wed, 01 Mar 2000 17:00:41 GMT
 Server: Apache/1.3.6 (UNIX)
 Last-Modified: Mon, 31 Jan 2000 04:28:15 GMT
 Connection: close
 Content-Type: text/html
 
 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
 <html> <head> <title> Presto Home Page </title> </head>
 <body>
 <h1>Welcome to Presto</h1>
 ...
 

The response is divided into two parts: a header containing information about the returned document, and the requested document itself. The two parts are separated by a blank line formed by two CRLF pairs.

We will delve into the structure of HTTP responses in more detail in Chapter 9, where we discuss the LWP library, and Chapter 13, where we develop a more sophisticated Web client capable of retrieving multiple documents simultaneously. The only issue to worry about here is that, whereas the header is guaranteed by the HTTP protocol to be nice human-readable lines of text, each terminated by a CRLF pair, the document itself can have any format. In particular, we must be prepared to receive binary data, such as the contents of a GIF or MP3 file.

Figure 5.5 shows the web_fetch.pl script in its entirety.

Lines 15: Initialize module We turn on strict syntax checking and load the IO::Socket module, importing the default and newline-related constants. As in previous examples, we are dealing with CRLF-delimited data. However, in this case, we set $/ to be a pair of CRLF sequences. Later, when we call the <> operator, it will read the entire header down through the CRLF pair that terminates it.

Lines 68: Parse URL We read the requested URL from the command line and parse it using a pattern match. The match returns the hostname, possibly followed by a colon and port number, and the path up through, but not including, the fragment.

Lines 910: Open socket We open a socket connected to the remote Web server. If the URL contained the port number, it is included in the hostname passed to PeerAddr, and the PeerPort argument is ignored. Otherwise, PeerPort specifies that we should connect to the standard "http" service, port 80.

Line 11: Send the request We send an HTTP request to the server using the format described earlier.

Lines 1214: Read and print the header Our first read is line-oriented. We read from the socket using the <> operator. Because $/ is set to a pair of CRLF sequences, this read grabs the entire header up through the blank line. We now print the header, but since we don't particularly want extraneous CRs to mess up the output, we first replace all occurrence of $CRLF with a logical newline ("\n", which will evaluate to whatever is the appropriate newline character for the current platform.

Line 15: Read and print the document Our subsequent reads are byte-oriented. We call read() in a tight loop, reading up to 1024 bytes with each operation, and immediately printing them out with print(). We exit when read() hits the EOF and returns 0.

Here is an example of what web_fetch.pl looks like when it is asked to fetch the home page for www.cshl.org:

% web_fetch.pl http://www.cshl.org/
 HTTP/1.1 200 OK
 Server: Netscape-Enterprise/3.5.1C
 Date: Wed, 16 Aug 2000 00:46:12 GMT
 Content-type: text/html
 Last-modified: Fri, 05 May 2000 13:19:29 GMT
 Content-length: 5962
 Accept-ranges: bytes
 Connection: close
 
 <HTML>
 <HEAD>
 <TITLE>Cold Spring Harbor Laboratory</TITLE>
 
 <META NAME="GENERATOR" CONTENT="Adobe PageMill 2.0 Mac"> <META
 Name="keywords" Content="DNA, genes, genetics, genome, genome 
 sequencing, molecular biology, biological science, cell biology, 
 James D. Watson, Jim Watson, plant genetics, plant biology, 
 bioinformatics, neuroscience, neurobiology, cancer, drosophila, 
 Arabidopsis, double-helix, oncogenesis, Cold Spring Harbor 
 Laboratory, CSHL">
 ...
 

Although it seems like an accomplishment to fetch a Web page in a mere 15 lines of code, client scripts that use the LWP module can do the same thingand morewith just a single line. We discuss how to use LWP to access the Web in Chapter 9.

Performance and Style

Although the IO::Socket API simplifies programming, and is generally a big win in terms of development effort and code maintenance, it has some drawbacks over the built-in function-oriented interface.

If memory usage is an issue, you should be aware that IO::Socket adds to the Perl process's memory "footprint" by a significant amount: approximately 800 K on my Linux/Intel-based system, and more than double that on a Solaris system.

The object-oriented API also slows program loading slightly. On my laptop, programs that use IO::Socket take about half a second longer to load than those that use the function-oriented Socket interface. Fortunately, the execution speed of programs using IO::Socket are not significantly different from the speeds of classical interface. Network programs are generally limited by the speed of the network rather than the speed of processing.

Nevertheless, for the many IO::Socket methods that are thin wrappers around the corresponding system calls and do not add significant functionality, I prefer to use the IO::Socket as plain filehandles rather than use the object-oriented syntax. For example, rather than writing:

$socket->syswrite("A man, a plan, a canal, panama!");
 

I will write:

syswrite ($socket,"A man, a plan, a canal, panama!");
 

This has exactly the same effect as the method call, but avoids its overhead.

For methods that do improve on the function call, don't hesitate to use them. For example, the accept() method is an improvement over the built-in function because it returns a connected IO::Socket object rather than a plain filehandle. The method also has a syntax that is more Perl-like than the built-in.

Concurrent Clients

This chapter concludes by introducing a topic that is one of the central issues of Part III, the problem of concurrency.

A Gab Client, First Try

To motivate this discussion, let's write a simple client that can be used for interactive conversations with line-oriented servers. This program will connect to a designated host and port, and simply act as a direct conduit between the remote machine and the user. Everything the user types is transmitted across the wire to the remote host, and everything we receive from the remote host is echoed to standard output.

We can use this client to talk to the echo servers from this chapter and the last, or to talk with other line-oriented servers on the Internet. Common examples include FTP, SMTP, and POP3 servers.

We'll call this client gab1.pl, because it is the first in a series of such clients. A simplebut incorrectimplementation of the gab client looks like Figure 5.6.

Lines 16: Initialize module We load IO::Socket as before and recover the desired remote hostname and port number from the command-line arguments.

Lines 78: Create socket We create the socket using IO::Socket::INET->new() or, if unsuccessful, die with an error message.

Lines 921: Enter main loop We enter a loop. Each time through the loop we read one line from the socket and print it to standard output. Then we read a line of input from the user and print it to the socket.

Because the remote server uses CRLF pairs to end its lines, but the user types conventional newlines, we need to keep setting and resetting $/. The easiest way to do this is to place the code that reads a line from the socket in a little block, and to localize (with local) the $/ variable, so that its current value is saved on entry into the block, and restored on exit. Within the block, we set $/ to CRLF.

If we get an EOF from either the user or the server, we leave the loop by calling last.

At first, this straightforward script seems to work. For example, this transcript illustrates a session with an FTP server. The first thing we see on connecting with the server is its welcome banner (message code 220). We type in the FTP USER command, giving the name "anonymous," and get an acknowledgment. We then provide a password with PASS, and get another acknowledgment. Everything seems to be going smoothly.

% gab1.pl phage.cshl.org ftp
 220 phage.cshl.org FTP server ready.
 USER anonymous
 331 Guest login ok, send your complete e-mail address as password.
 PASS jdoe@nowhere.com
 230 Guest login ok, access restrictions apply.
 

Unfortunately, things don't last that way for long. The next thing we try is the HELP command, which is supposed to print a multiline summary of FTP commands. This doesn't go well. We get the first line of the expected output, and then the script stops, waiting for us to type the next command. We type another HELP, and get the second line of the output from the first HELP command. We type QUIT, and get the third line of the HELP command.

HELP
 214-The following commands are recognized (* =>'s unimplemented).
 HELP
 USER     PORT    STOR    MSAM*    RNTO    NLST    MKD   CDUP
 QUIT
 PASS     PASV    APPE    MRSQ*    ABOR    SITE    XMKD  XCUP
 QUIT
 ACCT*    TYPE    MLFL*   MRCP*    DELE    SYST    RMD   STOU
 QUIT
 ...
 

Clearly the script has gotten out of synch. As it is written, it can deal with only the situation in which a single line of input from the user results in a single line of output from the server. Having no way of dealing with multiline output, it can't catch up with the response to the HELP command.

What if we changed the line that reads from the server to something like this?

while ($from_server = <$socket>) {
   chomp $from_server;
   print $from_server,"\n";
 }
 

Unfortunately, this just makes matters worse. Now the script hangs after it reads the first line from the server. The FTP server is waiting for us to send it a command, but the script is waiting for another line from the server and hasn't even yet asked us for input, a situation known as deadlock.

In fact, none of the straightforward rearrangements of the read and print orders fix this problem. We either get out of synch or get hopelessly deadlocked.

A Gab Client, Second Try

What we need to do is decouple the process reading from the remote host from the process of reading from the socket. In fact, we need to isolate the tasks in two concurrent but independent processes that won't keep blocking each other the way the naive implementation of the gab client did.

On UNIX and Windows systems, the easiest way to accomplish this task is using the fork() command to create two copies of the script. The parent process will be responsible for copying data from standard input to the remote host, while the child will be responsible for the flow of data in the other direction. Unfortunately, Macintosh users do not have access to this call. A good but somewhat more complex solution that avoids the call to fork() is discussed in Chapter 12, Multiplexed Operations.

As it turns out, the simple part of the script is connecting to the server, forking a child, and having each process copy data across the network. The hard part is to synchronize the two processes so that they both quit gracefully when the session is done. Otherwise, there is a chance that one process may continue to run after the other has exited.

There are two scenarios for terminating the connection. In the first scenario, the remote server initiates the process by closing its end of the socket. In this case, the child process receives an EOF when it next tries to read from the server and calls exit(). It somehow has to signal to the parent that it is done. In the second scenario, the user closes standard input. The parent process detects EOF when it reads from STDIN and has to inform the child that the session is done.

On UNIX systems, there is a built-in way for children to signal parents that they have exited. The CHLD signal is sent automatically to a parent whenever one of its subprocesses has died (or have either stopped or resumed; we discuss this in more detail in Chapter 10). For the parent process to detect that the remote server has closed the connection it merely has to install a CHLD handler that calls exit(). When the child process detects that the server has closed the connection, the child will exit, generating a CHLD signal. The parent's signal handler is invoked, and the process now exits too.

The second scenario, in which the user closes STDIN, is a bit more complicated. One easy way is for the parent just to kill() its child after standard input has closed. There is, however, a problem with this. Just because the user has closed standard input doesn't mean that the server has finished sending output back to us. If we kill the child before it has received and processed all the pending information from the server, we may lose some information.

The cleaner way to do this is shown in Figure 5.7. When the parent process gets an EOF from standard input, it closes its end of the socket, thereby sending the server an end-of-file condition. The server detects the EOF, and closes its end of the connection, thereby propagating the EOF back to the child process. The child process exits, generating a CHLD signal. The parent intercepts this signal, and exits itself.

The beauty of this is that the child doesn't see the EOF until after it has finished processing any queued server data. This guarantees that no data is lost. In addition, the scheme works equally well when the termination of the connection is initiated by the server. The risk of this scheme is that the server may not cooperate and close its end of the connection when it receives an EOF. However, most servers are well behaved in this respect. If you encounter one that isn't, you can always kill both the parent and the child by pressing the interrupt key.

There is one subtle aspect to this scheme. The parent process can't simply close() its copy of the socket in order to send an EOF to the remote host. There is a second copy of the socket in the child process, and the operating system won't actually close a filehandle until its last copy is closed. The solution is for the parent to call shutdown(1) on the socket, forcefully closing it for writing. This sends EOF to the server without interfering with the socket's ability to continue to read data coming in the other direction. This strategy is implemented in Figure 5.8, in a script named gab2.pl.

Lines 17: Initialize module We turn on strict syntax checking, load IO::Socket, and fetch the host and port from the command line.

Line 8: Create the socket We create the connected socket in exactly the same way as before.

Lines 910: Call fork We call fork(), storing the result in the variable $child. Recall that if successful, fork() duplicates the current process. In the parent process, fork() returns the PID of the child; in the child process, fork() returns numeric 0.

In case of error, fork() returns undef. We check for this and exit with an error message.

Lines 1115: Parent process copies from standard input to socket The rest of the script is divided into halves. One half is the parent process, and is responsible for reading lines from standard input and writing to the server; the other half is the child, which is responsible for reading lines from the server and writing them to standard output.

In the parent process, $child is nonzero. For the reasons described earlier, we set up a signal handler for the CHLD signal. This handler simply calls exit(). We then call the user_to_host() subroutine, which copies user data from standard input to the socket.

When standard input is closed, user_to_host() returns. We call the socket's shutdown() method, closing it for writing. Now we go to sleep indefinitely, awaiting the expected CHLD signal that will terminate the process.

Lines 1619: Child process copies from socket to standard output In the child process, we call host_to_user() to copy data from the socket to standard output. This subroutine will return when the remote host closes the socket. We don't do anything special after that except to warn that the remote host has closed the connection. We allow the script to exit normally and let the operating system generate the CHLD message.

Lines 2026: The user_to_host() subroutine This subroutine is responsible for copying lines from standard input to the socket. Our loop reads a line from standard input, removes the newline, and then prints to the socket, appending a CRLF to the end. We return when standard input is closed.

Lines 2734: The host_to_user() subroutine This subroutine is almost the mirror image of the previous one. The only difference is that we set the $/ input record separator global to CRLF before reading from the socket. Notice that there's no reason to localize $/ in this case because changes made in the child process won't affect the parent. When we've read the last line from the socket, we return.

You may wonder why the parent goes to sleep rather than simply exit after it has shutdown() its copy of the socket. The answer is simply esthetic. As soon as the parent exits, the user will see the command-line prompt reappear. However, the child may still be actively reading from the socket and writing to standard output. The child's output will intermingle in an ugly way with whatever the user is doing at the command line. By sleeping until the child exits, the parent avoids this behavior.

You may also wonder about the call to exit() in the CHLD signal handler. While this is a problematic construction on Windows platforms because it causes crashes, the sad fact is that the Windows port of Perl does not generate or receive CHLD signals when a child process dies, so this issue is moot. To terminate gab2.pl on Windows platforms, press the interrupt key.

When we try to connect to an FTP server using the revised script, the results are much more satisfactory. Multiline results now display properly, and there is no problem of synchronization or deadlocking.

% gab2.pl phage.cshl.org ftp
 220 phage.cshl.org FTP server ready.
 USER anonymous
 331 Guest login ok, send your complete e-mail address as password.
 PASS ok@better.now
 230 Guest login ok, access restrictions apply.
 HELP
 214-The following commands are recognized (* =>'s unimplemented).
    USER     PORT    STO     RMSAM*   RNTO   NLST    MKD     CDUP
    PASS     PASV    APP     EMRSQ*   ABOR   SITE    XMKD    XCUP
    ACCT*    TYPE    MLFL*   MRCP*    DELE   SYST    RMD     STOU
    SMNT*    STRU    MAIL*   ALLO     CWD    STAT    XRMD    SIZE
    REIN*    MODE    MSND*   REST     XCWD   HELP    PWD     MDTM
    QUIT     RETR    MSOM*   RNFR     LIST   NOOP    XPWD    
 214 Direct comments to ftp-bugs@phage.cshl.org
 QUIT
 221 Goodbye.
 Connection closed by foreign host.
 

This client is suitable for talking to many line-oriented servers, but there is one Internet service that you cannot successfully access via this clientthe Telnet remote login service itself. This is because Telnet servers initially exchange some binary protocol information with the client before starting the conversation. If you attempt to use this client to connect to a Telnet port (port 23), you will just see some funny characters and then a pause as the server waits for the client to complete the protocol handshake. The Net::Telnet module (Chapter 6) provides a way to talk to Telnet servers.

Оставьте свой комментарий !
Ваше имя: Комментарий: Оба поля являются обязательными  Автор   Комментарий к данной статье