Pipes and FIFOs

What are Pipes?

Pipes are a communication pathway between two "ends": a "read end" and a "write end". These two ends could be in different processes or within the same process; the behavior is the same. When some number of bytes is written into the write end of the pipe, that information is made available on the read end of the pipe, in the exact order that it was written. So, for example, if on the write end of the pipe, you wrote the string "hello", you could read five bytes out of the read end of the same pipe and print it and it would give "hello". Both the read end and the write end are worked with through a file descriptor (there is a file descriptor pointing at the read end, and a file descriptor pointing at the write end). Once the data has been read out of the pipe from the read end, it is no longer in the pipe; i.e. if the program loses the information that was read out from the pipe, it is gone forever!

What are FIFO Pipes (FIFOs)?

FIFO (short for "First-In-First-Out") pipes are similar to pipes but differ in significant ways. Like normal pipes, FIFOs have messages written into them that are then made available to read in the order that they were written. Also like normal pipes, FIFOs do not retain information after data has been written out; if a reader doesn't process some data that it read out on the pipe and drops it, it is gone forever. The similarities end there, however. Whereas a normal pipe can have only one reader and one writer, a FIFO pipe can have one or more writers and one or more readers (the readers must coordinate among themselves when and what they will read from the pipe though; due to the nature of FIFOs, once one reader pulls data out of the pipe, that data will not be available to the other readers from the pipe). Additionally, pipes are unnamed; the only way to get two processes to be connected via a pipe is to have one process create a pipe and then fork itself. FIFOs are named; they exist in the file system at a location that can be made known by multiple processes; these processes then need only open the FIFO like a normal file to gain access to the pipe.

Other Properties of Pipes and FIFOs

Generally, pipes are not two-way streets, i.e. you cannot create one pipe and expect to write into the read end and read what you wrote out of the write end; they just aren't built that way. Some recent versions of Linux have added two-way pipes, but this is still a relatively new development and not standardized. If you really need two-way communication, see sockets.

By default, when a process opens a FIFO pipe, it opens it in what's called "blocking" mode. This means that a reader and a writer must exist for that FIFO before the call to open returns. To override this behavior and cause a FIFO to be opened with no read end, the O_NONBLOCK option must be specified for the pipe when it is opened.

It is an error to try to write to a pipe with no read end. The write() call will generate the signal SIGPIPE, which, if not caught, will cause the program to terminate.

When both read end and write end are open for a pipe (or at least one write end is open for a FIFO), calling the read() function on the read end(s) will block until something is available on the pipe to read (as a result of the write end(s) writing something to the pipe). This behavior can be disabled by opening with the O_NONBLOCK flag (for a FIFO) or by calling fcntl to manually set the O_NONBLOCK option (for a pipe). This will cause read in the above scenario not to block but to return immediately with an error status (EAGAIN or EWOULDBLOCK) which indicate that the read would have blocked had the O_NONBLOCK flag not been set.

When a read is attempted on the read end(s) of a pipe or FIFO with no write end, the read will succeed and report that it read an EOF (end-of-file) character. That is the only condition where the read ends of a pipe will encounter an EOF.

Functions Used When Working With Pipes and FIFOs

int pipe(int fildes[2]);

This function creates a pipe between two new file descriptors in a process. The filedes argument is, as listed, an array of two integers, one which will contain the file descriptor for the read end of the new pipe (filedes[0]), and the other while will contain the file descriptor for the write end of the new pipe (filedes[1]).

After calling this function, you should be able to write to filedes[1] and then immediately read from filedes[0] and see what was just written to the pipe.

int mkfifo(const char *path, mode_t mode);

This function creates a FIFO pipe at the given path with the specified access mode. The path is essentially the name of the FIFO. The mode is a bitwise OR of various permissions flags to set the permissions of the to-be-created FIFO pipe.

Keep in mind that this function does NOT return the file descriptor needed to read and write to the FIFO (the return value of this function, like most other Unix functions, is a status code). To obtain the file descriptor needed to read and write to the FIFO, you must call open() on that fie path after a successful call to mkfifo().

Use in Runtime

A pipe is used in the test framework to route standard output to both the terminal and to an internal buffer that stores the test output at the same time. Read more about how that is done in the Test Framework wiki page.

A FIFO pipe is used in the logger to aggregate all the logs that are to be output to Dawn and send all the logs, one at a time, to the network handler thread in charge of sending those logs to Dawn. Read mor about how that is done in the Logger wiki page.

Additional Resources

There aren't many good online resources on pipes and FIFOs. I would look at the Linux man pages on the above two functions, but that's about it. Hopefully the explanation here is decent enough to help you get to know the codebase a bit better.