Skip to content

A command line program for large scale buffering between piped programs

License

Notifications You must be signed in to change notification settings

tfenne/pipebuffer

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

20 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

pipebuffer

Build Status Language License

A simple command line program for buffering stdin/stdout between piped processes when the operating system's pipe buffer is insufficient. Modern linux limits the buffer between pipes in a pipeline to 64kb, which can cause bottlenecks when working with processes with "lumpy" IO profiles.

pipebuffer is particularly useful when dealing with large volumes of data and a mix of processes that work in "chunks" of data and more stream-oriented processes.

Usage

To use, you just replace:

foo | bar

with

foo | pipebuffer | bar

or

foo | pipebuffer --size=512m | bar

And, of course, you can use many pipebuffers together:

foo | pipebuffer --size=128m | bar | pipebuffer --size=64m | splat | pipebuffer --size=1g | whee

License

pipebuffer is open source software released under the MIT License.

Building

pipebuffer is written in Rust and currently requires Rust 1.7 (beta) or greater. You'll need Rust installed. You can download here, or run the first command below:

// Optionally install Rust: if 1.7 is stable, which it should be after ~3rd March 2016, omit '--channel=beta'
curl -sSf https://static.rust-lang.org/rustup.sh | sh -s -- --channel=beta

// Clone the repo
git clone https://github.com/tfenne/pipebuffer.git

// Build and run the tests
pushd pipebuffer && (cargo test; cargo build --release); popd

// Produces executable at ./pipebuffer/target/release/pipebuffer

About

A command line program for large scale buffering between piped programs

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages