The pipeline feature is one of the driving forces behind the Linux philosophy, a single character that changes everything about how you work.By connecting the output of one command to the input of another, you can chain small programs together, creating a tool that is far greater than the sum of its parts.If you’re struggling to see the value in pipes, or just looking to understand their practical use a bit better, there’s no better place to begin than these common, effective examples.
grep | less Filter data and page the results A typical grep command can return many results, especially if you’re using trial and error to refine a regular expression.A command like grep '[Qq]' /usr/share/dict/words will produce more than one screen of output: To see more than just the last page of results, you’ll need to use a pager.The grep command doesn’t directly support one, but you can pipe any set of output to a pager and view it page-by-page: grep '[Qq]' /usr/share/dict/words | less tail | grep Extract relevant live data from a log file The tail command is useful any time you want to explore the last lines of a file, most often a log file.
Its -f option provides a live view of a file (or standard input), continuing to print lines that are appended to the file.It’s perfect for monitoring log files, like the apache access log: You’ll probably want to filter such log lines, however, looking for specific data that matches URLs, specific messages, response codes, and so on.The grep tool is great for this, and it pairs perfectly with tail in a pipeline like tail -f file | grep pattern.
You may forget about the order of these commands and expect them to work in reverse.However, piping grep to tail -f will cause the pipeline to end immediately because tail ignores -f when the input is a pipe.history | grep Quick access to commands you’ve run before The history command is incredibly useful, giving you access to commands you’ve previously run.
But your history contains a lot of detail; by default, in bash, it’s the last 500 commands.And it can be even more, especially if you’ve set up unlimited history.This is another perfect opportunity for grep to filter out only the lines that you’re really interested in: history | grep ls sort | uniq Simple summary info from structured data The sort and uniq filters are so useful when combined with other commands that you’ll see them used in pipelines time and time again.
uniq is short for "unique," and both commands do pretty much exactly what you’d expect, based on their names.Here’s an example of a pipeline that shows which IP addresses have visited a website, and how many times each has sent a request: </var/log/apache2/access_log cut -f1 -d' ' | sort | uniq -c This pipeline involves three commands, the first of which—cut—extracts the IP address field from a standard Apache access log.sort will order all the lines of input that are piped to it, while uniq removes duplicates and, with the -c option, counts the total number.
You’ll end up with a nice summary: Laptop With Linux Intel NUC13 Choose from three different Intel processors when you configure this mini PC, along with your choice of dozens of Linux distribution options.Expand Collapse df | tee Get disk space reports on screen and in a file The tee command was made for the pipeline; its core operation is to copy its standard input to standard output.That might seem useless, except for the one extra feature it provides: saving that same input to one or more files.
Normally, to save the output of a command, you can use redirection: echo "Hello world" > myfile.txt However, redirecting that output prevents you from viewing it on screen or piping it somewhere else: Using the tee command, you can save output to a file, but still have it in the pipeline, for further processing or output on screen: echo "Hello world" | tee myfile.txt You can even supply multiple arguments to tee, in case you want to save output to more than one file at once.tee is a great choice if you ever want to check the output of a command, but keep a record of it at the same time.For example, you can use the df command to report on disk usage: df -h | tee diskusage.txt Piping the output to tee means that it’ll still show up on screen, but you’ll get a permanent record stored in a file too.
echo | xargs Copy a file to multiple directories The xargs command is a bit like tee; it’s a “meta command” that alters how the command line itself operates.xargs lets you inject its input into arguments of another command, so it often comes in use as part of more complicated pipelines.By default, xargs converts each space-delimited value into an argument, appending them to a command that you pass it.
You can also use the -n option to call the command more than once, passing in a slice of arguments each time, as in this example: echo "dir1 dir2" | xargs -n 1 cp -v myfile.md The cp command will not copy a file to multiple directories, so this pipeline instructs xargs to provide one argument at a time, expanding to: cp -v myfile.md dir1 cp -v myfile.md dir2 curl | jq Process data retrieved from a web API If you’ve ever used curl to access a web API from the command-line, you’ve probably been faced with a long response containing JSON that you need to process further: The curl command is great at handling all sorts of web requests, but further processing requires other commands like, in this case, jq, the command-line JSON processor.Using jq, you can write filters to transform JSON data into more readable or appropriate forms.For example, the simplest thing you can do is use the identity operator (.) to copy the input, but validate and format it too: curl -s https://catfact.ninja/fact | jq '.' You can write all sorts of filters to slice and dice the data however you want.
The example above returns a single object with fact and length properties, which you can extract with a simple filter: curl -s https://catfact.ninja/fact | jq -r '.fact' du -sh * | sort -h Find your biggest files and directories The du program reports the total size of files and directories that you pass it: This is great, but you’re probably looking for the files that are taking up the most disk space.Unfortunately, du offers no built-in way to sort its output.Fortunately, there’s the good old sort command.
Subscribe to the newsletter for practical Linux pipeline tips Unlock practical shell skills by subscribing to the newsletter - clear, hands-on Linux pipeline patterns, ready-to-use command examples, and concise explanations for tools like grep, sort, tee, xargs, curl + jq.Get Updates By subscribing, you agree to receive newsletter and marketing emails, and accept our Terms of Use and Privacy Policy.You can unsubscribe anytime.
sort is a filter command that orders its input and prints it as output.You can use it to sort the standard output of du: du -s * | sort This is OK, but the output is in blocks, which isn’t the most readable.To do the same thing, with more readable sizes, use the -h option for both commands: du -sh * | sort -h Even though the output doesn’t sort naturally, sort is clever enough to recognize the format and sort kilobytes before megabytes, etc.
date | md5sum The quickest way to generate a decent password I reach for this one all the time when generating passwords; it’s quick, effective, and easy to remember.The md5sum will hash its input using the MD5 algorithm, and print the result: When I need a random set of hexadecimal characters, this is my go-to.You could use it for a variety of purposes, including generating random CSS colors, but it’s pretty convenient for password generation too.
This may not be the most secure way to pick a password.Please consult security advice before choosing a password, especially for high-risk uses! With practice, common pipelines will become second nature If there’s a pipeline you haven’t used before, it can take some time to become familiar with it.Practice any that seem useful and, after a short time, you’ll start piping to grep or sort so often that you’ll wonder how you ever survived without them.
Read More
