Redirection
Ninereasons (Talk | contribs) (edit description of redirectors) |
m (Reverted edits by 193.28.153.27 (Talk); changed back to last version by 24.20.109.51) |
||
(25 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
− | '''Redirection''' is sending the output of a program to somewhere other than where it would otherwise go - for example you can '''redirect''' the output of an [[ls]] command to a text file for later processing or to the [[grep]] command for filtering. | + | '''Redirection''' is sending the output of a program to somewhere other than where it would otherwise go - for example you can '''redirect''' the output of an [[ls]] command to a text file for later processing or to the [[grep]] command for filtering. Shell redirection operators include: '''>''', '''>>''', '''<''', '''<<''', and the ever popular '''|'''. |
> sends output to a file (may include [[special files]] such as [[/dev/null]]) | > sends output to a file (may include [[special files]] such as [[/dev/null]]) | ||
Line 9: | Line 9: | ||
If you're using the [[bash]] or [[bourne shell|bourne]] shells, you also have some special options available to you: you can redirect standard input, standard output and standard error messages with far greater flexibility and reliability. Other shells such as [[csh]] are notably limited in redirection capability, making them better suited to interactive use than to [[shell programming]] or other complex uses. | If you're using the [[bash]] or [[bourne shell|bourne]] shells, you also have some special options available to you: you can redirect standard input, standard output and standard error messages with far greater flexibility and reliability. Other shells such as [[csh]] are notably limited in redirection capability, making them better suited to interactive use than to [[shell programming]] or other complex uses. | ||
− | == | + | == Shell pipelines and redirection == |
− | + | There are three standard input/output file descriptors (''fd'') that are preconnected to the shell process running on your FreeBSD machine. Most commands that you would run from the command line expect these file descriptors to be open and accessible. The first (''fd 0'') provides a stream of ''input'' to your program, the second (''fd 1'') provides a stream of ''output'', and the third (''fd 2'') provides a stream of diagnostic messages (usually to your terminal). | |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | When you open a terminal, before running the shell the terminal device is opened three times to preconnect these file descriptors. The shell then inherits the file descriptors, and passes them on to each process run from the shell. | |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
+ | On FreeBSD systems it looks like this: | ||
+ | {| border="1" cellpadding="5" cellspacing="0" | ||
+ | |- | ||
+ | ! file descriptor || Stream || file descriptor file || device path | ||
+ | |- | ||
+ | | '''0''' || Standard input || <code>/dev/stdin</code> || /dev/fd/0 | ||
+ | |- | ||
+ | | '''1''' || Standard output || <code>/dev/stdout</code> || /dev/fd/1 | ||
+ | |- | ||
+ | | '''2''' || Standard error || <code>/dev/stderr</code> || /dev/fd/2 | ||
+ | |} | ||
+ | |||
+ | The example below demonstrates that by default, all of these input/output streams are directed to your terminal (color is added). | ||
+ | $ for i in stdin stdout stderr; do echo $i stuff > /dev/$i; done | ||
+ | <span style=color:red > stdin stuff</span> | ||
+ | <span style=color:green>stdout stuff</span> | ||
+ | <span style=color:blue >stderr stuff</span> | ||
+ | |||
+ | Redirection means that, the file descriptor is temporarily reassigned to somewhere other than the terminal device (a file, a pipe, another file descriptor). When the next process inherits the open ''fd'' from the shell the stream of data is passed along, for example, by writing to <code>stdout</code>, which has been temporarily reassigned to the <code>stdin</code> for the next process by means of a ''pipe'' ('''|'''). This handing off of open file descriptors by the shell, from one sub-process to another, is called a "shell pipeline". | ||
+ | |||
+ | There are ''fd n'' (0-9) descriptors potentially available, but only 0-2 are preconnected to the shell; the others must be created from the shell. In some shells, a standard file descriptor can be detached and reassigned to another file descriptor for as long as the terminal device is open, using the [[exec]] command (examples appear below). This capability is just one case among many of why <code>sh</code> is useful for scripting, and <code>tcsh</code> (which lacks it) is not (''see the [http://www.faqs.org/faqs/unix-faq/shell/csh-whynot/ FAQ]). | ||
+ | |||
+ | == Redirection in <code>sh</code> compared to <code>tcsh</code> == | ||
+ | The c-shells (<code>tcsh</code> or <code>csh</code>) and the Bourne shells (<code>sh</code> or <code>bash</code>) do not handle redirection or piping in quite the same way. | ||
+ | |||
+ | === ''tcsh'' and ''sh'' === | ||
+ | # Write output to a file | ||
+ | #: % mycmd > out.txt | ||
+ | #:: <span style=color:blue >stderr stuff</span> | ||
+ | # Append output to a file | ||
+ | #: % mycmd >> out.txt | ||
+ | #:: <span style=color:blue >stderr stuff</span> | ||
+ | # Redirect the output of a remote command to ''local.txt''. | ||
+ | #: % localcmd "remotecmd" > local.txt | ||
+ | #:: <span style=color:blue >stderr stuff</span> | ||
+ | # Same command as above, showing only the changes compared to ''local.txt''.<br />''Note: Many programs recognize '-' as a shortcut for '/dev/stdin'. These two commands are equivalent. '' | ||
+ | #: % localcmd "remotecmd" | diff /dev/stdin local.txt | ||
+ | #: % localcmd "remotecmd" | diff - local.txt | ||
+ | #:: <span style=color:blue >stderr stuff</span> | ||
+ | # Direct <code>stdout</code>+<code>stderr</code> to file | ||
+ | #: % mycmd >& out.txt | ||
+ | # Sort lines of jumble.txt into sorted.txt<br />''Note: the sequence in which redirection appears is not important. All of the following are exactly equivalent.'' | ||
+ | #: % <jumble.txt sort >sorted.txt | ||
+ | #: % >sorted.txt sort <jumble.txt | ||
+ | #: % sort < jumble.txt >sorted.txt | ||
+ | #: % <jumble.txt>sorted.txt sort | ||
+ | #:: <span style=color:blue>stderr stuff</span> | ||
+ | # Sort unique lines of jumble.txt into sorted.txt | ||
+ | #: % <jumble.txt sort | uniq >sorted.txt | ||
+ | #:: <span style=color:blue>stderr stuff</span> | ||
+ | # Sort ''HERE doc'' delimited by "lines" | ||
+ | #: % <<lines sort<br /> | ||
+ | #::? a second line<br /> | ||
+ | #::? a first line<br /> | ||
+ | #::lines<br /><span style=color:green >a first line</span><br /><span style=color:green >a second line</span><br /><span style=color:blue>stderr stuff</span> | ||
+ | |||
+ | === ''tcsh'' only === | ||
+ | # Discard errors, watch output (probably evil)<br />''Note: There is no reliable way to do this in tcsh. Here we exploit the fact that terminal reads from <code>stdin</code>. '' | ||
+ | #: % ( mycmd > /dev/stdin ) > & /dev/null | ||
+ | #:: <span style=color:green > stdout stuff</span> | ||
+ | #: % ( mycmd > /dev/tty ) > & /dev/null | ||
+ | #:: <span style=color:green > stdout stuff</span> | ||
+ | # Append output to out.txt; discard messages | ||
+ | #: % (mycmd >> out.txt) >& /dev/null | ||
+ | # Write output to out.txt; store and watch errors<br />''Note: Compare the same task in ''sh''. '' | ||
+ | #: % ( mycmd > out.txt ) | & tee err.txt | ||
+ | #:: <span style=color:blue >stderr stuff</span> | ||
+ | |||
+ | === ''sh'' only === | ||
+ | # Discard errors, watch output | ||
+ | #: $ mycmd 2> /dev/null | ||
+ | #:: <span style=color:green > stdout stuff</span> | ||
+ | # Append output to out.txt; discard messages | ||
+ | #: $ mycmd 2> /dev/null >> out.txt | ||
+ | # Write output to out.txt; store and watch errors<br />''Note: Compare the same task in ''tcsh''. '' | ||
+ | #: $ mycmd 2>&1 > out.txt | tee err.txt | ||
+ | #:: <span style=color:blue >stderr stuff</span> | ||
+ | # Write messages to err.txt; write output to out.txt and copy output to terminal | ||
+ | #: $ mycmd 2> err.txt | tee out.txt | ||
+ | #:: <span style=color:green>stdout stuff</span> | ||
+ | # Assign a variable from stored.txt | ||
+ | #: $ <stored.txt read var; mycmd $var | ||
+ | #:: <span style=color:blue>stderr stuff</span> | ||
+ | #:: <span style=color:green>stdout stuff</span> | ||
+ | # Assign first three lines of stored.txt to three different variables | ||
+ | #: $ exec 3<&0; exec <stored.txt; read v1; read v2; read v3; exec 0<&3 3<&-; echo $v1 $v2 $v3 | ||
+ | #:: <span style=color:blue>stderr stuff</span> | ||
+ | #:: <span style=color:green>stdout stuff</span> | ||
+ | # Use all unique lines in stored.txt as an argument variable for ''mycmd'', appending to result.txt | ||
+ | #: $ exec 3<&0; exec <stored.txt; sort | uniq | while read line; do mycmd $line >> result.txt ; done; exec 0<&3 3<&-; | ||
+ | #:: <span style=color:blue>stderr stuff</span> | ||
+ | : ''Note:'' By default, redirection pointed right represents <code>stdout</code>, so that these two commands are exactly equivalent: | ||
+ | :: $ mycmd 1> out.txt | ||
+ | :: $ mycmd > out.txt | ||
+ | ::: <span style=color:blue >stderr stuff</span> | ||
+ | : ''Note:'' By default, redirection pointed left represents <code>stdin</code>, so that these two commands are exactly equivalent: | ||
+ | :: $ mycmd 0< in.txt | ||
+ | :: $ mycmd < in.txt | ||
+ | ::: <span style=color:blue >stderr stuff</span> | ||
+ | ::: <span style=color:green >stdout stuff</span> | ||
+ | : ''Note:'' To close a file descriptor, say <code>n<&-</code> | ||
+ | :: $ mycmd >out.txt 1<&-; | ||
+ | :: ''-sh: fails without messages'' | ||
+ | :: <span style=color:blue >-bash: mycmd: write error: Bad file descriptor</span> | ||
+ | |||
+ | === A little more about ''fd n'' in <code>sh</code> === | ||
+ | |||
+ | Let's say you want to send output to your screen and errors to another command. You '''can't''' just do | ||
+ | samizdata# myprogram 1>&2 2>&1 | command > errors.txt | ||
+ | because when you switched output to the messages file descriptor, it's done ''right away''. When you tried to switch errors to output there's nothing left to pipe. It's exactly as though you had said | ||
+ | samizdata# myprogram 1>/dev/stderr | command > errors.txt | ||
+ | ''command'' receives no data, because it's all been sent to your screen. ''errors.txt'' will be empty. If you had tried it the other way, it's just as though you would have said: | ||
+ | samizdata# myprogram 2>/dev/stdout | command > errors.txt | ||
+ | In which case, errors and output are both piped to ''command'', which was not what you wanted. | ||
+ | |||
+ | This is where the other, normally unused, file descriptors 3-9 come in. You can use them as place-holders, like this: | ||
+ | samizdata# myprogram 3>&2 2>&1 1>&3 | command > errors.txt | ||
+ | |||
+ | Let's step through what this does. | ||
+ | # ''fd 3'' points to <code>stderr</code> | ||
+ | # ''fd 2'' points to <code>stdout</code> | ||
+ | # ''fd 1'' points to ''fd 3'' (which is connected to <code>stderr</code>) | ||
+ | # ''|'' passes the open file descriptor connected to <code>stdout</code> (''fd 2'') making it <code>stdin</code> for ''command'' | ||
+ | # ''>'' redirects the output of ''command'' to ''errors.txt''. | ||
+ | |||
+ | So, what you'll see on your screen is the output of ''myprogram'' on <code>stderr</code> (not passed to ''command''), followed by errors (if any) from ''command''. In short, what's been accomplished is, a swap of errors for output, from ''myprogram'' to ''command'', the output of which is redirected to ''errors.txt''. There is no reliable way to do the same thing in <code>tcsh</code>. | ||
+ | |||
+ | Also, see the [http://www.freebsd.org/cgi/man.cgi?query=mkfifo&apropos=0&sektion=1&manpath=FreeBSD+6.1-RELEASE&format=html man page] for [[mkfifo]], a utility for creating arbitrary [[FIFO]] files. | ||
[[Category:FreeBSD Terminology]] [[Category:Common Tasks]] | [[Category:FreeBSD Terminology]] [[Category:Common Tasks]] |
Latest revision as of 18:32, 31 December 2007
Redirection is sending the output of a program to somewhere other than where it would otherwise go - for example you can redirect the output of an ls command to a text file for later processing or to the grep command for filtering. Shell redirection operators include: >, >>, <, <<, and the ever popular |.
> sends output to a file (may include special files such as /dev/null) >> appends output to a file (without overwriting it) < read file tostdin
<< read to stdin from<<delimiter
todelimiter
(a HERE doc). | sends output to a program (frequently, a system command like grep)
If you're using the bash or bourne shells, you also have some special options available to you: you can redirect standard input, standard output and standard error messages with far greater flexibility and reliability. Other shells such as csh are notably limited in redirection capability, making them better suited to interactive use than to shell programming or other complex uses.
Contents |
[edit] Shell pipelines and redirection
There are three standard input/output file descriptors (fd) that are preconnected to the shell process running on your FreeBSD machine. Most commands that you would run from the command line expect these file descriptors to be open and accessible. The first (fd 0) provides a stream of input to your program, the second (fd 1) provides a stream of output, and the third (fd 2) provides a stream of diagnostic messages (usually to your terminal).
When you open a terminal, before running the shell the terminal device is opened three times to preconnect these file descriptors. The shell then inherits the file descriptors, and passes them on to each process run from the shell.
On FreeBSD systems it looks like this:
file descriptor | Stream | file descriptor file | device path |
---|---|---|---|
0 | Standard input | /dev/stdin |
/dev/fd/0 |
1 | Standard output | /dev/stdout |
/dev/fd/1 |
2 | Standard error | /dev/stderr |
/dev/fd/2 |
The example below demonstrates that by default, all of these input/output streams are directed to your terminal (color is added).
$ for i in stdin stdout stderr; do echo $i stuff > /dev/$i; done stdin stuff stdout stuff stderr stuff
Redirection means that, the file descriptor is temporarily reassigned to somewhere other than the terminal device (a file, a pipe, another file descriptor). When the next process inherits the open fd from the shell the stream of data is passed along, for example, by writing to stdout
, which has been temporarily reassigned to the stdin
for the next process by means of a pipe (|). This handing off of open file descriptors by the shell, from one sub-process to another, is called a "shell pipeline".
There are fd n (0-9) descriptors potentially available, but only 0-2 are preconnected to the shell; the others must be created from the shell. In some shells, a standard file descriptor can be detached and reassigned to another file descriptor for as long as the terminal device is open, using the exec command (examples appear below). This capability is just one case among many of why sh
is useful for scripting, and tcsh
(which lacks it) is not (see the FAQ).
[edit] Redirection in sh
compared to tcsh
The c-shells (tcsh
or csh
) and the Bourne shells (sh
or bash
) do not handle redirection or piping in quite the same way.
[edit] tcsh and sh
- Write output to a file
- % mycmd > out.txt
- stderr stuff
- % mycmd > out.txt
- Append output to a file
- % mycmd >> out.txt
- stderr stuff
- % mycmd >> out.txt
- Redirect the output of a remote command to local.txt.
- % localcmd "remotecmd" > local.txt
- stderr stuff
- % localcmd "remotecmd" > local.txt
- Same command as above, showing only the changes compared to local.txt.
Note: Many programs recognize '-' as a shortcut for '/dev/stdin'. These two commands are equivalent.- % localcmd "remotecmd" | diff /dev/stdin local.txt
- % localcmd "remotecmd" | diff - local.txt
- stderr stuff
- Direct
stdout
+stderr
to file- % mycmd >& out.txt
- Sort lines of jumble.txt into sorted.txt
Note: the sequence in which redirection appears is not important. All of the following are exactly equivalent.- % <jumble.txt sort >sorted.txt
- % >sorted.txt sort <jumble.txt
- % sort < jumble.txt >sorted.txt
- % <jumble.txt>sorted.txt sort
- stderr stuff
- Sort unique lines of jumble.txt into sorted.txt
- % <jumble.txt sort | uniq >sorted.txt
- stderr stuff
- % <jumble.txt sort | uniq >sorted.txt
- Sort HERE doc delimited by "lines"
- % <<lines sort
- ? a second line
- ? a first line
- lines
a first line
a second line
stderr stuff
- ? a second line
- % <<lines sort
[edit] tcsh only
- Discard errors, watch output (probably evil)
Note: There is no reliable way to do this in tcsh. Here we exploit the fact that terminal reads fromstdin
.- % ( mycmd > /dev/stdin ) > & /dev/null
- stdout stuff
- % ( mycmd > /dev/tty ) > & /dev/null
- stdout stuff
- % ( mycmd > /dev/stdin ) > & /dev/null
- Append output to out.txt; discard messages
- % (mycmd >> out.txt) >& /dev/null
- Write output to out.txt; store and watch errors
Note: Compare the same task in sh.- % ( mycmd > out.txt ) | & tee err.txt
- stderr stuff
- % ( mycmd > out.txt ) | & tee err.txt
[edit] sh only
- Discard errors, watch output
- $ mycmd 2> /dev/null
- stdout stuff
- $ mycmd 2> /dev/null
- Append output to out.txt; discard messages
- $ mycmd 2> /dev/null >> out.txt
- Write output to out.txt; store and watch errors
Note: Compare the same task in tcsh.- $ mycmd 2>&1 > out.txt | tee err.txt
- stderr stuff
- $ mycmd 2>&1 > out.txt | tee err.txt
- Write messages to err.txt; write output to out.txt and copy output to terminal
- $ mycmd 2> err.txt | tee out.txt
- stdout stuff
- $ mycmd 2> err.txt | tee out.txt
- Assign a variable from stored.txt
- $ <stored.txt read var; mycmd $var
- stderr stuff
- stdout stuff
- $ <stored.txt read var; mycmd $var
- Assign first three lines of stored.txt to three different variables
- $ exec 3<&0; exec <stored.txt; read v1; read v2; read v3; exec 0<&3 3<&-; echo $v1 $v2 $v3
- stderr stuff
- stdout stuff
- $ exec 3<&0; exec <stored.txt; read v1; read v2; read v3; exec 0<&3 3<&-; echo $v1 $v2 $v3
- Use all unique lines in stored.txt as an argument variable for mycmd, appending to result.txt
- $ exec 3<&0; exec <stored.txt; sort | uniq | while read line; do mycmd $line >> result.txt ; done; exec 0<&3 3<&-;
- stderr stuff
- $ exec 3<&0; exec <stored.txt; sort | uniq | while read line; do mycmd $line >> result.txt ; done; exec 0<&3 3<&-;
- Note: By default, redirection pointed right represents
stdout
, so that these two commands are exactly equivalent:- $ mycmd 1> out.txt
- $ mycmd > out.txt
- stderr stuff
- Note: By default, redirection pointed left represents
stdin
, so that these two commands are exactly equivalent:- $ mycmd 0< in.txt
- $ mycmd < in.txt
- stderr stuff
- stdout stuff
- Note: To close a file descriptor, say
n<&-
- $ mycmd >out.txt 1<&-;
- -sh: fails without messages
- -bash: mycmd: write error: Bad file descriptor
[edit] A little more about fd n in sh
Let's say you want to send output to your screen and errors to another command. You can't just do
samizdata# myprogram 1>&2 2>&1 | command > errors.txt
because when you switched output to the messages file descriptor, it's done right away. When you tried to switch errors to output there's nothing left to pipe. It's exactly as though you had said
samizdata# myprogram 1>/dev/stderr | command > errors.txt
command receives no data, because it's all been sent to your screen. errors.txt will be empty. If you had tried it the other way, it's just as though you would have said:
samizdata# myprogram 2>/dev/stdout | command > errors.txt
In which case, errors and output are both piped to command, which was not what you wanted.
This is where the other, normally unused, file descriptors 3-9 come in. You can use them as place-holders, like this:
samizdata# myprogram 3>&2 2>&1 1>&3 | command > errors.txt
Let's step through what this does.
- fd 3 points to
stderr
- fd 2 points to
stdout
- fd 1 points to fd 3 (which is connected to
stderr
) - | passes the open file descriptor connected to
stdout
(fd 2) making itstdin
for command - > redirects the output of command to errors.txt.
So, what you'll see on your screen is the output of myprogram on stderr
(not passed to command), followed by errors (if any) from command. In short, what's been accomplished is, a swap of errors for output, from myprogram to command, the output of which is redirected to errors.txt. There is no reliable way to do the same thing in tcsh
.
Also, see the man page for mkfifo, a utility for creating arbitrary FIFO files.