Wikipedia - Benutzerbeiträge [de]

Pipeline (Unix)

2007-07-13T15:13:50Z

Unixguy: /* Example */ clarify grep '[a-z]' to remove double negative (inludes lines with lowercase letters, excludes blank lines)

[[Image:Pipeline.svg|thumb|A pipeline of three programs run on a text terminal]]
In [[Unix-like]] computer [[operating system]]s, a '''pipeline''' is the original ''[[pipeline (software)|software pipeline]]'': a set of [[process (computing)|process]]es chained by their [[standard streams]], so that the output of each process (''[[stdout]]'') feeds directly as input (''[[stdin]]'') of the next one. Each connection is implemented by an [[anonymous pipe]]. [[Filter (Unix)|Filter program]]s are often used in this configuration. The concept was invented by [[Douglas McIlroy]] for [[Unix shell]]s and it was named by analogy to a physical [[pipeline transport|pipeline]].

==Example==

Below is an example of a pipeline that implements a kind of [[spell checker]] for the [[World Wide Web|web]] resource indicated by a [[Uniform Resource Locator|URL]]. An explanation of what it does follows. (Some machines have /usr/share/dict/words instead.)

[[CURL|curl]] <nowiki>"http://en.wikipedia.org/wiki/Pipeline_(Unix)"</nowiki> | \
[[sed]] 's/[^a-zA-Z ]/ /g' | \
[[tr (program)|tr]] 'A-Z ' 'a-z\n' | \
[[grep]] '[a-z]' | \
[[Sort (Unix)|sort]] -u | \
[[comm (Unix)|comm]] -23 - /usr/dict/words

*First, '''<tt>curl</tt>''' obtains the [[HTML]] contents of a web page (could use <tt>wget</tt> on some systems).
*Second, '''<tt>sed</tt>''' this expression removes all characters which are not spaces or letters from the web page's content, replacing them with spaces.
*Third, '''<tt>tr</tt>''' changes all of the uppercase letters into lowercase and converts the spaces in the lines of text to newlines (each 'word' is now on a separate line).
*Fourth, '''<tt>grep</tt>''' includes only lines that contain at least one lowercase [[alphabetical]] character (removing any blank lines).
*Fifth, '''<tt>sort</tt>''' sorts the list of 'words' into alphabetical order, and the <tt>-u</tt> switch removes duplicates.
*Finally, '''<tt>comm</tt>''' finds lines in common between two files, <tt>-23</tt> suppresses lines unique to the second file, and those which are common to both, leaving only those which are found only in the first file named. The <tt>-</tt> in place of a filename causes <tt>comm</tt> to use its standard input (from the pipe line in this case). This results in a list of "words" (lines) which are not found in /usr/dict/words.
*The special character "|" tells the operating system to pipe the output from the previous command in the line into the next command in the line. That is, the output of the <tt>curl</tt> command is given as the input of the <tt>sed</tt> command.
*The character "\" is used to place all five lines into a single command line.

== Pipelines in command line interfaces ==
Most [[Unix shell]]s have a special syntax construct for the creation of pipelines. Typically, one simply writes the filter commands in sequence, separated by the [[ASCII]] [[vertical bar]] character "|" (which, for this reason, is often called "pipe character" by Unix users). The shell starts the processes and arranges for the necessary connections between their standard streams (including some amount of [[Buffer (computer science)|buffer]] storage). 

===Error stream===
By default, the standard error streams ("[[stderr]]") of the processes in a pipeline are not passed on through the pipe; instead, they are merged and directed to the [[computer console|console]]. However, many shells have additional syntax for changing this behaviour. In the [[C shell|csh]] shell, for instance, using "|&" instead of "| " signifies that the [[standard error]] stream too should be merged with the standard output and fed to the next process. The [[Bourne Shell]] can also merge standard error, using <tt>2>&1</tt>, as well as redirect it to a different file.

=== Pipemill ===
In the most commonly used simple pipelines the shell connects a series of sub-processes via pipes, and executes external commands within each sub-process. Thus the shell itself is doing no direct processing of the data flowing through the pipeline.

However, it's possible for the shell to perform processing directly. This construct generally looks something like:

command | while read var1 var2 ...; do
# process each line, using variables as parsed into var1, var2, etc
done

... which is referred to as a "pipemill" (since the <tt>while</tt> is "milling" over the results from the initial command.

'''Example:'''

find / /usr /var -mount -user foo -printf "%m %p\n" | while read mode filename; do
chown $NEWOWNER "$filename"
chmod $MODE "$filename"
done

(This example will traverse file directory trees changing the ownership of all files while preserving all permissions, including those that are often stripped off by many versions of the <tt>chown</tt> command).

There are a number of variations of the pipemill construct including:

ps lax | { read x; while read x owner pid parent x x x x x stat x; do
[ "$owner"="foo" -a "$stat"="Z" ] && kill "$parent"
done
}

(This example kills the parent processes for zombies owned/created by the user "foo").

Here the <tt>while</tt> loop is enclosed in a command group (the braces); and preceded by a <tt>read</tt> command which effectively "throws away" the first line from the <tt>ps</tt> command. (Of course, in this particular example it would be harmless to process the header line, as it wouldn't match the "$owner"= test). Note that the other references to the "x" variable are simply being used as placeholders for "throwing away" irrelevant fields from each line.

The defining characteristics of a "pipemill" are: some command or series of commands feeds data into a pipe from which a shell <tt>while</tt> loop reads and processes it.

==Creating pipelines programmatically==

Pipelines can be created under program control.
The <code>pipe()</code> [[system call]] asks the operating system to construct a new [[anonymous pipe]] object.
This results in two new, opened file descriptors in the process: the read-only end of the pipe, and the write-only end.
The pipe ends appear to be normal, anonymous [[file descriptor]]s, except that they have no ability to seek.

To avoid [[deadlock]] and exploit parallelism, the process with one or more new pipes will then, generally, call
<code>[[fork (computing)|fork()]]</code> to create new
processes. Each process will then close the end(s) of
the pipe that it will not be using before producing or consuming any data.
Alternatively, a process might create a new [[pthreads|thread]] and use the pipe to communicate between them.

''[[Named pipe]]s'' may also be created using <code>mkfifo()</code> or <code>mknod()</code> and then presented as the input or output file to programs as they are invoked. They allow multi-path pipes to be created, and are especially effective when combined with standard error redirection, or with [[tee (Unix)|tee]].

==Implementation==

In most Unix-like systems, all processes of a pipeline are started at the same time, with their streams appropriately connected, and managed by the [[scheduler]] together with all other processes running on the machine.  An important aspect of this, setting Unix pipes apart from other pipe implementations, is the concept of [[Buffer (computer science)|buffering]]: a sending program may produce 5000 [[bytes]] per [[second]], and a receiving program may only be able to accept 100 bytes per second, but no data are lost. Instead, the output of the sending program is held in a buffer, or [[Queue (data structure)|queue]]. When the receiving program is ready to read data, the operating system sends it data from the buffer, then removes that data from the buffer. If the buffer fills up, the sending program is suspended (blocked) until the receiving program has had a chance to read some data and make room in the buffer.

=== Network pipes ===
Tools like [[netcat]] and [[socat]] can connect pipes to TCP/IP [[socket]]s, following the Unix philosophy of "[[everything is a file]]".

== History==
[[Image: Automator_Icon.png|thumb|75px|right|Apple Automator logo]]
The pipeline concept and the vertical-bar notation was invented by [[Douglas McIlroy]], one of the authors of the early [[Unix shell|command shells]], after he noticed that much of the time they were processing the output of one program as the input to another. The idea was eventually ported to other operating systems, such as [[DOS]], [[OS/2]], [[Windows NT]], and [[BeOS]], often with the same notation.

The robot in the icon for [[Apple Computer|Apple]]'s [[Automator (software)|Automator]], which also uses a pipeline concept to chain repetitive commands together, holds a pipe as recognition of the application's Unix heritage.<ref name="automator"/>

=== Other operating systems ===
{{main|pipeline (software)}}

This feature of [[Unix]] was borrowed by other operating systems, such as [[Taos operating system|Taos]] and [[MS-DOS]], and eventually became the [[pipeline (software)|pipes and filters design pattern]] of [[software engineering]].

== See also ==
* [[Tee (Unix)]] for fitting together two pipes
* [[Pipeline (software)]] for the general software engineering concept.
* [[Pipeline (computer)]] for other computer-related pipelines.
* [[Hartmann pipeline]]
* [[Anonymous pipe]] a [[FIFO]] structure used for [[interprocess communication]]
* [[Named pipe]] persistent pipes used for interprocess communication
* [[XML pipeline]] for processing of XML files

==External links==
*[http://www.linfo.org/pipe.html Pipes: A Brief Introduction] by The Linux Information Project (LINFO)
*[http://www.softpanorama.org/Scripting/pipes.shtml] Unix Pipes -- powerful and elegant programming paradigm (Softpanorama)
*[http://en.wikibooks.org/w/index.php?title=Ad_Hoc_Data_Analysis_From_The_Unix_Command_Line ''Ad Hoc Data Analysis From The Unix Command Line'' at Wikibooks] shows how to use pipelines composed of simple filters to do complex data analysis.
*[http://www.pixelbeat.org/programming/stdio_buffering/ stdio buffering]
*[http://www.debian-administration.org/articles/145 Use And Abuse Of Pipes With Audio Data] gives an introduction to using and abusing pipes with netcat, nettee and fifos to play audio across a network.
*[http://www.codecrazy.net/hitting-pipe-sunday.html hitting the pipe] A program that forks two processes that communicate with each other using pipes.

==References==
* [[Sal Soghoian]] on [[MacBreak]] Episode 5 "Enter the Automatrix"

[[Category:Inter-process communication]]
[[Category:Unix]]

[[it:Pipeline (Unix)]]
[[ja:パイプ (コンピュータ)]]
[[pt:Pipeline (Unix)]]
[[uk:Конвеєр (Юнікс)]]
[[zh:Pipe]]

Yes (Unix)

2006-05-10T13:15:14Z

Unixguy: /* Description */ clarify, single CPU systems only

{{lowercase|title=yes}}

'''<tt>yes</tt>''' is a [[Unix]] command.

==Purpose==

Outputs an affirmative response or a string of text continuously until killed.

==Syntax==

yes [ STRING ]
yes OPTION

===Options===

--help

Displays help and exits

--version

Outputs version information and exits

==Description==

By itself, the <tt>yes</tt> command outputs 'y' or whatever is specified as the STRING argument (above), followed by a [[newline]], until stopped by the user or otherwise [[kill]]ed; when piped into a command, it will continue until the pipe breaks (i.e., the program completes its execution).

It can also be used to test how well a system handles high loads, as using <tt>yes</tt> results in 100% processor usage, for systems with a single processor. This, for example, can be useful for investigating whether a system's cooling system will be effective when the processor is running at 100%

==Uses==

<tt>yes</tt> can be used to send an affirmative response to any command that would otherwise request one, and thereby causing the command to run non-interactively.

(Note, this may be deprecated - the vast majority of commands that would request response from the user have either a 'force' option (e.g., '[[rm]] -f') or an 'assume-yes' option (e.g., '[[apt-get]] -y' in [[Debian]]).)

==Example==

To display the word first to the screen, type:

yes first

This statement displays the word, once per line, until killed.

[[Category:Unix software]]

Sleep (Kommandozeilenbefehl)

2006-04-24T12:17:43Z

Unixguy: rm {{stub}} looks pretty complete to me

{{lowercase|title=sleep}}

'''sleep''' is an instruction for a [[computer]] that delays execution for a specified period of time.

The sleep instruction will pause for a number of seconds (the default), minutes, hours or days.

==Usage==
sleep '''number'''[suffix]...
or:
sleep option

Where '''number''' is a required floating point number, and suffix is an optional suffix to indicate the time period.

===Suffix===
'''s''' (seconds)
'''m''' (minutes)
'''h''' (hours)
'''d''' (days)

===Options===
'''--help''' display this help and exit
'''--version''' output version information and exit

==Examples==
sleep 5
Causes the current terminal session to wait 5 seconds. The default unit is seconds.

sleep 5h
Causes the current terminal session to wait 5 hours

Note that '''<tt>sleep 5h30m</tt>''' and '''<tt>sleep 5h 30m</tt>''' are illegal since sleep takes only one value and unit as argument.

Possible uses for <tt>sleep</tt> include scheduling tasks and delaying execution to allow a process to start.

[[Category:Unix software]]

Sleep (Kommandozeilenbefehl)

2006-04-23T17:56:39Z

Unixguy: correct lowercase template to be 'sleep', not 'file'

Sleep (Kommandozeilenbefehl)

2006-04-23T17:52:51Z

Unixguy: add {{lowercase|title=file}}

{{lowercase|title=file}}

'''sleep''' is an instruction for a [[computer]] that delays execution for a specified period of time.

The sleep instruction will pause for a number of seconds (the default), minutes, hours or days.

==Usage==
sleep '''number'''[suffix]...
or:
sleep option

Where '''number''' is a required floating point number, and suffix is an optional suffix to indicate the time period.

===Suffix===
'''s''' (seconds)
'''m''' (minutes)
'''h''' (hours)
'''d''' (days)

===Options===
'''--help''' display this help and exit
'''--version''' output version information and exit

==Examples==
sleep 5
Causes the current terminal session to wait 5 seconds. The default unit is seconds.

sleep 5h
Causes the current terminal session to wait 5 hours

Note that '''<tt>sleep 5h30m</tt>''' and '''<tt>sleep 5h 30m</tt>''' are illegal since sleep takes only one value and unit as argument.

Possible uses for <tt>sleep</tt> include scheduling tasks and delaying execution to allow a process to start.

{{Compu-lang-stub}}

[[Category:Unix software]]

Sleep (Kommandozeilenbefehl)

2006-04-21T11:44:19Z

Unixguy: article overhaul, complete format edit, remove {{move to wikisource}} template, add Category:Unix software

:''For [[The Beatles]] song, see [[Wait (song)]]''

'''wait''' is an instruction for a [[computer]] that delays execution for a specified period of time.

The wait instruction will pause for a number of seconds (the default), minutes, hours or days.

==Usage==
sleep '''number'''[suffix]...
or:
sleep option

Where '''number''' is a required floating point number, and suffix is an optional suffix to indicate the time period.

===Suffix===
'''s''' (seconds)
'''m''' (minutes)
'''h''' (hours)
'''d''' (days)

===Options===
'''--help''' display this help and exit
'''--version''' output version information and exit

==Examples==
sleep 5
Causes the current terminal session to wait 5 seconds. The default unit is seconds.

sleep 5h
Causes the current terminal session to wait 5 hours

Note that '''<tt>sleep 5h30m</tt>''' and '''<tt>sleep 5h 30m</tt>''' are illegal since sleep takes only one value and unit as argument.

Possible uses for <tt>wait</tt> include scheduling tasks and delaying execution to allow a process to start.

{{Compu-lang-stub}}

[[Category:Unix software]]

Yes (Unix)

2006-04-05T11:57:00Z

Unixguy: add <tt> tags to command name "yes"

{{lowercase|title=yes}}

'''<tt>yes</tt>''' is a [[Unix]] command.

==Purpose==

Outputs an affirmative response or a string of text continuously until killed.

==Syntax==

yes [ STRING ]
yes OPTION

===Options===

--help

Displays help and exits

--version

Outputs version information and exits

==Description==

By itself, the <tt>yes</tt> command outputs 'y' or whatever is specified as the STRING argument (above), followed by a [[newline]], until stopped by the user or otherwise [[kill]]ed; when piped into a command, it will continue until the pipe breaks (i.e., the program completes its execution).

==Uses==

<tt>yes</tt> can be used to send an affirmative response to any command that would otherwise request one, and thereby causing the command to run non-interactively.

(Note, this may be deprecated - the vast majority of commands that would request response from the user have either a 'force' option (e.g., '[[rm]] -f') or an 'assume-yes' option (e.g., '[[apt-get] -y' in [[Debian]]).)

==Example==

To display the word first to the screen, type:

yes first

This statement displays the word, once per line, until killed.

[[Category:Unix software]]

Yes (Unix)

2006-04-05T11:52:57Z

Unixguy: /* File */ rm section

{{lowercase|title=yes}}

'''yes''' is a [[Unix]] command.

==Purpose==

Outputs an affirmative response or a string of text continuously until killed.

==Syntax==

yes [ STRING ]
yes OPTION

===Options===

--help

Displays help and exits

--version

Outputs version information and exits

==Description==

By itself, the yes command outputs 'y' or whatever is specified as the STRING argument (above), followed by a [[newline]], until stopped by the user or otherwise [[kill]]ed; when piped into a command, it will continue until the pipe breaks (i.e., the program completes its execution).

==Uses==

yes can be used to send an affirmative response to any command that would otherwise request one, and thereby causing the command to run non-interactively.

(Note, this may be deprecated - the vast majority of commands that would request response from the user have either a 'force' option (e.g., '[[rm]] -f') or an 'assume-yes' option (e.g., '[[apt-get] -y' in [[Debian]]).)

==Example==

To display the word first to the screen, type:

yes first

This statement displays the word, once per line, until killed.

[[Category:Unix software]]