From 7d8e66b72272651c2428f475e16a5c81963362aa Mon Sep 17 00:00:00 2001 From: mo8it Date: Tue, 15 Aug 2023 00:27:25 +0200 Subject: [PATCH] Gone through glue --- src/day_2/glue.md | 171 ++++++++++++++++++++++++++++++++++------------ 1 file changed, 127 insertions(+), 44 deletions(-) diff --git a/src/day_2/glue.md b/src/day_2/glue.md index 65d52b0..211ae2d 100644 --- a/src/day_2/glue.md +++ b/src/day_2/glue.md @@ -1,14 +1,12 @@ # Shell glue -When you run something in the terminal, then you are interacting with the so called _shell_. - -The default shell on almost all Linux systems is `bash`. (We will learn about the `fish` shell later 🐠) - -The shell has the power to glue commands together to make the impossible possible! Let's use some gluing magic! +The shell has the power to glue commands together to make the impossible possible! +Let's use some gluing magic! ## Piping -We will start with pipes. In the last task of the last day, you did see the usage of the symbol `|` between two separate commands. +We will start with pipes. +In the last task of the last day, you did see the usage of the symbol `|` between two separate commands. By entering `cowsay "Hello" | lolcat`, the output of the first command `cowsay` is sent to the second command `lolcat`. @@ -16,110 +14,195 @@ By entering `cowsay "Hello" | lolcat`, the output of the first command `cowsay` Many Linux commands support handling input of another command. -You might have seen in the manual of `wc` in day 1 that the file as an argument is only optional. How could you use `wc` without arguments? +You might have seen in the manual of `wc` in day 1 that the file as an argument is only optional. +How could you use `wc` without arguments? You might have guessed it now, make some `wc` pipes. +OK, I admit that the naming is not the best πŸšΎπŸ˜‚ -OK, I admit that the naming is not the best πŸ˜‚ +Let's get some data to work with.c +To do so, we will use the command `curl` which graps content from the internet. -Let's get some data to work with. To do so, we will use the command `curl` which graps content from the internet. - -Let's count the number of lines of the html file of the homepage of this book: +Let's count the number of lines of the HTML file of the homepage of this book: ```console -$ curl -s https://how-to-linux.mo8it.com | wc -l -220 +$ curl -s https://dev-tools.mo8it.com | wc -l +254 ``` The option `-s` tells `curl` to be silent and not show progress information. -You can see that `wc` did count the number of lines. We did just combine two completely different tools with some pipes glue! +You can see that `wc` did count the number of lines. +We did just combine two completely different tools using a pipe! How about counting the number of lines that contain the word "Linux" on the homepage? - To do so, we will add a new pipe inbetween! -`grep` is a command that searches for matches of a specified pattern. Each line with a match is printed in a new line. +`grep` is a command that searches for matches of a specified pattern. +Each line with a match is printed in a new line. -To demonstrate `grep`, here is an usage example: +To demonstrate `grep`, here is one usage example: ``` $ curl --help | grep "silent" - -f, --fail Fail silently (no output at all) on HTTP errors -s, --silent Silent mode ``` -We did just filter the output of the help of a command. This way, you can also search quickly for command options! +We did just filter the output of the help message of a command. +This is one way to search quickly for command options! Back to the main example: ```console -$ curl -s https://how-to-linux.mo8it.com | grep "Linux" | wc -l -7 +$ curl -s https://dev-tools.mo8it.com | grep "Linux" | wc -l +6 ``` -You can see that you can use multiple pipes. This allows for almost infinite combinations! +You can see that you can use multiple pipes. +This allows for almost infinite combinations! -Being able to combine commands is the reason why many commands are simple. They do one thing and do it well! To do more, combine them! +Being able to combine commands is the reason why many commands are simple. +They do one thing and do it well! +To do more, combine them! -This is much more flexible and powerful than a program that tries to do a lot of things. +This is much more flexible and powerful than a program that tries to do many things at once. ## Input, output Before going any further, we need to understand an important concept in Linux. -A command accepts input and generates two types of output. The input is called _standard input_. The output is split to _standard output_ and _standard error_. +A command accepts input and generates two types of output. +The input is called _standard input_ (**stdin**). +The output is split to _standard output_ (**stdout**) and _standard error_ (**stderr**). +They actually have numbers that will be relevant later: -The standard output has the number 1 while the standard error has the number 2. +- 0: stdin +- 1: stdout +- 2: stderr -Normal output is sent to the standard output. Errors (and sometimes output that is not very important) are sent to the standard error. - -You can redirect the standard output or the standard error to a file! +Normal output is sent to the standard output. +Errors (and sometimes output that is not important) are sent to the standard error. ## Redirections -Let's see how you can redirect the output of commands to a file. +You can redirect the standard output or the standard error of a command to a file! -If you just run `curl -s https://how-to-linux.mo8it.com`, you will see the html file printed to the terminal. Let's redirect the output to a html file on your disk: +If you just run `curl -s https://dev-tools.mo8it.com`, you will see the HTML file printed to the terminal. +Let's redirect the output to an HTML file on the disk: ```console -$ curl -s https://how-to-linux.mo8it.com > how-to-linux.html +$ curl -s https://dev-tools.mo8it.com >dev-tools.html ``` -Now view the content of the new file `how-to-linux.html`. You will see the same output from the terminal without redirection. +Now, view the content of the new file `dev-tools.html`. +You will see the same output from the terminal without redirection. -Now try this command: +Now, try this command: ```console -$ curl https://non-existent-site.mo8it.com > test.html +$ curl https://non-existent-site.mo8it.com >test.html +(…) curl: (60) SSL certificate problem: self-signed certificate More details here: https://curl.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. + $ cat test.html ``` You will see that the file is empty since `curl` did not find a page to show as normal output. +The error message was displayed using `stderr`. If you are using this command in a script, then it might be wise to redirect the error to a log file: ```console -$ curl https://non-existent-site.mo8it.com 2> curl.log +$ curl https://non-existent-site.mo8it.com 2>curl.log + $ cat curl.log curl: (60) SSL certificate problem: self-signed certificate -More details here: https://curl.se/docs/sslcerts.html - -curl failed to verify the legitimacy of the server and therefore could not -establish a secure connection to it. To learn more about this situation and -how to fix it, please visit the web page mentioned above. +(…) ``` -You can see that the error is now not shown after running the `curl` command. It was redirected to the file `curl.log`. +You can see that the error is not shown anymore after running the `curl` command. +It was redirected to the file `curl.log`. Did you notice the number 2 before the redirection symbol `2>`? -The last section did mention that the number of the standard error is 2. Therefore, 2 has to be specified to redirect the errors. +The last section did mention that the number of the standard error is 2. +Therefore, 2 has to be specified to redirect it. -If you don't specify a number, then it is equivalent to 1 which stands for the standard output. This means that `>` is equivalent to `1>`. +If you don't specify a number, then it is equivalent to 1 which stands for the standard output. +This means that `>` is equivalent to `1>`. + +What if a command produces output using stdout _and_ stderr? +Take a look at the following example: + +```console +$ touch some_file.txt + +$ # Produces output to stdout and stderr. +$ ls some_file.txt does_not_exist.txt +ls: cannot access 'does_not_exist.txt': No such file or directory +some_file.txt + +$ # Redirect only stdout to a file. stderr is shown. +$ ls some_file.txt does_not_exist.txt >stdout.txt +ls: cannot access 'does_not_exist.txt': No such file or directory + +$ cat stdout.txt +some_file.txt + +$ # Redirect only stderr to a file. stdout is shown. +$ ls some_file.txt does_not_exist.txt 2>stderr.txt +some_file.txt + +$ cat stderr.txt +ls: cannot access 'does_not_exist.txt': No such file or directory + +$ # Redirect both stdout and stderr to different files. +$ ls some_file.txt does_not_exist.txt >stdout.txt 2>stderr.txt + +$ cat stdout.txt +some_file.txt + +$ cat stderr.txt +ls: cannot access 'does_not_exist.txt': No such file or directory + +$ # Redirect stdout and stderr to the same file with `&>`. +$ ls some_file.txt does_not_exist.txt &>mixed.txt + +$ cat mixed.txt +ls: cannot access 'does_not_exist.txt': No such file or directory +some_file.txt +``` + +### Appending + +So far, we did redirect output using the operators `>`, `1>`, `2>` and `&>`. +But these operators overwrite the files they are redirected to if they already exist. +If you want to append to a file instead, use the operators above but with double `>`, for example `&>>`. + +### Discarding + +There is a special file that you see some command redirect to: `/dev/null`. + +This file is like a black hole. +Everything redirected to that file is discarded. + +For example, if you don't care about the errors that some command throughs, then you can redirected its stderr to `/dev/null` using `2>/dev/null` + +### More details + +We did discuss the most important cases for redirections. +But there are some less important details like the following: + +- `command &>filename` is equivalent to `command >filename 2>&1`, but not to `command 2>&1 >filename` because the order matters. +- `0&1 | command2`. To only redirect stderr, you have to use `command 2>&1 >/dev/null | command2` (not `command >/dev/null 2>&1 | command2` since the order matters). + +You might have noticed how it can get complicated. +Therefore, refer to the ["Redirections" section in the Bash reference manual](https://www.gnu.org/software/bash/manual/bash.html#Redirections) for more details. + +Yes, the reference is about Bash, but Fish has the same behavior here.