mo8it/How_To_Linux
1
0
Fork 0
mirror of https://codeberg.org/Mo8it/How_To_Linux.git synced 2024-11-24 02:01:34 +00:00
Code Releases Activity

Gone through glue

Browse source
This commit is contained in:
Mo 2023-08-15 00:27:25 +02:00
parent 5e8f175bae
commit 7d8e66b722
1 changed files with 127 additions and 44 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

171
src/day_2/glue.md
View file

@ -1,14 +1,12 @@
# Shell glue # Shell glue
When you run something in the terminal, then you are interacting with the so called _shell_. The shell has the power to glue commands together to make the impossible possible!
Let's use some gluing magic!
The default shell on almost all Linux systems is `bash`. (We will learn about the `fish` shell later 🐠)
The shell has the power to glue commands together to make the impossible possible! Let's use some gluing magic!
## Piping ## Piping
We will start with pipes. In the last task of the last day, you did see the usage of the symbol `|` between two separate commands. We will start with pipes.
In the last task of the last day, you did see the usage of the symbol `|` between two separate commands.
By entering `cowsay "Hello" | lolcat`, the output of the first command `cowsay` is sent to the second command `lolcat`. By entering `cowsay "Hello" | lolcat`, the output of the first command `cowsay` is sent to the second command `lolcat`.
@ -16,110 +14,195 @@ By entering `cowsay "Hello" | lolcat`, the output of the first command `cowsay`
Many Linux commands support handling input of another command. Many Linux commands support handling input of another command.
You might have seen in the manual of `wc` in day 1 that the file as an argument is only optional. How could you use `wc` without arguments? You might have seen in the manual of `wc` in day 1 that the file as an argument is only optional.
How could you use `wc` without arguments?
You might have guessed it now, make some `wc` pipes. You might have guessed it now, make some `wc` pipes.
OK, I admit that the naming is not the best 🚾😂
OK, I admit that the naming is not the best 😂 Let's get some data to work with.c
To do so, we will use the command `curl` which graps content from the internet.
Let's get some data to work with. To do so, we will use the command `curl` which graps content from the internet. Let's count the number of lines of the HTML file of the homepage of this book:
Let's count the number of lines of the html file of the homepage of this book:
```console ```console
$ curl -s https://how-to-linux.mo8it.com | wc -l $ curl -s https://dev-tools.mo8it.com | wc -l
220 254
``` ```
The option `-s` tells `curl` to be silent and not show progress information. The option `-s` tells `curl` to be silent and not show progress information.
You can see that `wc` did count the number of lines. We did just combine two completely different tools with some pipes glue! You can see that `wc` did count the number of lines.
We did just combine two completely different tools using a pipe!
How about counting the number of lines that contain the word "Linux" on the homepage? How about counting the number of lines that contain the word "Linux" on the homepage?
To do so, we will add a new pipe inbetween! To do so, we will add a new pipe inbetween!
`grep` is a command that searches for matches of a specified pattern. Each line with a match is printed in a new line. `grep` is a command that searches for matches of a specified pattern.
Each line with a match is printed in a new line.
To demonstrate `grep`, here is an usage example: To demonstrate `grep`, here is one usage example:
``` ```
$ curl --help | grep "silent" $ curl --help | grep "silent"
-f, --fail Fail silently (no output at all) on HTTP errors
-s, --silent Silent mode -s, --silent Silent mode
``` ```
We did just filter the output of the help of a command. This way, you can also search quickly for command options! We did just filter the output of the help message of a command.
This is one way to search quickly for command options!
Back to the main example: Back to the main example:
```console ```console
$ curl -s https://how-to-linux.mo8it.com | grep "Linux" | wc -l $ curl -s https://dev-tools.mo8it.com | grep "Linux" | wc -l
7 6
``` ```
You can see that you can use multiple pipes. This allows for almost infinite combinations! You can see that you can use multiple pipes.
This allows for almost infinite combinations!
Being able to combine commands is the reason why many commands are simple. They do one thing and do it well! To do more, combine them! Being able to combine commands is the reason why many commands are simple.
They do one thing and do it well!
To do more, combine them!
This is much more flexible and powerful than a program that tries to do a lot of things. This is much more flexible and powerful than a program that tries to do many things at once.
## Input, output ## Input, output
Before going any further, we need to understand an important concept in Linux. Before going any further, we need to understand an important concept in Linux.
A command accepts input and generates two types of output. The input is called _standard input_. The output is split to _standard output_ and _standard error_. A command accepts input and generates two types of output.
The input is called _standard input_ (**stdin**).
The output is split to _standard output_ (**stdout**) and _standard error_ (**stderr**).
They actually have numbers that will be relevant later:
The standard output has the number 1 while the standard error has the number 2. - 0: stdin
- 1: stdout
- 2: stderr
Normal output is sent to the standard output. Errors (and sometimes output that is not very important) are sent to the standard error. Normal output is sent to the standard output.
Errors (and sometimes output that is not important) are sent to the standard error.
You can redirect the standard output or the standard error to a file!
## Redirections ## Redirections
Let's see how you can redirect the output of commands to a file. You can redirect the standard output or the standard error of a command to a file!
If you just run `curl -s https://how-to-linux.mo8it.com`, you will see the html file printed to the terminal. Let's redirect the output to a html file on your disk: If you just run `curl -s https://dev-tools.mo8it.com`, you will see the HTML file printed to the terminal.
Let's redirect the output to an HTML file on the disk:
```console ```console
$ curl -s https://how-to-linux.mo8it.com > how-to-linux.html $ curl -s https://dev-tools.mo8it.com >dev-tools.html
``` ```
Now view the content of the new file `how-to-linux.html`. You will see the same output from the terminal without redirection. Now, view the content of the new file `dev-tools.html`.
You will see the same output from the terminal without redirection.
Now try this command: Now, try this command:
```console ```console
$ curl https://non-existent-site.mo8it.com > test.html $ curl https://non-existent-site.mo8it.com >test.html
(…)
curl: (60) SSL certificate problem: self-signed certificate curl: (60) SSL certificate problem: self-signed certificate
More details here: https://curl.se/docs/sslcerts.html More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above. how to fix it, please visit the web page mentioned above.
$ cat test.html $ cat test.html
``` ```
You will see that the file is empty since `curl` did not find a page to show as normal output. You will see that the file is empty since `curl` did not find a page to show as normal output.
The error message was displayed using `stderr`.
If you are using this command in a script, then it might be wise to redirect the error to a log file: If you are using this command in a script, then it might be wise to redirect the error to a log file:
```console ```console
$ curl https://non-existent-site.mo8it.com 2> curl.log $ curl https://non-existent-site.mo8it.com 2>curl.log
$ cat curl.log $ cat curl.log
curl: (60) SSL certificate problem: self-signed certificate curl: (60) SSL certificate problem: self-signed certificate
More details here: https://curl.se/docs/sslcerts.html (…)
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
``` ```
You can see that the error is now not shown after running the `curl` command. It was redirected to the file `curl.log`. You can see that the error is not shown anymore after running the `curl` command.
It was redirected to the file `curl.log`.
Did you notice the number 2 before the redirection symbol `2>`? Did you notice the number 2 before the redirection symbol `2>`?
The last section did mention that the number of the standard error is 2. Therefore, 2 has to be specified to redirect the errors. The last section did mention that the number of the standard error is 2.
Therefore, 2 has to be specified to redirect it.
If you don't specify a number, then it is equivalent to 1 which stands for the standard output. This means that `>` is equivalent to `1>`. If you don't specify a number, then it is equivalent to 1 which stands for the standard output.
This means that `>` is equivalent to `1>`.
What if a command produces output using stdout _and_ stderr?
Take a look at the following example:
```console
$ touch some_file.txt
$ # Produces output to stdout and stderr.
$ ls some_file.txt does_not_exist.txt
ls: cannot access 'does_not_exist.txt': No such file or directory
some_file.txt
$ # Redirect only stdout to a file. stderr is shown.
$ ls some_file.txt does_not_exist.txt >stdout.txt
ls: cannot access 'does_not_exist.txt': No such file or directory
$ cat stdout.txt
some_file.txt
$ # Redirect only stderr to a file. stdout is shown.
$ ls some_file.txt does_not_exist.txt 2>stderr.txt
some_file.txt
$ cat stderr.txt
ls: cannot access 'does_not_exist.txt': No such file or directory
$ # Redirect both stdout and stderr to different files.
$ ls some_file.txt does_not_exist.txt >stdout.txt 2>stderr.txt
$ cat stdout.txt
some_file.txt
$ cat stderr.txt
ls: cannot access 'does_not_exist.txt': No such file or directory
$ # Redirect stdout and stderr to the same file with `&>`.
$ ls some_file.txt does_not_exist.txt &>mixed.txt
$ cat mixed.txt
ls: cannot access 'does_not_exist.txt': No such file or directory
some_file.txt
```
### Appending
So far, we did redirect output using the operators `>`, `1>`, `2>` and `&>`.
But these operators overwrite the files they are redirected to if they already exist.
If you want to append to a file instead, use the operators above but with double `>`, for example `&>>`.
### Discarding
There is a special file that you see some command redirect to: `/dev/null`.
This file is like a black hole.
Everything redirected to that file is discarded.
For example, if you don't care about the errors that some command throughs, then you can redirected its stderr to `/dev/null` using `2>/dev/null`
### More details
We did discuss the most important cases for redirections.
But there are some less important details like the following:
- `command &>filename` is equivalent to `command >filename 2>&1`, but not to `command 2>&1 >filename` because the order matters.
- `0<filename command` or `command 0<filename` can be used to redirect a file to the stdin of a command.
- A pipe `command1 | command2` redirects only the stdout of `command1` to the stdin of `command2`. But if you want to redirect stderr too, then you have to use `command1 2>&1 | command2`. To only redirect stderr, you have to use `command 2>&1 >/dev/null | command2` (not `command >/dev/null 2>&1 | command2` since the order matters).
You might have noticed how it can get complicated.
Therefore, refer to the ["Redirections" section in the Bash reference manual](https://www.gnu.org/software/bash/manual/bash.html#Redirections) for more details.
Yes, the reference is about Bash, but Fish has the same behavior here.