diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 0d63314..209a243 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -8,6 +8,8 @@ - [Packages](day_1/packages.md) - [Tasks](day_1/tasks.md) - [Day 2](day_2/README.md) + - [Shell glue](day_2/glue.md) + - [Tasks](day_2/tasks.md) - [Day 3](day_3/README.md) - [Day 4](day_4/README.md) - [Day 5](day_5/README.md) diff --git a/src/day_2/glue.md b/src/day_2/glue.md new file mode 100644 index 0000000..b4de6c9 --- /dev/null +++ b/src/day_2/glue.md @@ -0,0 +1,166 @@ +# Shell glue + +When you run something in the terminal, then you are interacting with the so called `shell`. + +The default shell on almost all Linux systems is `bash`. (We will learn about the `fish` shell later) + +The shell has the power to glue commands together to make the impossible possible! Lets use some gluing magic! + +## Piping + +We will start with pipes. In the last task of the last day, you did see the usage of the symbol `|` between two separate commands. + +By entering `cowsay "Hello" | lolcat`, the output of the first command `cowsay` is sent to the second command `lolcat`. + +`lolcat` takes the input, colors it and outputs it again! + +Many Linux commands support handeling input of another command. + +You might have seen in the manual of `wc` in day 1 that the file as an argument is only optional. How could you use `wc` without arguments? + +You might have guessed it now, make some `wc` pipes. + +OK, I admit that the naming is not the best 😂 + +Lets get some data to work with. To do so, we will use the command `curl` which graps content from the internet. + +Lets count the number of lines of the html file of the homepage of this book: + +```console +$ curl -s https://how-to-linux.mo8it.xyz | wc -l +201 +``` + +The option `-s` tells `curl` to be silent and not show progress information. + +You can see that `wc` did count the number of lines. We did just combine to completely different tools with some pipes glue! + +How about counting the number times the word "Linux" was mentioned on the homepage? + +To do so, we will add a new pipe inbetween! + +`grep` is a command that searches for matches of a specified pattern. Each match is printed in a new line. + +To demonstrate `grep`, here is an usage example: + +``` +$ curl --help | grep "silent" + -f, --fail Fail silently (no output at all) on HTTP errors + -s, --silent Silent mode +``` + +We did just filter the output of the help of a command. This way, you can also search quickly for command options! + +Back to the main example: + +```console +$ curl -s https://how-to-linux.mo8it.xyz | grep "Linux" | wc -l +6 +``` + +You can see that you can use multiple pipes. This allows for almost infinite combinations! + +Being able to combine commands is the reason why many commands are simple. They do one thing and do it well! To do more, combine them! + +This is much more flexibel and powerful that a program that tries to do a lot of things. + +## Input, output + +Before going any further, we need to understand an important concept in Linux. + +A command accepts input and generates two types of output. The input is called _standard input_. The output is split to _standard output_ and _standard error_. + +The standard output has the number 1 while the standard error has the number 2. + +Normal output is sent to the standard output. Errors (and sometimes output that is not very important) are sent to the standard error. + +You can redirect the standard output or the standard error to a _stream_. Normally you will redirect the output to a file. Other stream forms are not relevant for now. + +## Redirections + +Lets see how you can redirect the output of commands to a file. + +If you just run `curl -s https://how-to-linux.mo8it.xyz`, you will the html file printed in the terminal. Lets redirect the output to a html file on your disk: + +```console +$ curl -s https://how-to-linux.mo8it.xyz > how-to-linux.html +``` + +Now view the content of the new file how-to-linux.html. You will be able to see the same output from the terminal without redirection. + +Now try this command: + +```console +$ curl https://not-existent-site.mo8it.xyz > test.html +curl: (60) SSL certificate problem: self-signed certificate +More details here: https://curl.se/docs/sslcerts.html + +curl failed to verify the legitimacy of the server and therefore could not +establish a secure connection to it. To learn more about this situation and +how to fix it, please visit the web page mentioned above. +$ cat test.html +``` + +You will see that the file is empty since `curl` did not find a page to show as normal output. + +If you are using this command in a script, then it might be wise to redirect the error to a log file: + +```console +$ curl https://not-existent-site.mo8it.xyz 2> curl.log +$ cat curl.log +curl: (60) SSL certificate problem: self-signed certificate +More details here: https://curl.se/docs/sslcerts.html + +curl failed to verify the legitimacy of the server and therefore could not +establish a secure connection to it. To learn more about this situation and +how to fix it, please visit the web page mentioned above. +``` + +You can see that the error is now not shown after running the `curl` command. It was redirected to the file `curl.log`. + +Did you notice the number 2 before the redirection symbol `2>`? + +The last section did mention that the number of the standard error is 2. Therefore, 2 has to be specified to redirect the errors. + +If you don't specify a number, then it is equivalent to 1 which stands for the standard output. This means that `>` is equivalent to `1>`. + +## Bash scripts + +Lets write our first Bash script (also for some more demonstration of the two outputs): + +```bash +#!/usr/bin/bash + +echo "What is your favorite operating system after reading this book?" +echo "1. Linux" +echo "2. Windows" +echo "3. Mac" + +echo -n "Enter a number: " +read ANSWER + +if [ "$ANSWER" == "1" ] +then + echo "Good choice!" +else + echo "Nah, that can't be right! It must be an error!" > /dev/stderr +fi +``` + +Copy this code into a file called `which-os.sh`. + +Now run `chmod +x which-os.sh`. Then run `./which-os.sh`. + +--- + +I am really sorry, but I can not continue this book during the course. It was not only meant for the couse but also like a personal project that I wanted to keep on the internet even after the course. + +But I did underestimate the huge amount of time it takes to write such a book/script. + +Continuing the book during the course would mean that I can not cover a lot of topics because I can not write enough. It would also harm my health. I need sleep :') + +I will continue it after the course, not only for the participants as output, but also for the purpose of a personal project. So please don't be disappointed. You will find all contents on this website some time after the course. I hope that it would then help you as a resource for looking things up. + +I do especially apologize to the online participants. I did give my best. + +Again, it is not lost. You will get it, but after the course :) diff --git a/src/day_2/tasks.md b/src/day_2/tasks.md new file mode 100644 index 0000000..732c4c9 --- /dev/null +++ b/src/day_2/tasks.md @@ -0,0 +1,67 @@ +# Tasks + +Organize the files and directories of your tasks in separate directories! + +## Task: Job scheduler + +> Warning ⚠️ : This task is not an easy task. Don't give up quickly and ask for help if you don't get further! + +In this task, we want to write our own job scheduler. + +Understanding how job schedulers work is important when you are working on a computer cluster. + +Computer clusters are shared by many users. Therefore, running jobs on a cluster has to be scheduled to make sure that the resources are shared probebly. + +In this task, we will keep it simple. No aspects of multiple users or any optimizations. + +We want to be able to submit a job as a single script (without any dependencies). The submitted scripts should run one after the another to save CPU usage for example. + +We will use the program `inotifywait`. This program can monitor a directory and notify on changes within this directory. + +1. Find out which package installs `inotifywait` and install it. +1. Read the manual of `inotifywait` for a better understanding of what it does. +1. Find out how to tell `inotifywait` to keep monitoring a directory and not exit after the first event. +1. Find out what events mean in the context of `inotifywait`. +1. Create a new directory called `jobs` to be monitored. +1. Create a new directory called `logs` that will be used later. +1. Run `inotifywait` while telling it to monitor the directory `jobs`. Leave the command running in a terminal and open a second terminal (tab) to continue the work in. +1. Create a file **outside** of the directory `jobs` and then copy it to the directory `jobs`. +1. Go back to the first terminal and see the output of `inotifywait` was. +1. Based on the output, choose an event that you want to listen to with `inotifywait` that tells you when a file is _completely_ added to the directory `jobs`. Use the manual to read more about specific events. +1. Find an option that lets you tell `inotifywait` to only notify when the choosen event occurs. +1. Find an option that lets you format the output of the notification of `inotifywait`. Since we only listen on one event and monitor only one directory, an output that shows only the name of the new file should be enough. +1. Enter the command that you have until now in a script. Now extend it by using a `while` loop that continously listens on the notifications of `inotifywait`. +1. After a notification, the body of the `while` loop should first print the name of the script that was added. From now on, we only want to add scripts to the `jobs` directory. +1. After printing the script name, run the script! +1. Save the standard output and standard error of the script into two separate files in the `logs` directory. If the name of the script is `job.sh` for example, then the output should be in the files `logs/job.sh.out` and `logs/job.sh.err`. + +Tipps: + +- Take a look at the examples from the sections of this day. +- Take care of permissions. + +If you have extra time, read about the command `screen` in the internet. `screen` allows you to run commands in the background. This way, you don't need two terminals. + +## Task: Job submitter + +In this task we will write a small script that lets us submit a job script to the scheduler from the last task. + +The script should take the path to the job script as a single required argument. + +The script should then copy the job script to the directory `jobs` while adding the time and date to the beginning of the name of the job script in the `jobs` directory. + +Read the manual of the command `date` to know how to get the time and date in the following format: `2022-08-22T20:00:00+00:00`. + +If the name of the job script is `job.sh` for example, the job script should be named `2022-08-22T20:00:00+00:00_job.sh` in the `jobs` directory. + +Use variables to write the script to make it more understandable. + +## Task: Submit a job + +Write a small scripts of your choice that require a long time to run and submit them using the script from the last task. Make sure that the scheduler is running in the background. + +You can use the command `sleep` to simulate a job that needs long time to run. + +Submit your job script multiple times and take a look at the terminal that is running the scheduler to make sure that the job scripts are running one after the other. + +Verify the redirection of the standard output and standard error in the directory `logs`.