What is Shell Scripting?
Shell scripting is a powerful tool often used across organizations to automate processes, test ideas, and increase productivity. It is a text file with a list of commands that gives instructions to the operating system to perform certain tasks. These commands from the shell script are interpreted, processed, and carried out via an interface called “shell”.
Shell scripting is typically used to automate regular, repetitive system tasks like file backup, resource monitoring, and user account management. System administrators can shorten execution times, improve accuracy, and streamline challenging processes by putting a series of instructions into a script.
Uses of Shell Scripting
Shell scripting is used to automate repetitive tasks like account creation for various users, monitoring system status, memory usage, and running processes on a system. System administrators can interact directly with the operating system and make changes that will take effect after a system reboot by using shell scripting.
Shell scripts can also be used for data archiving, restoring the system to a specific date, and monitoring the system status while performing backup operations.
Programmers can use shell scripts to create programs, system-level tasks, pattern-finding in files, processing user input-based data in files, etc. Shell scripts can also be used to execute other activities.
Several daily jobs, repetitive tasks, etc., can be automated using shell scripting. The built-in routines for for loops, while loops, and other shell scripts can be used if we wish to run the same command more than once.
System-level scripts that execute during system boot-up are created using shell scripts. By automating application starting scripts, we may create scripts to automate an application's running. We can also create scripts to automate an application's installation of dependency packages.
Shell Scripting Benefits for DevOps Engineers
DevOps Engineers can automate operations like software installation, system configuration, network connection setup, and file and directory management by using shell.
Let's take the case where a DevOps engineer wants to automate the server deployment of an application. They can automate every step of the process using shell scripting, including downloading the application code, installing any required dependencies, configuring the server, and launching the application. The deployment process can then be completed faster and with less effort by having this script run automatically or by issuing a short command.
Basic Example of a Shell Script
In the following example, we will create and execute a simple shell script that basically creates a folder with two files in it.
1. vim is a text editor that lets us create and edit a file. In this example, we have created a file called “demo-shell-script.sh” to write a simple executable shell script. (Note that .sh is the extension of a bash script).
2. Here is a simple shell script
“#!” is an operator called SHEBANG which directs the script to the interpreter location. A line starting with #! is used to tell the operating system which interpreter to use to execute the file.
There are many interpreters like ksh, csh, ssh, bash (Bourne) shell etc. This particular first line asks the system to use the bash interpreter.
# is used to write comments about the script or the instructions. It primarily provides better readability of the script.
mkdir is the command used to create a directory (nothing but a folder)
cd is used to change the directory.
touch command is used to create files.
:wq tells the editor to save/write and exit/quit the editor.
3. cat command is used to show the contents of a file. As we can see here, cat demo-shell-script.sh displays the content we added in vim editor.
4. chmod is used to change the permissions of the user, group, or owner. In this example, for simplicity sake, we have given chmod 777 which basically gives read, write, and execute permissions to everyone (which is not a good practice in real-time). After changing the permissions, to test whether the file is actually created, we use the “ls” command to see what all files are created.
5. To execute the shell script we have written, use ./demo-shell-script.sh. To test if everything is executed correctly, use the ls command. Change the directory using cd and see if the two files are created as well.
6. ls -ltr command is used to list the files with a timestamp.
DevOps Specific Shell Script Commands
1. Script for finding out Node Health:
It’s always best practice to include details such as the author, date of creation, description of usage of the script, version, etc.,
set -x command is used to show the command, along with the output of that particular command in the script. But, in some cases where we don’t want the users to see the commands, we can comment out the set -x command. Alternatively, we can also use “echo” command to print the information about the output of a certain command
df -h is used to display the disk space
free -g is used to display the free memory on the disk
nproc is used to know the number of CPUs
Changed permissions for the shell script as discussed earlier using chmod 777 and executed the script using ./Node_Health.sh
We can see that the script gives the output as shown below.
2. Finding out Processes and Process IDs:
ps -ef is the command used to find out all the processes that are running or stopped in full format, i.e., with process ID as well.
Now, in order to find out details about a particular process, we use “grep” command along with ps -ef and the pipe parameter “|”
ps -ef | grep “amazon”
Now, to test the power of executing a shell script along with pipe parameter | and grep command, let’s create a new shell script file and print some numbers in it.
As we can see below, by executing the shell script along with the grep command and specific search parameter, in this case, number 2, we are able to output all the numbers that have 2 in them.
- The important thing to note here is that the pipe command “|” sends the output of the first command to the second one.
One other powerful command that is used to get even more specific output is the “awk” command.
ps -ef | grep amazon | awk -F" " '{print $2}'
This command that includes awk basically outputs the processes related to amazon, not the complete details but only the process ID which is in column 2.
3. Commands for Best Practices:
Edited the shell script, added process ID look-up command pipe that we implemented earlier, and we have made use of two other new commands that come under the best practices of shell scripting. They are set -e and set -o pipefail
set -e command is used to enable the "exit immediately if a command exits with a non-zero status" option.
set -o pipefail command is used to set the "pipefail" option. When this option is enabled, the exit status of a pipeline (a series of commands separated by pipes, such as command1 | command2 | command3) will be determined by the last command in the pipeline that exits with a non-zero status.
4. Some Important Use Cases in DevOps:
1. curl - The curl command is to interact with web services, download files, and perform various operations related to APIs and web resources. Here are some common use cases for the curl command in DevOps.
We can use curl to download files from the internet or within our network.
Example:
curl -O https://example.com/file.zip
We can make HTTP GET requests to retrieve data from a URL, which is useful for fetching information from APIs or web services.
Example:
curl https://api.example.com/data
We can save the response to a file or display it on the terminal.
Example:
curl -o output.json https://api.example.com/data
These are just a few examples of how the curl command can be used in a DevOps context.
2. wget - The wget command is another powerful tool commonly used in DevOps and system administration tasks. It allows us to download files from the web, retrieve resources, and automate file transfers.
We can use wget to download files from the internet.
Example:
wget https://example.com/file.zip
3. find - is a common command used to search for files and directories within a specified directory and its subdirectories. It is a versatile tool that can be helpful for tasks like locating specific files, cleaning up old or unused files, or processing a batch of files.
To find all files with the ".log" extension in the current directory and its subdirectories.
Example:
find . -name "*.log"
4. if else - is used for conditional execution of code. It allows you to make decisions based on the evaluation of a condition, and then execute different code blocks depending on whether the condition is true or false. This is incredibly useful for automating tasks, handling exceptions, and controlling the flow of your scripts or automation processes.
5. for loops - for loops are frequently used for iterating over a list of items, such as files, directories, servers, or any other elements, in DevOps, as well as in shell scripting and programming in general, to carry out a series of activities or operations.
For loops can be used for a variety of DevOps-related operations, such as server provisioning, configuration management, deployment, and more.
Example:
for i in {1..5}
do
echo "Processing iteration $i"
# Perform some operation in each iteration
done
6. trap - is used to set up and define signal handlers. In Unix-like operating systems, signals are notifications that are given to processes. They can be used for a number of things, such as handling errors, clearing out resources, and regulating the behavior of scripts and processes. It is particularly useful for managing signals within scripts and automation processes.
Graceful shutdown - When a script or process is terminated—whether by user action (such as Ctrl+C) or by some other mechanism—you can use the trap command to make sure that resources are appropriately released and cleanup activities are carried out.
Example:
trap 'cleanup_function' SIGINT SIGTERM
SIGINT (interrupt signal, often generated by pressing Ctrl+C), SIGTERM (terminate signal), and SIGHUP (hang-up signal), among others.
There are many more use cases of trap command in DevOps.