¶ Ubuntu
To be fair, when I say Linux, I actually mean Ubuntu. I have very limited experience with other distributions, apart from Debian (Proxmox), which counts as Ubuntu (or the other way around. That being said, a lot of topics in this section are not necessarily Ubuntu specific. However, they will be written from an Ubuntu/Debian perspective, so change and adapt where necessary.
¶ Installation
This is an example of an Ubuntu 18.04 install on VirtualBox. Installing other versions like 19, 20, 21 on a bare metal server or as a virtual machine on another hypervisor will be the very similar.
-
Download Ubuntu from the official website: https://ubuntu.com/download.
-
Boot the (virtual) machine from the attached .iso file (or create a bootable USB stick) and the Ubuntu installer will start. The first question to answer is what language you want.
-
Next we can select the Keyboard configuration. You can have the installer Identify your keyboard or select it yourself. I know I have the
English (US)layout and the...(US, euro on 5)variant.
-
Now select the Network connection. As I want my router to assign a fixed IP address, so I'll go into the router now and link this mac-address to a fixed IP. You can accept the defaults if you only want to setup a VM for testing.
-
In the next screen you can set your proxy settings. I have never done that, so I'll skip this and continue.
-
Ubuntu then asks you if you want to add a mirror for downloading software. I never did it, but you can if you know there is a fast one nearby. Like on the same network.
-
Next we come to the Filesytem setup screen. Here we select Use An Entire Disk.
-
Now we can choose the disk to use. We only have the virtual disk we created earlier, so not much to choose from.
Yes, I forgot to increase the disk size to at least 16 GB. That's going to hurt later.
-
The installer will show you an overview of the changes it's going to make to the volume. Which is fine for us, so let's do it.
-
Enter your Profile setup: name, server name, username and password.
-
We definitely want to use SSH to log into the server through a terminal later, so we want to select Install OpenSSH server.
-
Now we can select some Featured Server Snaps. These are pieces of software that will be installed with Ubuntu. We don't want any, so select none of them.
-
The installation will now start.
-
When the installation is done, the bottom "button" will now say Reboot. That is what we'll do.
-
The next screen will now say "Please remove the installation medium, then press ENTER:". Remove the image from the virtual drive by clicking on the cd icon in the bottom bar below the virtual screen. Now press ENTER.
-
After reboot, the VM will show the Host SSH keys. We can ignore them, but obviously copy and store them if you need to secure your VM.
Normally you should never share these keys in a screen or otherwise, but this VM will be destroyed anyway.
-
SSH into the VM. It will show a warning that the server and terminal key don't match. As I'm confident I'm connecting to my own VM, and not a hacked server, I'll press Yes to continue.
-
Ubuntu will ask for my password and after entering it successfully I'm connected.
-
Run the update commands after which the installation is finished:
sudo apt update && sudo apt upgrade
¶ Hardening a Linux system
¶ Audit the system
Install and run tools to audit the system.
¶ Lynis
More info on https://cisofy.com/lynis/.
Download the key from a central keyserver:
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 013baa07180c50a7101097ef9de922f1c2fde6c4
The software repository uses preferably HTTPS for secure transport. Install the 'https' method for APT, if it was not available yet.
sudo apt install apt-transport-https
Using your software in English? Then configure APT to skip downloading translations. This saves bandwidth and prevents additional load on the repository servers.
echo 'Acquire::Languages "none";' | sudo tee /etc/apt/apt.conf.d/99disable-translations
Next step is adding the repository:
echo "deb https://packages.cisofy.com/community/lynis/deb/ stable main" | sudo tee /etc/apt/sources.list.d/cisofy-lynis.list
Refresh the local package database with the new repository data and install Lynis:
sudo apt update
sudo apt install lynis
Confirm Lynis version
lynis show version
Audit the system and store the results in a file
sudo lynis audit system --report-file 20220111-lynis-audit-pcjoep
¶ Uchecker
More info on https://github.com/cloudlinux/kcare-uchecker.
curl -s -L https://kernelcare.com/uchecker | sudo python
Consider adding an alias to your .bashrc or create a file .bash_aliases in your home directory and add the line below.
alias uchecker="curl -s -L https://kernelcare.com/uchecker | sudo python"
¶ Nmap
Install Nmap on a system. If you want to check your router, this system should be on the outside of your network.
Ubuntu: https://nmap.org/book/inst-linux.html#inst-debian + Zenmap as GUI: https://nmap.org/book/zenmap.html
Windows: https://nmap.org/book/inst-windows.html#inst-win-exe
Ubuntu install:
sudo apt update
sudo apt install nmap zenmap
- Scan for open ports:
nmap <domain- or hostname or ip> - Scan for available services:
nmap -sV <domain- or hostname or ip> - Scan for known vulnerabilities:
nmap -v --script vuln <domain- or hostname or ip> - Scan for known DOS vulnerabilities:
nmap -v --script dos <domain- or hostname or ip>
More info:
Nmap cheat sheet: From discovery to exploits – Part 1: Introduction to Nmap
How to Detect CVEs Using Nmap Vulnerability Scan Scripts
How to scan for services and vulnerabilities with Nmap
¶ Harden the system
¶ Harden SSH server
Once you have verified that your key-based logins are working, you can disable username/password logins to achieve better security.
-
To do this, you need to edit your SSH server's configuration file. On Debian/ Ubuntu systems, this file is located at
/etc/ssh/sshd_config.sudo nano /etc/ssh/sshd_config -
Clear the contents and paste the contents below:
# $OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $ # This is the sshd server system-wide configuration file. See # sshd_config(5) for more information. # This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin # The strategy used for options in the default sshd_config shipped with # OpenSSH is to specify options with their default value where # possible, but leave them commented. Uncommented options override the # default value. Include /etc/ssh/sshd_config.d/*.conf #Port 22 #AddressFamily any #ListenAddress 0.0.0.0 #ListenAddress :: #HostKey /etc/ssh/ssh_host_rsa_key #HostKey /etc/ssh/ssh_host_ecdsa_key #HostKey /etc/ssh/ssh_host_ed25519_key # Ciphers and keying #RekeyLimit default none # Logging #SyslogFacility AUTH LogLevel VERBOSE # Authentication: #LoginGraceTime 2m PermitRootLogin no #StrictModes yes MaxAuthTries 3 MaxSessions 2 #PubkeyAuthentication yes # Expect .ssh/authorized_keys2 to be disregarded by default in future. #AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2 #AuthorizedPrincipalsFile none #AuthorizedKeysCommand none #AuthorizedKeysCommandUser nobody # For this to work you will also need host keys in /etc/ssh/ssh_known_hosts #HostbasedAuthentication no # Change to yes if you don't trust ~/.ssh/known_hosts for # HostbasedAuthentication #IgnoreUserKnownHosts no # Don't read the user's ~/.rhosts and ~/.shosts files #IgnoreRhosts yes # To disable tunneled clear text passwords, change to no here! PasswordAuthentication no #PermitEmptyPasswords no # Change to yes to enable challenge-response passwords (beware issues with # some PAM modules and threads) ChallengeResponseAuthentication no # Kerberos options #KerberosAuthentication no #KerberosOrLocalPasswd yes #KerberosTicketCleanup yes #KerberosGetAFSToken no # GSSAPI options #GSSAPIAuthentication no #GSSAPICleanupCredentials yes #GSSAPIStrictAcceptorCheck yes #GSSAPIKeyExchange no # Set this to 'yes' to enable PAM authentication, account processing, # and session processing. If this is enabled, PAM authentication will # be allowed through the ChallengeResponseAuthentication and # PasswordAuthentication. Depending on your PAM configuration, # PAM authentication via ChallengeResponseAuthentication may bypass # the setting of "PermitRootLogin without-password". # If you just want the PAM account and session checks to run without # PAM authentication, then enable this but set PasswordAuthentication # and ChallengeResponseAuthentication to 'no'. UsePAM yes AllowAgentForwarding no AllowTcpForwarding no #GatewayPorts no X11Forwarding yes #X11DisplayOffset 10 #X11UseLocalhost yes #PermitTTY yes PrintMotd no #PrintLastLog yes TCPKeepAlive no #PermitUserEnvironment no Compression no #ClientAliveInterval 0 ClientAliveCountMax 2 #UseDNS no #PidFile /var/run/sshd.pid #MaxStartups 10:30:100 #PermitTunnel no #ChrootDirectory none #VersionAddendum none # no default banner path #Banner none # Allow client to pass locale environment variables AcceptEnv LANG LC_* # override default of no subsystems Subsystem sftp /usr/lib/openssh/sftp-server # Example of overriding settings on a per-user basis #Match User anoncvs # X11Forwarding no # AllowTcpForwarding no # PermitTTY no # ForceCommand cvs server AllowUsers jodibooks@192.168.1.0/24To save, press ctrl+s followed by ctrl+x.
-
Now, reload the SSH server's configuration:
sudo service ssh reload
¶ Install auditing/logging/security tools
sudo apt install apt-show-versions debsums aide
Initialize aide but note this will take forever. I think because it also scans mounted NFS/SMB drives. I don't know how to exclude those. It seems not to be possible at all: https://github.com/aide/aide/issues/39.
aideinit
Sources:
How To Configure The AIDE (Advanced Intrusion Detection Environment) File Integrity Scanner For Your Website
Securing Ubuntu with AIDE
https://wiki.archlinux.org/title/AIDE
¶ Install malware scanners
sudo apt install clamav clamav-daemon rkhunter chkrootkit
ClamAV can use up a lot of CPU and memory resources. So only run it manually when you know it won't interfere with other system tasks. Then uninstall (
sude apt purge clam*) or disable it.
¶ Check for malware
Sources:
https://upcloud.com/community/tutorials/scan-ubuntu-server-malware/
¶ ClamAV
sudo clamscan --infected --recursive --exclude-dir="^/sys" /
Use below for local systems (workstations) with large (compressed) files:
sudo clamscan --max-scansize=1024M --max-filesize=250M --infected --recursive --exclude-dir="^/sys" /
¶ rkHunter
sudo rkhunter --propupd
sudo rkhunter --checkall
sudo cat /var/log/rkhunter.log | grep -i warning
¶ Chrootkit
sudo chkrootkit | sudo tee /var/log/chkrootkit/chkrootkit.log
sudo cat /var/log/chkrootkit/chkrootkit.log | grep -i warning
¶ Purge removed application data
sudo apt-get purge $(dpkg -l | grep '^rc' | awk '{print $2}')
¶ Find and remove unused compilers
WARNING: do not blindly remove all compilers the first command returns! Remove one by one with the second command and read carefully which dependencies are removed alongside the compiler.
dpkg -l | egrep -i "compil" | awk '{print $2}'
sudo apt remove ...
¶ Check debsums installed packages
dpkg -l | egrep '^ii' | awk '{print $2}' | xargs sudo debsums | egrep -v 'OK'
¶ nano
¶ Shortkeys
-
Open and edit files. Keys:
^means Ctrl,Mmeans Alt: -
Exit: Ctrl+X or F2
-
Save: Ctrl+O or Ctrl+S or F3
-
Cut: Ctrl+K or F9
-
Copy: Alt+6
-
Paste: Ctrl+U or F10
-
Mark text: Alt+A
-
Undo: Alt + U
-
Redo: Alt + E
-
Search forward: Ctrl+W or F6
-
Search next occurence forward: Alt+W
-
Search backward: Ctrl+Q
-
Search next occurence backward: Alt+Q
-
Go to line x: Ctrl+_ or Alt + G
nano file.ext
¶ SSH key authentication
¶ Create a key pair
Determine the use case and start creating a key pair. Some known limitations:
-
Guacamole 1.4.0 only supports PEM style public keys, see this Reddit topic. This seems to be solved with the next release, 1.5.0. For now use:
ssh-keygen -t rsa -b 4096 -m PEM -
Ubuntu should be fine with all key formats, so use the strongest:
ssh-keygen -t ed25519
When asked for a passphrase enter one. This makes the private key safer. If you lose it, not all is lost; you have some time to change it on your server/application.
¶ Copy public key to server
Now, you need to copy the public key to the file ~/.ssh/authorized_keys on your server.
¶ Option 1
The server does not have a public key and pasword authentication is not disabled.
-
Copy the key to the server:
ssh-copy-id user@12.34.56.78
¶ Option 2
The server already has a public key and password authentication is disabled.
-
Log in to your destination server through ssh with the existing private key.
ssh -i ~/.ssh/<key> user@12.34.56.78 -
Paste the public key into your
~/.ssh/authorized_keysfile:sudo nano ~/.ssh/authorized_keysTo save, press ctrl+s followed by ctrl+x.
¶ Update client config
-
Browse to the
.sshfolder inside your users home folder Windows:C:\Users\<user>\.ssh, Ubuntu:~/.ssh. -
Create or open a file
configand add the machine):Host jodibooks-server HostName 192.168.1.100 User jodibooks Port 22 IdentityFile ~/.ssh/jodibooks -
Save the file and open a command prompt (Windows) or shell (Ubuntu) and connect to the server:
ssh jodibooks-server
¶ Users and groups
¶ Add a user
sudo useradd -M -d /usr/jodibooks/joep -s /usr/bin/bash -G jodibooks smbadmins joep -p <PASSWORD>
Sources:
https://www.computerhope.com/unix/useradd.htm
https://linux.die.net/man/8/useradd
-d HOME_DIR: The new user is created using HOME_DIR as the value for the user's login directory.-g GROUP: The group name or number of the user's initial login group. The group name must exist.-G GROUP1[,GROUP2,...[,GROUPN]]]: A list of groups containing the user as a member. Each group is separated from the next by a comma, with no intervening whitespace.-m: Create the user's home directory if it does not exist.-M: Do no create the user's home directory-N: Do not create a group with the same name as the user, but add the user to the group specified by the -g option-s <path to shell>: The name of the user's login shell.-p: The encrypted password, as returned by crypt. The default is to disable the password.-U: Create a group with the same name as the user, and add the user to this group.
¶ Add a group
sudo groupadd jodibooks
¶ Add user to group
¶ Current user
sudo usermod -aG jodibooks ${USER}
¶ Specific user
sudo usermod -aG jodibooks joep
¶ File sharing
¶ NFS shares
Source:
https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nfs-mount-on-ubuntu-20-04
¶ Network info
- Subnet: 192.168.1.0/24
- Server address: 192.168.1.100
- Client address: 192.168.1.200
¶ Server
-
Install NFS on server:
sudo apt update && sudo apt install nfs-kernel-server -
Create shared folder and set ownership:
sudo mkdir /usr/nfs/jodibooks -p sudo chown nobody:nogroup /usr/nfs/jodibooks -
Export the directory:
sudo nano /etc/exportsAdd a new line with the directory followed by the allowed client(s) address(es) and settings. The file should look something like this:
# /etc/exports: the access control list for filesystems which may be exported # to NFS clients. See exports(5). # # Example for NFSv2 and NFSv3: # /srv/homes hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check) # # Example for NFSv4: # /srv/nfs4 gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check) # /srv/nfs4/homes gss/krb5i(rw,sync,no_subtree_check) # /usr/nfs/jodibooks 192.168.1.0/24(rw,sync,no_root_squash,no_subtree_check) -
Restart the NFS service:
sudo systemctl restart nfs-kernel-server -
[Optional] Open the NFS port in the firewall:
sudo ufw allow from 192.168.1.200 to any port nfs
¶ Client
-
Install nfs-common:
sudo apt update && sudo apt install nfs-common -
Create a directory:
sudo mkdir ~/mnt/jodibooks -p -
Mount that directory to the NFS shared directory:
sudo mount 192.168.1.100:/usr/nfs/jodibooks /mnt/jodibooks -
Check if it’s mounted:
df -hWhich should return something like this:
Filesystem Size Used Avail Use% Mounted on udev 3.9G 0 3.9G 0% /dev tmpfs 797M 2.0M 795M 1% /run /dev/sda2 40G 31G 7.3G 81% / tmpfs 3.9G 0 3.9G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup 192.168.1.100:/usr/nfs/jodibooks 5.3T 901G 4.4T 17% /mnt/jodibooks tmpfs 797M 0 797M 0% /run/user/1000 -
Mount the directory at boot:
sudo nano /etc/fstabAdd a new line with the remote directory followed by the local directory and some settings. The file should look something like this:
# /etc/fstab: static file system information. # # Use 'blkid' to print the universally unique identifier for a # device; this may be used with UUID= as a more robust way to name devices # that works even if disks are added and removed. See fstab(5). # # <file system> <mount point> <type> <options> <dump> <pass> # / was on /dev/sda2 during curtin installation /dev/disk/by-uuid/591bf60a-048f-42b7-b716-280b251ca5cf / ext4 defaults 0 0 /swap.img none swap sw 0 0 192.168.1.100:/usr/nfs/jodibooks /mnt/jodibooks nfs auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0
¶ Samba (SMB) shares
This guide is untested!
This guide is compiled from multiple guides and prior knowledge. I have not actually tested it, as I have all my shares running through TrueNAS and I didn't want to fuck things up.Use the sources below when running into any issues.
Source:
https://linuxize.com/post/how-to-install-and-configure-samba-on-ubuntu-18-04/
https://www.linuxbabe.com/ubuntu/install-samba-server-file-share
https://www.techrepublic.com/article/how-to-create-a-samba-share-on-ubuntu-server-20-04/
¶ Shares info
- Server IP:
192.168.1.100 - Admin group:
smbadmins - User group:
jodibooks - Admin user:
joep - User:
diana - Windows workgroup:
JODIBOOKS - Password protected share for group:
/usr/samba/jodibooks - Password protected private share for user:
/user/samba/<USER>
¶ Server
-
Install Samba on server:
sudo apt update && sudo apt install samba -
Configure Samba:
sudo cp /etc/samba/smb.conf{,.backup} sudo nano /etc/samba/smb.confChange the workgroup to your Windows workgroup (
JODIBOOKS) if you have one:# Change this to the workgroup/NT-domain name your Samba server will part of workgroup = JODIBOOKS -
Create samba groups:
sudo groupadd jodibooks sudo groupadd smbadmins -
Create users and add to group(s) (add existing system users with
sudo usermod -aG sambagrp <USER>):sudo useradd -M -d /usr/jodibooks/joep -s /usr/sbin/nologin -G jodibooks smbadmins joep -p <PASSWORD> sudo smbpasswd -a joep sudo smbpasswd -e joep sudo useradd -M -d /usr/jodibooks/diana -s /usr/sbin/nologin -G jodibooks diana -p <PASSWORD> sudo smbpasswd -a diana sudo smbpasswd -e diana
Samba user passwords aren't automatically the same as the system users passwords. I always forget that. You have to add that password to Samba too, hence the second command above.
-
Create shared folders and set ownership:
# create main folder sudo mkdir /usr/samba sudo chgrp jodibooks: /usr/samba # create shared group folder sudo mkdir /usr/samba/jodibooks sudo chgrp jodibooks: /usr/samba/jodibooks sudo chmod g+rw /usr/samba/jodibooks # create admin user folder sudo mkdir /usr/samba/joep sudo chown joep:jodibooks /usr/samba/joep sudo chmod 2770 /usr/samba/joep # create user folder sudo mkdir /usr/samba/diana sudo chown diana:jodibooks /usr/samba/diana sudo chmod 2770 /usr/samba/dianaScroll all the way to the bottom and add the share(s):
[jodiBooks] path = /usr/samba/jodibooks valid users = @jodibooks @sadmin browseable = yes read only = no force create mask = 0660 force directory mask = 2770 guest ok = no [joep] path = /usr/samba/joep valid users = joep @sadmin browseable = no read only = no force create mask = 0660 force directory mask = 2770 guest ok = no [diana] path = /usr/samba/diana valid users = diana @sadmin browseable = no read only = no force create mask = 0660 force directory mask = 2770 guest ok = no -
Restart the Samba service:
sudo systemctl restart smbd sudo systemctl restart nmbd -
[Optional] Open the Samba port in the firewall:
sudo ufw allow samba
¶ Client
-
Install Samba:
sudo apt update && sudo apt install cifs-utils -
Create a directory:
sudo mkdir /media/jodibooks sudo mkdir /media/joep
You can mount the shares anywhere you like. The unwritten rule, however, is to mount them in the mount folder
/mnt. For servers I would indeed use this folder.
However, on my Ubuntu PC I mounted the shares in the media folder:/media. This has the advantage that the shares are conveniently displayed in the Nautilus file browser, so you don't have to browse away from the/homefolder and to/media.
-
Mount that directory to the Samba shared directory:
sudo mount //192.168.1.100/jodibooks /media/jodibooks -U joep sudo mount //192.168.1.100/joep /media/joep -U joep -
Check if they are mounted:
df -hWhich should return something like this:
Filesystem Size Used Avail Use% Mounted on udev 3.9G 0 3.9G 0% /dev tmpfs 797M 2.0M 795M 1% /run /dev/sda2 40G 31G 7.3G 81% / tmpfs 3.9G 0 3.9G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup //192.168.1.100/jodibooks 5.3T 901G 4.4T 17% /media/jodibooks //192.168.1.100/joep 5.3T 901G 4.4T 17% /media/joep tmpfs 797M 0 797M 0% /run/user/1000 -
To mount the directory at boot, first add the shares to
/etc/fstab:sudo nano /etc/fstabAdd a new line with the remote directory followed by the local directory and some settings. The file should look something like this:
# /etc/fstab: static file system information. # # Use 'blkid' to print the universally unique identifier for a # device; this may be used with UUID= as a more robust way to name devices # that works even if disks are added and removed. See fstab(5). # # <file system> <mount point> <type> <options> <dump> <pass> # / was on /dev/sda2 during curtin installation /dev/disk/by-uuid/591bf60a-048f-42b7-b716-280b251ca5cf / ext4 defaults 0 0 /swap.img none swap sw 0 0 //192.168.1.100/jodibooks /media/jodibooks cifs credentials=/home/joep/.smbcredentials,uid=1000,gid=1000,vers=3.0,file_mode=0777,dir_mode=0777 0 0 //192.168.1.100/joep /media/joep cifs credentials=/home/joep/.smbcredentials,uid=1000,gid=1000,vers=3.0,file_mode=0777,dir_mode=0777 0 0 -
Create credentials file:
nano /home/joep/.smbcredentialsAdd your samba username password:
username=joep password=<PASSWORD> -
Mount the shares in
/etc/fstab:sudo mount -a -
Check if the shares are mounted:
df -hWhich should return something like this:
Filesystem Size Used Avail Use% Mounted on udev 3.9G 0 3.9G 0% /dev tmpfs 797M 2.0M 795M 1% /run /dev/sda2 40G 31G 7.3G 81% / tmpfs 3.9G 0 3.9G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup //192.168.1.100/jodibooks 5.3T 901G 4.4T 17% /media/jodibooks //192.168.1.100/joep 5.3T 901G 4.4T 17% /media/joep tmpfs 797M 0 797M 0% /run/user/1000
¶ File transfer
- From: local
- To:
12.34.56.78orjodibooks-server - Private key:
jodibooks-server - User:
joep
¶ Rsync
-a: archive mode-e ssh: enable ssh transfer with known key-e "ssh -i ~/.ssh/jodibooks-server": enable ssh transfer with specific key-h: preserve hard links-P: combines the flags--progressand--partial-v: verbose-z: compress--bwlimit=10240: bandwidth limit by backup transfer measured in KBPS--delete: delete files on destination that don't exist on the source--progress: show transfer progress
¶ With user-password authentication
rsync -a --delete /var/www/html/ joep@12.34.56.78:/var/www/html
¶ With ssh key authentication
rsync -a --delete -e "ssh -i ~/.ssh/jodibooks-server" /var/www/html/ joep@12.34.56.78:/var/www/html
¶ SCP (Secure copy)
Sources:
https://snapshooter.com/learn/linux/copy-files-scp
https://linuxize.com/post/how-to-use-scp-command-to-securely-transfer-files/
-C: Used to enable the compression.-i: Used to specify the private key file for authenticating a remote system.-l: Used to limit the bandwidth for transferring files.-P: Used to specify the SSH port to connect to the remote host.-p: Used to preserves modification times and access times from the original file.-r: Used to copy entire directories recursively.-v: Used to print debugging messages.
¶ With user-password authentication
scp -r /var/www/html joep@12.34.56.78:/var/www/
¶ With ssh key authentication
scp -i ~/.ssh/jodibooks-server" /var/www/html joep@12.34.56.78:/var/www/
¶ SFTP (SSH file transfer)
¶ Hardware configuration
¶ Sensors and fancontrol
Todo
Advanced lm-sensors Tips and Tricks on Linux
SSD temperature sensor readout with hddtemp
¶ Network tuning
If you are measuring the performance of your system with Netdata (see System measurements) you will get some alarms. Some are benign, others more serious. This is also when you learn that Linux is way more interesting than Windows, as you can tweak about everything. I'll list the tweaks I found below:
¶ Inbound packets dropped ratio
¶ Alarm details
Alarm name: inbound_packets_dropped_ratio
Warning message: Warning, inbound_packets_dropped_ratio = 5.08%, on <server>
Warning message: <server> needs attention - inbound packets dropped ratio - net_packets.<interface>
¶ Core issue
The
netdev_max_backlogis a queue within the Linux kernel where traffic is stored after reception from the NIC, but before processing by the protocol stacks (IP, TCP, etc). There is one backlog queue per CPU core.A given core's queue can grow automatically, containing a number of packets up to the maximum specified by the
netdev_max_backlogsetting.The
netif_receive_skb()kernel function will find the corresponding CPU for a packet, and enqueue packets in that CPU's queue. If the queue for that processor is full and already at maximum size, packets will be dropped.Source: https://steemit.com/linux/@blake.heo/linux-net-core-netdevmaxbacklog-tuning
¶ Solution
-
Find out if packets are dropped: if second column <> 0, which is not the case in the example below. We thus have dropped packages as the alarm suggested.
root@jplsrv2:~# cat /proc/net/softnet_stat 0257b6bc 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 022e23b3 00000000 00000001 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000001 0c83b7c5 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000002 05c5a92c 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000003 0f5bff18 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000004 0b7076d9 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000005 03ace21a 00000000 00000001 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000006 09eaff85 00000b03 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000007 1817bc0a 001cbac7 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000008 13777b64 00087795 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000009 05b307fa 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 0000000a 03415b00 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 0000000b -
Check the current backlog size
netdev_max_backlogand double it.root@jplsrv2:~# sudo sysctl net.core.netdev_max_backlog net.core.netdev_max_backlog = 1000 root@jplsrv2:~# sudo sysctl -w net.core.netdev_max_backlog=2000 net.core.netdev_max_backlog = 2000 -
Wait for the alarm to go silent ~10 minutes. If this doesn't happen, repeat step 1 and 2. In my current setup
JPLSRV1: 128000andJPLSRV2: 16000. -
This change will not survive a reboot. To make it permanent open file
/etc/sysctl.confsudo nano /etc/sysctl.confand add line:
net.core.netdev_max_backlog=8000Close the file and then apply the changes with:
sudo sysctl -p
¶ Commands
¶ awk
https://en.wikipedia.org/wiki/AWK
¶ cat, head, tail
Show the contents of a file. cat shows all, head the first 10 lines, tail the last 10 lines. Use -n xx to specify the number of lines.
Say we have a log file example.log with 10 lines:
line 1
line 2
line 3
line 4
line 5
line 6
line 7
line 8
line 9
line 10
¶ cat
joep@PCJOEP:~$ cat example.log
line 1
line 2
line 3
line 4
line 5
line 6
line 7
line 8
line 9
line 10
¶ head
joep@PCJOEP:~$ head example.log -n 3
line 1
line 2
line 3
¶ tail
joep@PCJOEP:~$ tail example.log -n 3
line 8
line 9
line 10
¶ chmod
Change permissions. Permissions can be set for "user" u, "group" g and "other" o. Permissions can be "read" r, "write" w and "execute" x. This can be translated to numbers as well: r = 4, w = 2, x = 1; sum them to get the final number. See https://en.wikipedia.org/wiki/Chmod for more info.
¶ Allow execution, but not for "others":
chmod o-x
¶ Allow read, write and execute for user:
chmod u+rwx
¶ Allow read for everyone:
chmod +r
¶ Set credentials for all files to 644 and folders to 755 (WordPress):
find /var/www/html -type d -exec chmod 755 {} \;
find /var/www/html -type f -exec chmod 644 {} \;
¶ chown
Change ownership. Ownership can be specified for a user and group.
chown user:group directory
Example, set owner of WordPress directory to www-data:www-data:
chown www-data:www-data -R /var/www/html
¶ df
Show information about the file system. Interesting flags (not all):
-hdisplays the output in "human readable form"-lonly show local file systems-Tprint type of file system-tonly show certain type of file system-xexclude a certain type of file system
¶ Show file systems without "snaps"
df -hlx "squashfs"
¶ du
Summarize disk usage of the set of FILEs, recursively for directories.
¶ Get a list of folder sizes in /:
sudo du -chax --max-depth=1 / | sort -hr | grep -E "M|G"
¶ Get a list of folder sizes in /home:
sudo du -chax --max-depth=1 /home | sort -hr | grep -E "M|G"
For
duto usesort -hrit needs disk space. So if the disk is completely full, the command above won’t work. You have to sift through the results manually first and delete some data. To do that remove| sort -hrfrom the command.
¶ echo
Returns a string. Mostly used in scripts to print output with notifications or instructions for the user. But also to show the value of environment variables.
¶ Show the exit code of the last command:
echo $?
¶ find
Find files and directories.
¶ Find log files and delete errors from output:
find / -name *.log 2> /dev/null
¶ Find files that have been modified within the last 30 days and delete them:
find /var/log -type f -name *.log* -mtime 30 -exec rm {} \;
¶ grep
Use this to filter the output of a command. For example only show a line that contains "8". See "cat, head, tail".
joep@PCJOEP:~$ cat example.log | grep 8
line 8
¶ journalctl
Print the system logs of a service. Use -f to keep the file open and follow new entries. Use -u with the unit (service) name to filter for that unit.
¶ Follow the Nginx unit logs
journalctl -fu nginx
¶ ps
Show system processes. Use grep or others to filter the output. See ps --help all for more info.
¶ sed
Search and replace.
Change words in a file; search for "xxx" and replace with "yyy" (you can use any delimiter: e.g. / | \ . , 'space'):
sed -i 's/xxx/yyy/' filename.ext
¶ service and systemctl
Source:
https://serverfault.com/questions/867322/what-is-the-difference-between-service-and-systemctl
service is an "high-level" command used for starting and stopping services in different unixes and linuxes. Depending on the "lower-level" service manager, service redirects on different binaries.
For example, on CentOS 7 it redirects to systemctl, while on CentOS 6 it directly calls the relative /etc/init.d script. On the other hand, in older Ubuntu releases it redirects to upstart.
service is adequate for basic service management, while directly calling systemctl give greater control options.
¶ Start a service:
sudo service nginx start
sudo systemctl start nginx
¶ Status of a service:
sudo service nginx status
sudo systemctl status nginx
¶ Restart a service:
sudo service nginx restart
sudo systemctl restart nginx
¶ Reload Nginx service:
sudo service nginx reload
sudo systemctl reload nginx
¶ Stop a service:
sudo service nginx stop
sudo systemctl stop nginx
¶ touch
Create en empty file.
touch file.ext
¶ uptime
Check resource usage. The load averages should be below the number of cores.
uptime
result:
16:01:39 up 2:58, 1 user, load average: 1,43, 1,86, 1,95
¶ Background - foreground switching
On pressing Ctrl+Z the program opened in the command line (e.g. nano) is moved to the background. To retrieve it press fg.
¶ Automation
¶ Bash scripting
Todo
¶ Cronjobs
Todo
¶ Recommended sources
- Learn Linux TV: A super video series Linux commands for beginners to get you comfortable to use the Linux command-line. You can also dive deeper into Docker, Ansible, Proxmox and more.
- DigitalOcean Community tutorials: Want to install something? Search here for a tutorial.