How to navigate/browse shell history in Linux (using hstr/hh)

hstr@github is a ncurses used to view, navigate, select and search command history from bash and zsh. Its an improved history|grep or ctrl-r.

$ wget -O - | tar -xz

# ncurses list of history filtered by PATTERN
# use options
$ HH_CONFIG=hicolor,regexp hh

from hstr: history the easy way


How to monitoring/show the progress of CLI tools in Linux (using cv and pv)

cv/coreutils viewer@github supports all basic utilities in coreutils package. It is written in C and shows the progress as percentage. Works by scanning /proc for commands that it supports, checks directories fd and fdinfo for opened files and seek positions and finally reports the progress for the largest file.

$ apt-get|yum|yaourt cv

'-m,--monitor' loop while monitored processes are still running
'-w,--wait' estimate I/O throughput and ETA
'-c,--command command' monitor only this command

# see all current and upcoming coreutils commands
$ watch cv -q
# see downloads progress
$ watch cv -wc firefox
# launch and monitor any heavy command using '$!'
$ cp bigfile newfile & cv -mp $!

from cv: progress bar for cp, mv, rm, dd…

pv/pipe viewer/pv@man is a terminal-based tool for monitoring the progress of data through a pipeline.

$ apt-get|yum|pacman install pv

command1 | pv | command2
pv input.file | command1 | pv > output.file

# display options
'-p,--progress/-t,--timer/-e,--eta/-b,--byte' show progress/timer/ETA/bytes
# output modifiers
'-N,--name NAME' prefix the output information
'-c,--cursor' use cursor positioning escape sequences
'-l,--line-mode' instead of counting bytes, count lines
'-s,--size SIZE' assume the total amount of data to be transferred is SIZE
# data transfer modifiers
'-L,--rate-limit RATE' limite transfer rate by bytes/sec
'-B,--buffer-size BYTES'

# watch how quickly a file is transferred using nc
$ pv file | nc -w 1 host 3000
# see progress of both pipes
$ pv -cN rawlogfile file.log | gzip | pv -cN gziplogfile > file.log.gz
# with ncurses's dialog
$ (pv -n backup.tar.gz | tar xzf - -C path/to/data ) 2>&1 | dialog --gauge "Running tar, please wait..." 10 70 0
# rsync and pv
$ rsync options source dest | pv -lpes Number-Of-Files

from pv@nixcraft and rsync and pv@nixcraft

How to lock your Linux terminal (using vlock and physlock)

vlock is a program to lock one or more sessions on the Linux console. This is especially useful for Linux machines which have multiple users with access to the console. One user may lock his or her session(s) while still allowing other users to use the system on other virtual consoles.

$(el) yum install kbd
$(deb) apt-get install 
$(arch) pacman -Sy kdb

'-c,--current' lock only this virtual console, allowing user to switch to other virtual consoles
'-a,--all' lock all virtual consoles by preventing other users from switching virtual consoles
'-n,--new' switch to a new virtual console before locking all console sessions, needs to be compiled with plugin

# lock current console
$ vlock
# lock all consoles (including virtual console); must run from virtual console
$ vlock -a
# lock whole system, switches to a virtual console and then locks as 'vlock -a'
$ sudo vlock -n

from vlock: lock your terminal (or system)

physlock@github lightweight alternative to vlock, equivalent to `vlock -an’. It is written because vlock blocks some linux kernel mechanisms like hibernate and suspend and can therefore only be used with some limitations.

$ git clone
$ cd physlock ; make ; sudo make install

'-d' fork and detach, useful for suspend/hibernate scripts
'-l' only lock console switching
'-L' only enable console switching
'-u USER'  allow the given user to unlock the computer

$ /usr/local/bin/physlock

Using pacapt universal package manager (pacman for all)

pacapt is a shell script that wraps native package managers in a pacman@arch syntax. Intended for Arch users having to temporarily deal with another distribution.

It supports pacman by arch, dpkg/apt-get by debian/ubuntu, homebrew by OSX, yum/rpm by redhat/centos/fedora, portage by gentoo, zypper by opensuse, pkgng by freebsd, pkg_tools by openbsd. See pacman_rosetta@arch

## install
$ wget -O /usr/local/bin/pacapt
$ chmod 755 /usr/local/bin/pacapt

'-Q' list packages installed
'-Qq FILE' query which package provides file
'-Ql PACKAGE' list package files
'-S PACKAGE' install package
'-Ss PACKAGE' search package
'-Suy' upgrade system and package db
'-R PACKAGE' remove package
'-Scc' delete all downloaded packages

How to remote syslog in Linux (using rsyslog)

rsyslog is an open-source implementation of syslog protocol / rfc3164 and extends it with content-based filtering, rich filtering capabilities, flexible configuration options and adds features such as using TCP for transport. Its used prior to migration to systemd-journald.

  • Facility level is type of processes to monitor: auth, cron, daemon, kernel, local0..local7
  • Severity/Priority level is type of log message: emerg/0, alert/1_, crit/2, err/3, warn/4, notice/5, info/6, debug/7
  • Destination is either local file or remote rsyslog server @ip:port

As a rsyslog client it can filter and sends internal log messages to either local file system or a remote rsyslog server. As rsyslog server it collects logs from other hosts and sends them into internal log messages. See syslogserver@windows.

$ yum install rsyslog | apt-get install rsyslog | pacman -S rsyslog

##(server) enable listener
$(host1) vi /etc/rsyslog.conf
# udp
$ModLoad imudp 
$UDPServerRun 514
# tcp (slower but more reliable)
$ModLoad imtcp 
$InputTCPServerRun 514 

##(server) create template to log to filesystem
# see
$(host1) vi /etc/rsyslog.d/remote_host
# log everything to 'host/progname.log'
$template RemoteLogs,"/var/log/%HOSTNAME%/%PROGRAMNAME%.log" *
# format it '[facility-level].[severity-level] ?RemoteLogs'
*.* ?RemoteLogs 
# stop processing messages
& ~

# same but using ip
$ vi /etc/rsyslog.d/remote_ip
$template IpTemplate,"/var/log/%FROMHOST-IP%.log" 
*.*  ?IpTemplate 
& ~

##(client) route all messages to remote server
$(host2) vi /etc/rsyslog.d/route_all
*.*  @host1:514 
# same but using tcp instead
#*.*  @@host1:514
# same but only for some kernel facility
kern.* @

$(both) service rsyslog restart | systemctl restart rsyslog

from rsyslog server@xmodulo and rsyslog client@xmodulo

syslog(3) is the syscall used to send messages to system logger. There are wrappers in all languages, including shells

## from shell
# see
$ logger -p -t PROGNAME MESSAGE

## forward journald to local syslog daemon
# see
$ vi {/etc,/run,/usr/lib}/systemd/journald.conf.d/*.conf
# same as kernel command line option 'systemd.journald.forward_to_syslog=True'

How to USB flash a bootable Linux installation media (using dd/ddrescue and testdrive)

dd@man/dd@wiki is a command-line utility for Unix and Unix-like operating systems whose primary purpose is to convert and copy files.

$ yum|apt-get|pacman coreutils

# find device name
$ lsblk
# unmount it
$ sudo umount /dev/sdx

# write/data transfer
$ sudo dd bs=4M if=/path/to/linux.iso of=/dev/sdx && sync

from usb flash@arch

gnu ddrescue@man combines both dd_rescue’s ability to read big blocks and then shift gears, with dd_rhelp’s ability to remember what parts of the disk have been looked at already. It’s written in C++ and it’s small and fast.

$(deb) apt-get install gddrescue 
$(el) yum install ddrescue 
$(arch) pacman -Su ddrescue

'-d,--direct' direct disc access for input file
'-D,--synchronous' use synchronous writes for output file

$ sudo ddrescue -d -D --force /path/to/linux.iso /dev/sdx

from disk drive recovery: ddrescue, dd_rescue, dd_rhelp

testdisk@arch can be used to recover if you dd/ddrescue to the wrong device

How to write a Linux kernel module

Kernel modules are pieces of code that can be loaded and unloaded into the kernel upon demand. They extend the functionality of the kernel without the need to reboot the system.

## obtaining information
# list loaded modules
$ lsmod
# show module info
$ modinfo MODULENAME
# list dependencies
$ modprobe --show-depends MODULENAME

## automatic module load
# configure udev/systemd-modules to what modules to load at boot, see 'man modules-load.d'
$ vi {/etc,/run,/usr/lib}/modules-load.d/PROGRAM.conf

## manual module load
# load by name
$ modprobe MODULENAME
# load by filename from '/lib/modules/$(uname -r)/'
$ insmod FILENAME [ARGS]
# unload module
$ modprobe -r MODULENAME
# same

## passing parameters to module
# either from '/etc/modprobe.d'
$ vi /etc/modprobe.d/FILENAME.conf
options MODULENAME parametername=parametervalue
# or from kernel command line

## blacklisting: prevent the kernel module from loading
# either from '/etc/modprobe.d'
$ vi /etc/modprobe.d/FILENAME.conf
blacklist MODULENAME
# or from kernel command line

from kernel modules@arch

You can write your own modules, see the linux kernel module programming guide.

# install build dependencies (kernel source)
$(deb) apt-get install build-essential linux-headers-$(uname -r)
$(el) yum install yum install gcc gcc-c++ make kernel-headers
$(arch) pacman -Syu base-devel linux-headers

# write a hello world module
$ vi hello.c
#include <linux/module.h> // all kernel modules
#include <linux/kernel.h> // KERN_EMERG, KERN_ALERT, KERN_CRIT, ... 
#include <linux/init.h>   // __init and __exit macros
MODULE_DESCRIPTION("A Simple Hello World module");
static int __init hello_init(void) {
    printk(KERN_NOTICE "Hello world!n");
    return 0; // non-0 means init_module failed
static void __exit hello_cleanup(void) {
    printk(KERN_NOTICE "Cleaning up module.n");

$ Makefile
obj-m += hello.o
    make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules
    make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean

# testing
$ make ; sudo insmod hello.ko
$ dmesg|grep -i hello
$ sudo rmmod hello.ko

from how to write your own linux kernel module

Using one-time passwords (OTPW) in SSH authentication

One-time passwords/OTPW consists of one-time password generator and PAM-integrated verification routines.

One-time passwords are generated apriori with the generator, given to the user and cryptographic hash of the generated passwords are stored in the host. When a user logs in with a one-time password, OTPW’s PAM module verifies the password, and invalidates it to prevent re-use.

## install (deb/ubuntu)
$ apt-get install otpw-bin libpam-otpw
$ vi /etc/pam.d/sshd
#@include common-auth
auth       required
session    optional

## install (arch)
$ yaourt -S otpw 
$ vi /etc/pam.d/ssh-otpw
auth sufficient
session optional
$ vi /etc/pam.d/sshd
auth      include   ssh-otpw
#auth      include   system-remote-login

## install (el/fedora)
$ yum git gcc pam-devel
$ git clone ; cd optw
$ vi Makefile
$ make ; sudo make install
# disable SELinux
$ vi /etc/selinux/config
$ vi /etc/pam.d/sshd
#auth       substack     password-auth
auth       required
session    optional

## configure sshd
$ vi /etc/ssh/sshd_config
UsePAM yes
UsePrivilegeSeparation yes
ChallengeResponseAuthentication yes
PubkeyAuthentication yes
PasswordAuthentication no
$ systemctl restart sshd

## generate passwords
# asks for a prefix to user cannot login even if list is public
$ otpw-gen > ~/temporary_password.txt
# print 'otpw_passwords' file; passwords are store in '~/.otpw'

## using
$ ssh user@remote_host
Password INDEX:
# use password in INDEX prefixed with your prefix
# if successful '~/.otpw' is updated to void used password

from otpw@xmodulo and otpw@arch/otpw@ubuntu

How to generate (random) and store (key-derived) passwords in Linux (using pwgen/apg/makepasswd/mkpasswd and crypt/bcrypt/scrypt)

Password strength is a measure of the effectiveness of a password in resisting guessing and brute-force attacks. They should not be repeated hence cryptographically secure random or one-time password.

Generating passwords/strings

# generates a base64 encoded 32 chars length random string
$ openssl rand -base64 32
  • pwgen@man generates pronounceable random (using /dev/urandom) passwords which are designed to be easily memorized by humans, while being as secure as possible.
$ yum install pwgen (EPEL) | apt-get install pwgen | pacman -S pwgen

'-s,--secure' generate completely random, hard-to-memorize passwords
'-B,--ambiguous' dont use characters 'I,1,O,0' that could be confused when printed
'-N,--num-passwords=NUM' number of passwords to generate
'-y,--symbols' include at least one special character in the password
# generate 2 passwords of 20 chars length each (default is 8)
$ pwgen -n 2 20

from pwgen@tecmint and pwgen@nixcraft

  • apg@man generates random passwords using pronounceable password generation algorithm (default) or /dev/urandom.
$ apt-get install apg | yum install apg | pacman -S apg

'-a ALG' 0 for pronounceable or 1 for urandom
'-n NUM' number of passwords to generate
'-m MIN, -x MAX' min/max password length
'-E CHARS' exclude chars from password
'-M MODE' 'S/s' must/should include special chars, 'N/n' numeral, 'C/c' capital, 'L/l' lowercase
'-r DICTFILE' check against a dictionary file
'-y' print encrypted password using crypt(3)
'-t' print pronunciation
$ apg -n 1 -t
  • makepasswd@ubuntu is a perl script that generates random (using /dev/urandom as seed) passwords with the emphasis on security over pronounceability. Optionally it also hashes/encrypts using crypt(3) des or md5.
$ apt-get install makepasswd | pacman -S makepasswd
# note (el/fedora): rawhide's makepasswd is a different program, same idea but in in C also using /dev/random and crypt(3)

'-chars N' password length (default is 10)
'--count N' number of passwords to generate
'-string STRING' uses chars in STRING to generate random password
# generates 2 passwords of 20 chars each
$ makepasswd --char 20 --count 2

'--clearfrom FILE' use tpassword from FILE instead of generating passwords
'--crypt' generates encrypted passwords
'--crypt-md5' encrypts using MD5 (default to DES)
'--cryptsalt N' uses crypt() salt N, defaults to 0 (random seeds)
# encrypt a password (instead of generating)
$ cat PASSWORD > pwd.txt
$ makepasswd --clearfrom pwd.txt --crypt

from makepasswd@tecmint and makepasswd@nixcraft

$ apt-get install expect | yum install expect | pacman -S expect
# note(deb/ubuntu): mkpasswd (from whois) is just a crypt(3) wrapper, use expect_mkpasswd instead

# generates a 10 char password and applies to current user
$ ?expect_?mkpasswd -l 10 $USER

from mkpasswd@tecmint and mkpasswd@nixcraft

Using key derivation functions (to store passwords)

Passwords should not be stored as plain-text, but instead be stored as result of key derivation functions. Basically a hash with a salt (a long random string that needn’t be kept secret) called many times.

passwd@wki is used to change user’s password. It is stored in /etc/shadow file encrypted/hashed by crypt(3). Format is $id$salt$encrypted where $id$ is $1$ for MD5, $2$ for blowfish, $5$ for SHA-256 and $6$ for SHA-512(default).

  • openssl dgst digest/hash functions output the message digest of a supplied file or files in hexadecimal.
# sha512 hash a script
$ echo -n STRING | openssl dgst -sha512
  • crypt(3) is the traditional Unix key derivation function. It uses DES by default, but also supports others.
# using deb/ubuntu mkpasswd (from whois), a crypt(3) wrapper
# SHA-512/crypt a password using given a salt
$(deb) mkpasswd PASSWORD -s SALTSASLT -m SHA-512

# DES/crypt a password using a random salt
$ openssl passwd -crypt PASSWORD
  • pbkdf2 is a key derivation function that is part of RSA. It applies a pseudorandom function, such as a cryptographic hash, cipher, or HMAC to the input password along with a salt value and repeats the process many times to produce a derived key. OpenSSL’s PKCS5_PBKDF2_HMAC is one implementation.
$ vi pbkdf2.c
/* usage:  % $0 password salt 1024 56 | openssl base64 */
#include <err.h>
#include <errno.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <openssl/evp.h>
int main(int argc, char *argv[])
    const char *usage = "usage: %s <password> <salt> <iterations> <keylen>";
    if (argc != 5) errx(1, usage, argv[0]);
    const char *pass = argv[1], *salt = argv[2]; 
    int iter = (int)strtol(argv[3], (char **)NULL, 10);
    int keylen = (int)strtol(argv[4], (char **)NULL, 10);

    unsigned char *out = malloc(keylen);
    PKCS5_PBKDF2_HMAC_SHA1(pass, strlen(pass), salt, strlen(salt), iter, keylen, out); 
    return (write(1, out, keylen) == -1) ? 1: 0;

from pbkdf2.c

# blowfish in CFB mode, prompts for password
$ openssl enc -base64 -e -bf -in <infile> -out <outfile>
$ yum install scrypt | apt-get install scrypt | pacman -S scrypt 

# prompts for password 
$ scrypt enc infile outfile

How to non-interactive/unattended password login to ssh (using sshpass and ssh_askpass)

  • sshpass@man is a noninteractive ssh password provider. Its an less secure alternative to public key authentication. sshpass only works if the password prompt ends in assword:.
## install
$ apt-get install sshpass | yum install sshpass (EPEL) | pacman -S sshpass

'-p PASSWORD' take password from cli
'-e' take password from SSHPASS env var

# need to disable host key checking
$ sshpass -p PASSWORD ssh -o StrictHostKeyChecking=no user@host

# rsync (and scp) over password ssh
$ SSHPASS=PASSWORD rsync --rsh='sshpass -e ssh -l user' host:path .
$ sshpass -p PASSWORD scp user@host:path .

from sshpass@cyberciti

  • ssh@man using SSH_ASKPASS. If ssh does not have a terminal associated with it but DISPLAY and SSH_ASKPASS are set, it will execute the program specified by SSH_ASKPASS and open an X11 window to read the passphrase.
$ vi
$ export DISPLAY=:0 ; export SSH_ASKPASS=`pwd`/
$ setsid ssh -o StrictHostKeyChecking=no user@host

from ssh password from stdin