hstr@github is a ncurses used to view, navigate, select and search command history from bash and zsh. Its an improved history|grep or ctrl-r.
$ wget https://github.com/dvorka/hstr/releases/download/1.16/hh-1.16-bin-64b.tgz -O - | tar -xz # ncurses list of history filtered by PATTERN $ hh PATTERN # use options $ HH_CONFIG=hicolor,regexp hh
cv/coreutils viewer@github supports all basic utilities in coreutils package. It is written in C and shows the progress as percentage. Works by scanning /proc for commands that it supports, checks directories fd and fdinfo for opened files and seek positions and finally reports the progress for the largest file.
$ apt-get|yum|yaourt cv '-m,--monitor' loop while monitored processes are still running '-w,--wait' estimate I/O throughput and ETA '-c,--command command' monitor only this command # see all current and upcoming coreutils commands $ watch cv -q # see downloads progress $ watch cv -wc firefox # launch and monitor any heavy command using '$!' $ cp bigfile newfile & cv -mp $!
from cv: progress bar for cp, mv, rm, dd…
pv/pipe viewer/pv@man is a terminal-based tool for monitoring the progress of data through a pipeline.
$ apt-get|yum|pacman install pv command1 | pv | command2 pv input.file | command1 | pv > output.file # display options '-p,--progress/-t,--timer/-e,--eta/-b,--byte' show progress/timer/ETA/bytes # output modifiers '-N,--name NAME' prefix the output information '-c,--cursor' use cursor positioning escape sequences '-l,--line-mode' instead of counting bytes, count lines '-s,--size SIZE' assume the total amount of data to be transferred is SIZE # data transfer modifiers '-L,--rate-limit RATE' limite transfer rate by bytes/sec '-B,--buffer-size BYTES' # watch how quickly a file is transferred using nc $ pv file | nc -w 1 host 3000 # see progress of both pipes $ pv -cN rawlogfile file.log | gzip | pv -cN gziplogfile > file.log.gz # with ncurses's dialog $ (pv -n backup.tar.gz | tar xzf - -C path/to/data ) 2>&1 | dialog --gauge "Running tar, please wait..." 10 70 0 # rsync and pv $ rsync options source dest | pv -lpes Number-Of-Files
from pv@nixcraft and rsync and pv@nixcraft
vlock is a program to lock one or more sessions on the Linux console. This is especially useful for Linux machines which have multiple users with access to the console. One user may lock his or her session(s) while still allowing other users to use the system on other virtual consoles.
$(el) yum install kbd $(deb) apt-get install $(arch) pacman -Sy kdb '-c,--current' lock only this virtual console, allowing user to switch to other virtual consoles '-a,--all' lock all virtual consoles by preventing other users from switching virtual consoles '-n,--new' switch to a new virtual console before locking all console sessions, needs to be compiled with plugin # lock current console $ vlock # lock all consoles (including virtual console); must run from virtual console $ vlock -a # lock whole system, switches to a virtual console and then locks as 'vlock -a' $ sudo vlock -n
from vlock: lock your terminal (or system)
physlock@github lightweight alternative to vlock, equivalent to `vlock -an’. It is written because vlock blocks some linux kernel mechanisms like hibernate and suspend and can therefore only be used with some limitations.
$ git clone https://github.com/muennich/physlock $ cd physlock ; make ; sudo make install '-d' fork and detach, useful for suspend/hibernate scripts '-l' only lock console switching '-L' only enable console switching '-u USER' allow the given user to unlock the computer $ /usr/local/bin/physlock
pacapt is a shell script that wraps native package managers in a pacman@arch syntax. Intended for Arch users having to temporarily deal with another distribution.
It supports pacman by arch, dpkg/apt-get by debian/ubuntu, homebrew by OSX, yum/rpm by redhat/centos/fedora, portage by gentoo, zypper by opensuse, pkgng by freebsd, pkg_tools by openbsd. See pacman_rosetta@arch
## install $ wget -O /usr/local/bin/pacapt https://github.com/icy/pacapt/raw/ng/pacapt $ chmod 755 /usr/local/bin/pacapt '-Q' list packages installed '-Qq FILE' query which package provides file '-Ql PACKAGE' list package files '-S PACKAGE' install package '-Ss PACKAGE' search package '-Suy' upgrade system and package db '-R PACKAGE' remove package '-Scc' delete all downloaded packages
rsyslog is an open-source implementation of syslog protocol / rfc3164 and extends it with content-based filtering, rich filtering capabilities, flexible configuration options and adds features such as using TCP for transport. Its used prior to migration to systemd-journald.
- Facility level is type of processes to monitor: auth, cron, daemon, kernel, local0..local7
- Severity/Priority level is type of log message: emerg/0, alert/1_, crit/2, err/3, warn/4, notice/5, info/6, debug/7
- Destination is either local file or remote rsyslog server @ip:port
As a rsyslog client it can filter and sends internal log messages to either local file system or a remote rsyslog server. As rsyslog server it collects logs from other hosts and sends them into internal log messages. See syslogserver@windows.
$ yum install rsyslog | apt-get install rsyslog | pacman -S rsyslog ##(server) enable listener $(host1) vi /etc/rsyslog.conf # udp $ModLoad imudp $UDPServerRun 514 # tcp (slower but more reliable) $ModLoad imtcp $InputTCPServerRun 514 ##(server) create template to log to filesystem # see http://linux.die.net/man/5/rsyslog.conf $(host1) vi /etc/rsyslog.d/remote_host # log everything to 'host/progname.log' $template RemoteLogs,"/var/log/%HOSTNAME%/%PROGRAMNAME%.log" * # format it '[facility-level].[severity-level] ?RemoteLogs' *.* ?RemoteLogs # stop processing messages & ~ # same but using ip $ vi /etc/rsyslog.d/remote_ip $template IpTemplate,"/var/log/%FROMHOST-IP%.log" *.* ?IpTemplate & ~ ##(client) route all messages to remote server $(host2) vi /etc/rsyslog.d/route_all *.* @host1:514 # same but using tcp instead #*.* @@host1:514 # same but only for some kernel facility kern.* @192.168.1.25:514 $(both) service rsyslog restart | systemctl restart rsyslog
from rsyslog server@xmodulo and rsyslog client@xmodulo
syslog(3) is the syscall used to send messages to system logger. There are wrappers in all languages, including shells
## from shell
# see http://linux.die.net/man/1/logger
$ logger -p local0.info -t PROGNAME MESSAGE
## forward journald to local syslog daemon
# see http://www.freedesktop.org/software/systemd/man/journald.conf.html
$ vi {/etc,/run,/usr/lib}/systemd/journald.conf.d/*.conf
ForwardToSyslog=True
# same as kernel command line option 'systemd.journald.forward_to_syslog=True'
dd@man/dd@wiki is a command-line utility for Unix and Unix-like operating systems whose primary purpose is to convert and copy files.
$ yum|apt-get|pacman coreutils # find device name $ lsblk # unmount it $ sudo umount /dev/sdx # write/data transfer $ sudo dd bs=4M if=/path/to/linux.iso of=/dev/sdx && sync
from usb flash@arch
gnu ddrescue@man combines both dd_rescue’s ability to read big blocks and then shift gears, with dd_rhelp’s ability to remember what parts of the disk have been looked at already. It’s written in C++ and it’s small and fast.
$(deb) apt-get install gddrescue $(el) yum install ddrescue $(arch) pacman -Su ddrescue '-d,--direct' direct disc access for input file '-D,--synchronous' use synchronous writes for output file $ sudo ddrescue -d -D --force /path/to/linux.iso /dev/sdx
from disk drive recovery: ddrescue, dd_rescue, dd_rhelp
testdisk@arch can be used to recover if you dd/ddrescue to the wrong device
Kernel modules are pieces of code that can be loaded and unloaded into the kernel upon demand. They extend the functionality of the kernel without the need to reboot the system.
## obtaining information
# list loaded modules
$ lsmod
# show module info
$ modinfo MODULENAME
# list dependencies
$ modprobe --show-depends MODULENAME
## automatic module load
# configure udev/systemd-modules to what modules to load at boot, see 'man modules-load.d'
$ vi {/etc,/run,/usr/lib}/modules-load.d/PROGRAM.conf
MODULENAME
## manual module load
# load by name
$ modprobe MODULENAME
# load by filename from '/lib/modules/$(uname -r)/'
$ insmod FILENAME [ARGS]
# unload module
$ modprobe -r MODULENAME
# same
$ rmmod MODULENAME
## passing parameters to module
# either from '/etc/modprobe.d'
$ vi /etc/modprobe.d/FILENAME.conf
options MODULENAME parametername=parametervalue
# or from kernel command line
MODULENAME.parametername=parametercontents
## blacklisting: prevent the kernel module from loading
# either from '/etc/modprobe.d'
$ vi /etc/modprobe.d/FILENAME.conf
blacklist MODULENAME
# or from kernel command line
modprobe.blacklist=modname1,modname2,modname3
from kernel modules@arch
You can write your own modules, see the linux kernel module programming guide.
# install build dependencies (kernel source)
$(deb) apt-get install build-essential linux-headers-$(uname -r)
$(el) yum install yum install gcc gcc-c++ make kernel-headers
$(arch) pacman -Syu base-devel linux-headers
# write a hello world module
$ vi hello.c
#include <linux/module.h> // all kernel modules
#include <linux/kernel.h> // KERN_EMERG, KERN_ALERT, KERN_CRIT, ...
#include <linux/init.h> // __init and __exit macros
MODULE_LICENSE("GPL");
MODULE_AUTHOR("You");
MODULE_DESCRIPTION("A Simple Hello World module");
static int __init hello_init(void) {
printk(KERN_NOTICE "Hello world!n");
return 0; // non-0 means init_module failed
}
static void __exit hello_cleanup(void) {
printk(KERN_NOTICE "Cleaning up module.n");
}
module_init(hello_init);
module_exit(hello_cleanup);
$ Makefile
obj-m += hello.o
all:
make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules
clean:
make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean
# testing
$ make ; sudo insmod hello.ko
$ dmesg|grep -i hello
$ sudo rmmod hello.ko
One-time passwords/OTPW consists of one-time password generator and PAM-integrated verification routines.
One-time passwords are generated apriori with the generator, given to the user and cryptographic hash of the generated passwords are stored in the host. When a user logs in with a one-time password, OTPW’s PAM module verifies the password, and invalidates it to prevent re-use.
## install (deb/ubuntu) $ apt-get install otpw-bin libpam-otpw $ vi /etc/pam.d/sshd #@include common-auth auth required pam_otpw.so session optional pam_otpw.so ## install (arch) $ yaourt -S otpw $ vi /etc/pam.d/ssh-otpw auth sufficient pam_otpw.so session optional pam_otpw.so $ vi /etc/pam.d/sshd auth include ssh-otpw #auth include system-remote-login ## install (el/fedora) $ yum git gcc pam-devel $ git clone https://www.cl.cam.ac.uk/~mgk25/git/otpw ; cd optw $ vi Makefile PAMLIB=/usr/lib64/security $ make ; sudo make install # disable SELinux $ vi /etc/selinux/config SELINUX=disabled $ vi /etc/pam.d/sshd #auth substack password-auth auth required pam_otpw.so session optional pam_otpw.so ## configure sshd $ vi /etc/ssh/sshd_config UsePAM yes UsePrivilegeSeparation yes ChallengeResponseAuthentication yes PubkeyAuthentication yes PasswordAuthentication no $ systemctl restart sshd ## generate passwords # asks for a prefix to user cannot login even if list is public $ otpw-gen > ~/temporary_password.txt # print 'otpw_passwords' file; passwords are store in '~/.otpw' ## using $ ssh user@remote_host Password INDEX: # use password in INDEX prefixed with your prefix # if successful '~/.otpw' is updated to void used password
from otpw@xmodulo and otpw@arch/otpw@ubuntu
Password strength is a measure of the effectiveness of a password in resisting guessing and brute-force attacks. They should not be repeated hence cryptographically secure random or one-time password.
# generates a base64 encoded 32 chars length random string $ openssl rand -base64 32
$ yum install pwgen (EPEL) | apt-get install pwgen | pacman -S pwgen '-s,--secure' generate completely random, hard-to-memorize passwords '-B,--ambiguous' dont use characters 'I,1,O,0' that could be confused when printed '-N,--num-passwords=NUM' number of passwords to generate '-y,--symbols' include at least one special character in the password # generate 2 passwords of 20 chars length each (default is 8) $ pwgen -n 2 20
from pwgen@tecmint and pwgen@nixcraft
$ apt-get install apg | yum install apg | pacman -S apg '-a ALG' 0 for pronounceable or 1 for urandom '-n NUM' number of passwords to generate '-m MIN, -x MAX' min/max password length '-E CHARS' exclude chars from password '-M MODE' 'S/s' must/should include special chars, 'N/n' numeral, 'C/c' capital, 'L/l' lowercase '-r DICTFILE' check against a dictionary file '-y' print encrypted password using crypt(3) '-t' print pronunciation $ apg -n 1 -t
$ apt-get install makepasswd | pacman -S makepasswd # note (el/fedora): rawhide's makepasswd is a different program, same idea but in in C also using /dev/random and crypt(3) '-chars N' password length (default is 10) '--count N' number of passwords to generate '-string STRING' uses chars in STRING to generate random password # generates 2 passwords of 20 chars each $ makepasswd --char 20 --count 2 '--clearfrom FILE' use tpassword from FILE instead of generating passwords '--crypt' generates encrypted passwords '--crypt-md5' encrypts using MD5 (default to DES) '--cryptsalt N' uses crypt() salt N, defaults to 0 (random seeds) # encrypt a password (instead of generating) $ cat PASSWORD > pwd.txt $ makepasswd --clearfrom pwd.txt --crypt
from makepasswd@tecmint and makepasswd@nixcraft
$ apt-get install expect | yum install expect | pacman -S expect # note(deb/ubuntu): mkpasswd (from whois) is just a crypt(3) wrapper, use expect_mkpasswd instead # generates a 10 char password and applies to current user $ ?expect_?mkpasswd -l 10 $USER
from mkpasswd@tecmint and mkpasswd@nixcraft
Passwords should not be stored as plain-text, but instead be stored as result of key derivation functions. Basically a hash with a salt (a long random string that needn’t be kept secret) called many times.
passwd@wki is used to change user’s password. It is stored in /etc/shadow file encrypted/hashed by crypt(3). Format is $id$salt$encrypted where $id$ is $1$ for MD5, $2$ for blowfish, $5$ for SHA-256 and $6$ for SHA-512(default).
# sha512 hash a script $ echo -n STRING | openssl dgst -sha512
# using deb/ubuntu mkpasswd (from whois), a crypt(3) wrapper # SHA-512/crypt a password using given a salt $(deb) mkpasswd PASSWORD -s SALTSASLT -m SHA-512 # DES/crypt a password using a random salt $ openssl passwd -crypt PASSWORD
$ vi pbkdf2.c
/* usage: % $0 password salt 1024 56 | openssl base64 */
#include <err.h>
#include <errno.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <openssl/evp.h>
int main(int argc, char *argv[])
{
const char *usage = "usage: %s <password> <salt> <iterations> <keylen>";
if (argc != 5) errx(1, usage, argv[0]);
const char *pass = argv[1], *salt = argv[2];
int iter = (int)strtol(argv[3], (char **)NULL, 10);
int keylen = (int)strtol(argv[4], (char **)NULL, 10);
unsigned char *out = malloc(keylen);
PKCS5_PBKDF2_HMAC_SHA1(pass, strlen(pass), salt, strlen(salt), iter, keylen, out);
return (write(1, out, keylen) == -1) ? 1: 0;
}
from pbkdf2.c
# blowfish in CFB mode, prompts for password $ openssl enc -base64 -e -bf -in <infile> -out <outfile>
$ yum install scrypt | apt-get install scrypt | pacman -S scrypt # prompts for password $ scrypt enc infile outfile
## install $ apt-get install sshpass | yum install sshpass (EPEL) | pacman -S sshpass '-p PASSWORD' take password from cli '-e' take password from SSHPASS env var # need to disable host key checking $ sshpass -p PASSWORD ssh -o StrictHostKeyChecking=no user@host # rsync (and scp) over password ssh $ SSHPASS=PASSWORD rsync --rsh='sshpass -e ssh -l user' host:path . $ sshpass -p PASSWORD scp user@host:path .
from sshpass@cyberciti
$ vi mypass.sh #!/bin/sh echo PASSWORD $ export DISPLAY=:0 ; export SSH_ASKPASS=`pwd`/mypass.sh $ setsid ssh -o StrictHostKeyChecking=no user@host
Rui's Blog 