I am a danish programmer living in Bangkok.
Read more about me @ rasmus.rummel.dk.
Webmodelling Home > Ubuntu Top Management Commands

Ubuntu shoot first ask later

Ammo for the everyday CLI cowboy - Ubuntu top management commands.

Appendixes

Learning the Shell - I should go through and see if it is good

Package Management

Package management involves especially installing the packages that you need. There are many many packages and it is often difficult to guess what is in it, eg. apache2.2-common - what is that ? Luckily there is :

difference between aptitude and apt-get.
  • tasksel
    • shell> tasksel : opens up a GUI for package selection from the installer
  • apt-get : Advanced Package Tool : installs, updates & removes packages (more apt-get examples)
    • /etc/apt/sources.list : defines the repositories used by apt-get (main, universe & multiverse)
    • /var/cache/apt/archives : the package cache holding all the downloaded packages (sometimes called the local repository of retrieved package files).
    • shell> apt-get install squirrelmail : install the squirrelmail package and any dependencies.
    • shell> apt-get remove squirrelmail : remove squirrelmail packages but keep the configuration files.
    • shell> apt-get --purge remove squirrelmail : remove squirrelmail packages including the configuration files.
    • shell> apt-get update : updates the APT package index based on the package repositories defined in /etc/apt/sources.list file.
    • shell> apt-get upgrade : upgrades the installed packages using the APT package index (you should always run apt-get update first).
    • shell> apt-get -u upgrade : will display all packages that will be upgraded.
    • shell> apt-get clean : totally purges the package cache.
    • shell> apt-get autoclean purges all .deb files in the package cache for packages no longer installed (prevents the package cache from growing large over time).
    • shell> apt-get autoremove
    • shell> apt-get install -f squirrelmail : -f tells APT to try to fix a broken dependency problem.
    • shell> apt-get install -s squirrelmail : -s tells APT to simulate an install without actually installing so you can see what packages will will be installed.
  • apt-cache :
    • shell> apt-cache search clamav :
    • shell> apt-cache show squirrelmail : will show the version of squirrelmail from the apt-cache, that is : the version that will be installed if issuing apt-get install squirrelmail.
  • dpkg
    • shell> dpkg -l | grep php5 : will list all packages containing the string php5 and also tell the version of each package.
    • shell> dpkg -s /full/path/to/file : will list the package that is responsible for installing the file.
    • shell> dpkg-reconfigure squirrelmail : opens the install configuration for squirrelmail, eg if you have configured squirrelmail wrongly then you installed it or if squirrelmail have stopped working.
    • shell> dpkg --get-selections | grep php5 : shows status (install, deinstall, hold) for all packages containing php5 in their name.
    • shell> echo "squirrelmail hold" | dpkg --set-selections : sets the squirrelmail package status to hold (default status is install) so that it cannot be upgraded to a newer version.
    • shell> echo "squirrelmail install" | dpkg --set-selections : remove the hold on the squirrelmail package so that it can be automatically upgraded.
    • shell> dpkg -l| grep -i linux-image : confirm the existing kernel.
  • which
    • shell> which mysql : shows the path to the mysql executable.
  • whereis
    • shell> whereis

Archiving & Compression

  • tar : used to collect multiple files into one file (called an archive), eg. for backup or transfer. Automatically collect directory structures.
    • Most used parameters :
      • c : create a new archive
      • v : verbose list files that are processed
      • f : following is the archive file name (also called tar ball)
      • z : apply gzip on the archive
      • j : apply bzip2
      • x : extract files from archive
    • shell> tar cf myDirectory.tar mydirectory : pack the directory myDirectory into a tar ball called myDirectory.tar.
    • shell> tar xf myDirectory.tar : extract all content from the .tar ball (the .tar ball will persist).
    • shell> tar cfz myDirectory.tar.gz mydirectory : pack the directory myDirectory into a tar ball and then compress (gzip) the tar ball.
    • shell> tar xfz myDirectory.tar.gz : decompress the tar ball and then extract all content from it.
    • shell> tar cf alldirs.tar dir1 dir2 dir3 : packs the specified directories into a tar ball.
    • shell> tar tvf myDirectory.tar : view the content of the tar ball (but do not extract from it).
    • shell> tar tvfz MyDirectory.tar.gz : view the content of the compressed tar ball (but do not decompress nor extract from it).
    • shell> tar xf MyDirectory.tar /path/to/file : extract a specific file from the tar pack.
    • shell> tar xf MyDirectory.tar /path/to/dir1 /path/to/dir2 : extract 2 specified directories from the tar ball.
    • On Windows 7z can decompress gzip and extract tar files - all in one process just like it was another compressed archive.
  • untar :
  • gzip :
    • shell> gzip myfile myfile.gz : will compress myfile and change the filename to myfile.gz (myfile is not preserved).
    • shell> gzip -c myfile > myfile.gz : will create a new file, myfile.gz, which is a compressed version of myfile. -c tells gzip to write result to stdout (myfile is preserved).
  • gunzip :
    • shell> gunzip myfile.gz : decompress myfile.gz and write it as myfile (myfile.gz is not preserved).n
    • shell> gunzip -c myfile.gz > myfile : decompress myfile.gz and write it as myfile (keeps the original myfile.gz).

File management

  • ls : list files & directories - one of the most important commands, here I give it quite a lot of attention
    • shell> ls : compact listing of all files in current directory.
    • shell> ls -1 : 1 entry per line, no attributes.
    • shell> ls -l : long listing (-l) as one entry per line with attributes :
      • field 1 1st char is file type : (unfortunately field 1 & 2 are not separated)
        • - : normal file
        • d : directory
        • s : socket file
        • l : link file
        • c : character special file (used for eg. device communication)
        • b : block special file (used for eg. device communication)
      • field 2 next 9 chars are file permissions (Read,Write,Execute for Owner,Group,All).
      • field 3 is the number of links to this file.
      • field 4 is the name of the owner this file belongs to.
      • field 5 is the name of the group this file belongs to.
      • field 6 is the file size in bytes (though this can be tweaked with the -h parameter, see below).
      • field 7 is the time the file was last modified.
      • field 8 is the filename.
    • shell> ls -lh : long listing (-l) but size attribute is displayed in human (-h) readable form (my favourite).
    • shell> ls -l /etc : long listing of the contents of the /etc directory.
    • shell> ls -ld /etc : long listing of the directory file /etc/.
    • shell> ls -ld */ : list only directories (a file that is not a directory cannot have the slash character, while on the other hand a directory always have the slash character)
    • shell> ls -l | grep ^d : list only directories (long listing starts with file type which is 'd' for directory files).
    • shell> ls -l | grep ^- : list only normal files (long listing starts with file type which is '-' for normal files).
    • shell> ls -l | grep ^- | awk '{print $8}' list only the filename (which is field 8) of the normal files long listing.
    • shell> ls -lt : long listing ordered by last modification time.
    • shell> ls -ltr : long listing in reverse order of last modificiation time.
    • shell> ls -la : includes all hidden files in the list including current (.) and parent (..) directory.
    • shell> ls -lA : includes all hidden files in the list excluding current (.) and parent (..) directory.
    • shell> ls -R : list files recursively (that can fast be a lot of files to show).
  • cp : copying files
    • shell> cp file1 file2 : makes a copy of file1 and call it file2 (in same directory)
    • shell> cp /var/file1 /web/file1 : makes a copy of /var/file1 and place it in /web
    • shell> cp -r dir1 dir2 : makes a copy of directory dir1 and call it dir2. The -r switch is necessary and it secures that all content of dir1 is copied to dir2.
    • scp : secury cp - copying files between servers
      • shell> scp -r root@myoldserver.com:/var/www/mycarsite/* /var/www/mycarsite : secure cp recursively (-r) everything in mycarsite on myoldserver.com to mycarsite on current server. Here the root user on myoldserver.com is used for login and you will be prompted for the root password on myoldserver.com.
      • shell> scp rasmus@myoldserver.com:/var/backup/mycarsite.tar /var/www : secure cp the mycarsite.tar file from the backup folder on myoldserver.com to the www folder on current server. Here I use the rasmus user and you I am prompted for the rasmus password on the myoldserver.com.
  • scp : secure copying file - perfect for network/internetwork copying as well as for between virtual machines.
    • shell> scp source target
  • mv : moving files
    • shell> mv file1 file2 : renames file1 to file2.
    • shell> mv file1 /var/file1 : moves file1 from current directory to /var
    • shell> mv file1 /var/file2 : moves file1 from current directory to /var and renames it to file2.
    • shell> mv * /var : moves all files & directories of current directory to /var (no -r switch is necessary nor supported).
    • shell> mv .* /var : moves all files & directories including files that starts with dot, especially .htaccess.
  • rm : deleting files
    • shell> rm file1 : deletes file1 if found in current directory.
    • shell> rm -r dir1 : deletes dir1 if dir1 is found in current directory. -r is necessary then deleting directories.
    • shell> rm -r * : deletes all files and directories in current directory.
    • shell> unlink symlinkfile : removes the symlink file, however rm symlinkfile will work the same even if the file is a symlink to a directory - the directory will not be deleted.
  • Finding files :
    • locate :
      • shell> updatedb : the locate command does not actually search the filesystem but instead search a search optimized datastore of filenames. Issue updatedb to index any new files before using the locate command.
      • shell> locate mail.log : find path to all file occurences having mail.log in its name.
    • find : used to locate files (15 good examples)
      • Where to find files :
        • find . : find all files recursively from current folder.
        • find web1/httpdocs : find all files recursively from web1/httpdocs there web1 is a subfolder of current folder.
        • find /etc : find all files recursively within the etc folder which is a subfolder of the root folder.
        • find / : find all files recursively from root, that is : find all files in the whole system.
      • What files to find :
        • find . -type f : find files of type file.
        • find . -type d : find files of type directory.
        • find . -name "*.php" : find all php files.
        • find . -iname "rasmus*" : find all files whose name starts with rasmus case-insensitive.
      • -print switch :
        • find . -type f : display files and path separated by newline.
        • find . -type f -print : exactly the same as without the -print switch.
        • find . -type f -print0 : display files and path separated by NULL.
      • Execute actions on the files found :
        • Execute using -exec switch :
          • find . -name "*.php" -exec wc -l {} \; : count the php files recursively from current folder :
            • -exec wc -l {} : execute the wc command (with -l switch) on the result of the find command {}.
            • {} : result of the find command (which here is a list of php files).
            • \; : end of the -exec switch.
          • find sourceFolder -name '*rasmus*' -exec cp {} targetFolder \;
          • find -type f -exec mv {} folderName \;
        • Execute using piping :
          • find . -print0 | xargs -0 : -print0 tells find to use NULL as delimeter instead of newline. -0 tells xargs to use NULL as delimeter instead of space. find & xargs now use the same delimeter.
      • shell> find /home -name '*.avi' : case-sensitive find all .avi files recursively starting from /home directory
      • shell> find /home -iname '*.avi' : case-insensitive find all .avi files recursively starting from /home directory
      • shell> find $HOME -iname '*.avi' -o -iname '*.mp3' : find all .avi OR -o .mp3 files starting from $HOME directory.
      • shell> find $HOME -iname '*.avi' -size +20 : find all files bigger than 20 megabyte.
      • shell> find $HOME -iname '*.avi' ! -size +20 : find all files NOT bigger than 20 megabyte.
      • shell> find / -iname '*.sh' -mtime 7 : find all .sh files recursively in the whole system modified within the last 7 days.
      • shell> find /directory_path -mtime -1 -ls : find all files recursively from directory_path that have changed the last day.
      • shell> find $HOME -iname '*.avi' -exec du -h '{}' \; : execute du -h (list file size) on each file found '{}'
      • shell> find -type f -exec mv {} folderName \; : move all files from current folder to folderName (remember -exec MUST end with \;).
      • shell> find sourceFolder -name '*rasmus*' -exec cp {} targetFolder \; : copy all files found in sourceFolder and below with 'rasmus' in its name to targetFolder.
    • grep : the ultimate command, here we use it to find files that contains a specific text (note that grep regular expressions are greedy and that grep does not support the non-greedy modifier)
      • most used parameters :
        • -i : ignore case
        • -l : suppress normal output, instead print only filename that contains the match.
        • -n : display line number within the file for each match.
        • -R : recursive search
        • -H : print filename
      • shell> grep -nHR 'webmodelling' * : search current folder recursively for files containing the string webmodelling and print the filename and the linenumber & line containing the string
      • shell> grep -B3 -A3 "text to find" myfile.txt : outputs 3 lines Before and 3 lines After the line with the text to find in myfile.txt file.
      • shell> grep -l -n "text to find" * : search all files in current directory and output only the names of the files containing "text to find" and also prefix the file names with a line number where the text was found.
      • shell> grep -l -n "text to find" /etc/* : search all files in the /etc folder for files containing "text to find", displaying only file names and line numbers.
      • shell> grep -R -i -l -n 'text to find' * search all files recursively from current directory and shows only filenames with linenumber prefixed.
      • shell> grep -i -n 'textToSearch' * : will list all files in current folder that contains the string textToSearch.
      • shell> grep -R -i -n 'textToSearch' * : will list all files recursively that contains the string textToSearch.
  • Counting files :
    • wc : Word Count : typically used to count files and always used with other commands to delivery the input to wc.
      • shell> ls -A | wc -l : list all files (ls) including directory files but excluding current & parent directory (-A). Send the list to Word Count (wc) and count all new lines (-l) - in effect count all files & directories in current directory.
      • shell> find . -type f | wc -l
    • grep : grep can also be used to for counting
      • shell> ls -A | grep -c . : send the list to grep and count (-c) all lines that match anything (.).
      • shell> ls -lA | grep ^- | grep -c . : send the list (this time long listing -l) to grep and list only files (^- will match only files in long listing) and then to grep again for counting (-c).
      • shell> ls -l | grep -c

Disk management

  • shell> cat /proc/partitions : show all mounted partitions (does not show the mount points)
  • du : Disk Usage : summarizes disk usage of each file recursively for directories
    • shell> du -hs /var/www : shows the size of the /var/www folder (h) human readable (s) summary
    • shell> du -h --max-dept=1 /var/www : will show the size for each subfolder of /var/www.
    • shell> du -h | less : output can be piped to the less filter, which is convenient then listing a lot of files or directories.
    • shell> du | sort -n : output can be piped to the sort filter (-n tells sort to order numeric)
    • shell>
  • df : (reports free & used disk space for each partition)
    • shell> df : reports free & used disk space for each partition.
    • shell> df -h : human readable (print sizes in K, M & G).
    • shell> df -T : show file system for each partition (only mounted partitions).
  • fdisk : partition manager
    • shell> fdisk -l : list partitions for all devices including USB connected disks.
    • shell> fdisk /dev/sdb : start partition manager for the /dev/sdb device (press 'm' to list all options).
  • mkfs : make file system
    • shell> mkfs.ext4 /dev/sdb1 : create an ext4 file system on the /dev/sdb1 partition.
  • mount :
    • shell> mount : show all mounted partitions including mount points.
    • shell> mount /dev/sdb1 /media/disk2 : will temporarily mount the partition sdb1 as /media/disk2.
    • shell> mount -a : will temporarily mount all partitions according to /etc/fstab that is not yet mounted.
  • umount :
    • shell> umount /media/disk2 : will temporarily unmount the partition on the /media/disk2 mount point.

User management

  • shell> cat /etc/group : list all user groups.
  • shell> cat /etc/passwd : list all users (format is : Username:Password:UserId:GroupId:UserInfo:HomeDir:Shell).
  • shell> ls /home : list all non-system users.
  • shell> nano /etc/sudoers : edit the list of user groups that have sudo access.
  • groups :
    • shell> groups rasmus : list all groups that the user rasmus belongs to.
  • members :
    • shell> members adm : list all users in the adm group.
  • passwd :
    • shell> passwd rasmus : will prompt for the new password to give the rasmus user.
    • shell> passwd root : will prompt for the new password to give the root user - this makes it possible to logon as root.
    • shell> passwd -l root : locks the root user making it impossible to logon as the root user (you can still sudo but not su).
  • adduser :
    • shell> adduser rasmus : creates a user called rasmus together with /home/rasmus and starts a prompt for more info like password etc.
    • shell> adduser rasmus employees : creates a user called rasmus together with /home/rasmus and add the new user to the employees group.
  • useradd :
    • shell> useradd -g employees -u 5000 rasmus -d /var/www/rasmus.com -m : creates a user called rasmus with userID 5000 and add to the employees group setting the home directory to /var/www/rasmus.com and specify to create the folder if it does not exist (-m).
  • usermod :
    • shell> usermod -g adm rasmus : change the primary group (-g) to adm for the rasmus user - a user can only belong to one primary gourp.
    • shell> usermod -G postfix,sshlogins rasmus : change the supplementary groups (-g) for the rasmus user to postfix & sshlogins - a user can belong to multiple supplementary groups.
    • shell> usermod -a -G employees rasmus : add (-a) the rasmus user to the employees groups without removing rasmus from the postfix and sshlogins groups).
    • shell> usermod -d /var/www/rasmus.com rasmus : changes the home directory for rasmus to /var/www/rasmus.com (owner, group and access modifiers are not changed on the target folder)
    • shell> usermod -s /bin/false rasmus : will prevent the rasmus user from opening any shell including ssh.
    • shell> usermod -s /bin/bash rasmus : sets the default shell to bash for the rasmus user (rasmus can now open a shell).
  • deluser :
    • shell> deluser rasmus : delete the rasmus user.
    • shell> deluser rasmus sshlogins : remove the rasmus user from the sshlogins group (it is not possible to remove a user from his primary group).
  • addgroup :
    • shell> addgroup employees : creates a user group called employees.
  • groupadd :
    • shell> groupadd -g 5000 employees : creates a user group called employees with groupID 5000.
  • delgroup :
    • shell> delgroup sshlogins : delete the sshlogins user group.
  • who : shows who are logged on the system
    • shell> who : all logged on users in list format.
    • shell> w : all logged on users in tabular format - more information.
    • shell> users : all logged on users in short format.

Networking

  • shell> cat /etc/network/interfaces : network device configuration
  • ifconfig : check & configure network interface cards
    • shell> ifconfig -a : display all network devices
    • shell> ifconfig -s : display all active devices
  • ifup & ifdown :
    • shell> ifdown eth0 && sudo ifup eth0 : restarts the eth0 interface device.
      Before Ubuntu 11 we would normally use /etc/init.d/networking restart, however that may now in some circumstances display an error Running /etc/init.d/networking restart is deprecated because it may not enable again some interfaces.
    • shell> ifdown --exclude=lo -a && sudo ifup --exclude=lo -a : restart all interfaces except the lo-interface.
  • netstat : summary of network connections and status of sockets
    • shell> netstat -tap | grep ftp :
    • shell> netstat -tap | grep mysql :
    • shell> netstat -anp | grep 9101 : will show if any server is listening on port 9101
    • shell> netstat -ltpn : show all daemons and what port they are listening on.
    • shell> netstat -ntulp
  • nmap : find open ports, that is : find what services are listening to what ports.
    • Syntax : nmap [scan type(s)] [options] {target} : target is either localhost or an IP address.
    • shell> nmap localhost : search the 1000 most common ports for services listening - it will show you port numbers and service names.
    • shell> nmap -p 0-65535 localhost : search port range 0-65535 (all ports) for services listening.
    • shell> nmap -sV localhost : include service/Version information.
    • shell> nmap 27.254.33.97 : default scan of remote server (27.254.33.97).
  • dig :
    • shell> dig google.com
    • shell> dig webmodelling.com @ns1.favouritehosting.com : ask a specific nameserver (ns1.favouritehosting.com) for records for the domain webmodelling.com.
    • shell> dig webmodelling.com @27.254.33.61 : also possible to use an IP address for the nameserver.
  • nslookup :
  • arp : Address Resolution Protocol - from IP to MAC - ethernet connectivity
  • ping :
  • route :
  • traceroute :

System information

  • shell> cat /etc/lsb-release : display version information about ubuntu OS.
  • shell> cat /proc/cpuinfo : display information about the CPU.
  • shell> cat /etc/services : shows all services and their port numbers among others.
  • uname :
    • shell> uname -r : display the kernel release, eg. 2.6.32-24 server
    • shell> uname -v : display the kernel version, eg. #39 Ubuntu SMP Wed Jul 28 06:21:40 UTC 2010
  • fuser : find processes that uses a particular file, directory or executable - handy if eg. you cannot delete a specific file or unmount a usb drive because the file or device is busy.
    • Access types : (a process can access a files in different ways)
      • c : current directory.
      • e : executable being run.
      • f : open file.
      • F : open file for writing.
      • r : root directory.
      • m : mmap'ed file or shared library.
    • shell> fuser . : display processes using the current directory.
    • shell> fuser -v . : display more information about process using the current directory.
    • shell> fuser -v -n tcp 80 display information about processes using port 80.
    • Most important parameters :
      • -v : verbose.
      • -n : namespace. 3 namespaces are supported : file (default), udp & tcp, so to look for processes that accesses tcp we need to fuser -n tcp PORT.
  • top : displays running processes
    • shell> top : displays processes ordered by CPU usage.
    • shell> top -> 1 : press 1 to divide the CPU usage out on cores.
    • shell> top -> M : press M to order processes by RAM usage.
    • shell> top -> O : press O to select a column to order after.
    • shell> top -> n : press n to specify how many processes you want to see (default top will fill up your windows height).
    • shell> top -> k : press k to kill a process by PID without leaving top.
    • shell> top -> h : press h to get help with interactive commands.
    • shell> top -> q : press q to quit top (you can also use ctrl+c).
    • shell> top -u rasmus : show only processes for the rasmus user.
    • shell> top -p 123 234 : show top only for the specified process IDs.
    • shell> top -d1 : show CPU & Memory consumption by users.
  • htop : same as top but better.
  • ps : Process Status : displays a snapshot of processes (here I only show a small subset of what the ps command can do). Note that all processes are in /proc/PID.
    • Params :
      • -e : select all processes (identical to -A).
      • -C cmdlist : select only those processes those executable name is in the cmdlist, eg. ps -C apache2.
      • -U userlist : select only those processes that was created by the username or userid in the userlist, eg. ps -U rasmus.
      • -p pidlist : select only those processes in the pidlist, eg. ps -p 275 4332.
      • Formatters : (formatters don't select processes, formatters controls what information to show about the selected processes)
        • -f : full format which shows some extra information (but not all information available).
        • -F : extra full format which shows a little more than -f (but still not all information available).
        • -l : long format.
        • --forest : IMPORTANT : display the parent child relationship between the processes (nearly same as -H).
        • -o columnlist : user defined format, only columns in columnlist are printed, eg. ps -eo pid,cmd will select all processes and show only PID & CMD. Some examples for the columnlist :
          • pid :
          • cmd : command that created the process.
          • args : args for the command then it created the process.
          • user : user who created the process.
          • group : group name.
          • lstart : start time.
          • etime : elapsed time.
          • tid :
          • pcpu :
          • state :
          • nlwp :
    • shell> ps -e : list all processes.
    • shell> ps -fe : list all processes in full format.
    • shell> ps -fe --forest : list all processes in full format but arrange the processes in a recursive parent child format as to show the process relationships (usefull if you cannot kill a process because it is being respawned).
    • shell> ps -fC apache2 : list all processes created by the apache2 command in full format.
    • shell> ps -fU rasmus : list all processes created by the rasmus user in full format.
    • shell> ps -fp 223 228 : list the 223 & 228 PIDs in full format.
    • shell> ps -eo pid,etime : list all processes but show only pid and etime columns (etime shows how long time the process have been running formattet as : [[dd-]hh:]mm:ss).
    • shell> px aux | grep apache : show all apache related daemons, their ports ect.
    • shell> ps -e | wc -l : count all processes.
    • Using ps with grep :
      • shell> ps -fe | grep apache2 : find all process IDs for apache2 and display them in full format. Note that this is equivalent to ps -fC apache2 and that using the -C option instead of piping to grep is more powerful because you can aggregate multiple commands.
      • shell> ps -eo pid,user,group,args,etime,lstart | grep 'PIDNUMBER' : display custom information about a specific process.
  • kill : kill processes
    • shell> kill 123 : kill process with process ID 123.
    • shell> kill -9 123 : forcefully (-9) kill (if the process seems not to be killed another process is watching and respawning the process).
    • shell> kill -15 123 : gracefully (-15) kill.
    • shell> kill mysql-server : kill all processes started by mysql-server.
    • shell> pkill -KILL -u rasmus : logout the rasmus user.
  • Hardware related :
    • nproc : displays number of processing units (threads)
    • lscpu : general CPU information
    • lshw : list hardware
      • shell> lshw : reports all your hardware in a huge list.
      • shell> lshw -short : reports all your hardware in an easy to read list.
      • shell> lshw -short | grep memory : reports only your memory hardware.
      • shell> lshw -c memory : reports hardware classified as memory.
      • shell> lshw -c disk : reports hardware classified as disk.
      • lshw classes : address, bridge, bus, communication, disk, display, generic, input, memory, multimedia, network, power, printer, processor, storage, system, tape, volume.
    • free :
      • shell> free -m : reports used and non-used memory.
  • reboot :
    • shell> reboot now : will reboot the system.
  • shutdown :
    • shell> shutdown -P now : will shutdown the system.
  • dmesg : lots of info - needs to be used with grep
  • dmidecode :
    • shell> dmidecode --type 0 : will display info about the bios (type 0). Types from 0-39 (use man dmidecode to get all types).
  • iostat
  • vmstat
  • prstat
  • pidstat
  • tcpdump
  • cpustat
  • crontab : used to schedule execution of commands & shell scripts, eg. a database backup script every day.
    • You can schedule commands & scripts under any system user, so that commands & scripts are executed with that system user privileges. That is : there is potentially a crontab file for each system user.
    • Crontab format : each line starts with 5 space separated time parts and then the command(s) to execute at the specified time(s):
      • minute (0-59) hour (0-23) day (1-31) month (1-12) weekday (0-6) command
      • 05 02 * * * command : run command 5 minutes past 2 am every day of every month (as well as of every weekday).
      • 05 02 01 * * command : run command 5 minutes past 2 am the first of every month.
      • 05 02 * * 6 command : run command 5 minutes past 2 am every saturday (every day and every month but only saturdays).
      • 05 02 * * * command1 && command2 : executing multiple commands in a sequence (command2 executes then command1 returns).
      • 05 02,14 * * * command executing the command both 5 minutes 2 am and 5 minutes past 2 pm.
      • 05 02 * * 0,6 command : executing the command both sunday & saturday.
      • 05 08-17 * * * command : executing the command 5 minutes past every hour from 8 in the morning to 5 in the evening.
      • 05 17 * * 1-5 command : executing the command 5 minutes past 5 in the evening monday to friday.
    • shell> crontab -l : display the content of the crontab.
    • shell> crontab -e : edit the crontab file for the current user (commands are executed with current user privileges).
    • shell> crontab -u www-data -e : edit the crontab file for the www-data user (commands are executed with www-data privileges).
    • shell> crontab -r : remove the crontab of current user.
    • shell> crontab -l > myCrontabBackupFile : make a copy of the crontab entries eg. for backup purposes.
    • shell> crontab myCrontabBackupFile : load the crontab for current user using a file.
    • shell> service cron status : check if cron is running.
    • shell> service cron restart : restart the crontab daemon.
    • If your commands or scripts are not executing from crontab but you can execute the commands or scripts from the command line, then the problem is typically either :
      • crontab is running under the sh SHELL while you yourself is running under the bash SHELL.
      • crontab does not have the same PATH value as you yourself have.
      Solution :
      1. shell echo $SHELL : display the SHELL you are using, mine is /bin/bash.
      2. shell> echo $PATH : display the PATH variable, mine is /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games.
      3. Add the SHELL and PATH variables to the top of the crontab file :
        1. shell> crontab -e : open crontab in the default editor and insert the following 2 lines at the top :
          • SHELL=/bin/bash : use your own value.
          • PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games : use your own value.

Security

  • chmod :
  • chgrp :
  • chown :
  • ufw : Uncomplicated FireWall : ufw is default firewall configuration tool on Ubuntu (Ubuntu Community UFW)
    The Ubuntu linux kernel includes the netfilter subsystem to decide the fate (acceptance, manipulation or rejection) of network packages. Netfilter is managed by iptables. However iptables is difficult to learn, so many easier to use frontends for iptables have been developed - ufw is such a frontend simplifying creating iptable rules enable us to fast and easy manage network traffic on a server.
    • shell> ufw enable : enables the ufw firewall.
    • shell> ufw disable : disables the ufw firewall, but IPTABLES rules still applies.
    • shell> ufw status : shows whether ufw is enabled or disabled and also shows ufw rules.
    • shell> ufw status verbose : in addition shows default rules and security mods on /etc, /lib, /usr.
    • shell> ufw status numbered : in addition shows a number in front of each rule (which is useful then deleting rules).
    • shell> ufw logging on : enable logging and set log level to low (see man page for more options).
    • shell> ufw logging off : disable logging.
    • shell> ufw logging medium : increases the loglevel from off to medium (possible values are : off, low (default), medium, high, full). Log levels above medium may quickly fill up your harddrive.
    • shell> ufw --dry-run allow 80 : display what would be the result of ufw allow 80 but does not actually implement it.
    • Adding rules :
      • shell> ufw allow 80 : will allow traffic on port 80.
      • shell> ufw allow 80/tcp will allow only TCP traffic on port 80
      • shell> ufw deny 3306 : no traffic allowed on port 3306 (default port for MySQL).
      • shell> ufw deny 3306/udp : no udp traffic allowed on port 3306.
      • Using service names instead of ports : (To use a service name, it must be listed in /etc/services)
        • shell> cat /etc/services : show all the services registered.
        • shell> cat /etc/services | grep http : shows all http relevant services.
        • shell> ufw allow https : will allow traffic on ports used by https (typically on port 443).
        • shell> ufw deny mysql : will deny traffic on ports used by mysql.
      • Using application names instead of ports : (To use an application name it must be listed in /etc/ufw/applications.d/*)
        • shell> ufw app list : display all the programs known to ufw as specified in /etc/ufw/applications.d/* (it is said that these programs have a ufw profile).
        • shell> ufw allow dovecot-common : will allow traffic on all ports as specified in /etc/ufw/applications.d/dovecot-common (which is 4 different ports : pop3, pop3s, imap & imaps).
        • shell> ufw app info dovecot-common : display the ufw profile info for the dovecot-common service.
        • shell> ufw app update dovecot-common : if you have updated the dovecot-common ufw profile, you need to let ufw know it was updated.
        • shell> ufw app update all : reload all ufw profiles.
      • Port ranges :
        Then adding a port range, you MUST specify the protocol and you MUST specify either the from- or the to- IP address
        • shell> ufw allow proto tcp to any port 9101:9103 : Allowing protocol TCP to ANY IP on the server on port range 9101:9103 (these are the Bacula ports).
    • Deleting rules : you simply put a delete before rule name :
      • shell> ufw delete allow 80 : will delete the exact rule "allow 80" (so "allow 80/tcp" would not be deleted, nor would "allow www").
      • shell> ufw delete allow https : will delete the exact rule "allow https".
      • shell> ufw delete deny 3306/udp : will delete the rule "deny 3306/udp" (so tcp on 3306 may still be allowed if there is a rule like "allow 3306" or "allow 3306/tcp").
      • shell> ufw reset : set ufw to factory defaults (deleting all your rules).
    • IP address based rules :
      • shell> ufw allow from 192.168.1.100 : will allow anything on any port from 192.168.1.100.
      • shell> ufw allow from 192.168.1.100 to any port 22 : will allow any traffic on port 22 from 192.168.1.100 to any IP binding on the server.
      • shell> ufw allow from 192.168.1.100 to 192.168.1.4 port 22 : will allow any traffic on port 22 if it comes through the 192.168.1.4 IP binding.
      • shell> ufw allow from 192.168.1.100 to any port 22 proto tcp : will allow only TCP traffic on port 22 from 192.168.1.100.
      • shell> ufw allow from 192.168.1.0/24 : will alow any traffic on any port from IP's on the C class 192.168.1 subnet.
    • Outgoing rules :
      Traffic originating from the server, outgoing traffic, is not considered as dangerous and default it is all allowed, however there can be good reasons to control outgoing traffic as well, here are 3 examples :
      • Malicious code creating reverse shell connections.
      • Malicious code downloading files.
      • Malicious code sending spam mails - also known as mass emailers, especially selfcontained mass emailers can be stopped this way by denying non-approved programs to send email.

      • shell> ufw default deny outgoing : block all outgoing traffic that is not specific allowed.
      • shell> ufw allow out to 8.8.8.8 port 53 : allow requests for domain resolutions, here on google dns server 8.8.8.8 on port 53 (default dns port). Lots of website code relies on domain name resolution, that code will stop working unless you open for your nameservers.
      • shell>
  • clamscan :
    • shell> apt-get install clamav : ClamAV is not default installed.
    • shell> clamscan -V : get the clamscan version.
    • shell> freshclam : update the signature database to be able to identify the newest threats.
    • shell> clamscan -r -i /var/www : scan the /var/www folder recursively (-r) and show only infected files (-i).
    • On errors like :
      • /var/log/clamav/freshclam.log is locked by another process
      • Problem with internal logger (UpdateLogFile = /var/log/clamav/freshclam.log)
      Restart freshclam :
      • shell> /etc/init.d/clamav-freshclam stop : stop the updater service
      • shell> /etc/init.d/clamav-freshclam start : start the updater service

Other

  • xargs : will take the output from eg. one command and for each record in that output execute another command on that record
    • Syntax : input | xargs [options] command : input is typically the output from find, ls or grep which is then piped to xargs that will execute command on each record in the input.
    • shell> ls | grep "old" | xargs rm : will delete all files that have "old" in its filename.
    • shell> grep -lr 'small' * | xargs sed -i 's/small/big/g' : will substitute all occurrences of small with big in all files recursively from current folder.
    • shell> ls | grep "2012" | xargs -n1 -i cp {} dir2 : will copy all files with "2012" in its filename to the dir2 subfolder.
    • shell> ls dir1 | grep "2012" | xargs -I '{}' mv dir1/'{}' dir2/'{}' : moves all files with "2012" in its filename from dir1 to dir2.
  • sed : typically used to replace text within files (manual, comprehensive tutorial, fast tutorial)
    • sed loop through an input stream, typically a file, line by line. For each line :
      1. Current line is read into the sed pattern buffer without any trailing newline character.
      2. Some command is executed on the pattern buffer depending on conditions.
      3. The pattern buffer is printed to output stream, typically screen or file, adding any removed newline character.
      4. The pattern buffer is emptied (and next line is read).
    • Syntax : sed OPTIONS SCRIPT INPUTFILE (options & input file are optional)
    • Most used options : (note that sed regular expressions are greedy and that sed does not support the non-greedy modifier)
      • -e : next parameter is a script (inline expression).
      • -f : next parameter is a script file.
      • -i : in-place editing - the input file is edited in-place instead of writing the editing result to screen or another file.
      • -n : silent - do not print the pattern buffer to output stream (unless using the p flag and a condition match).
    • Commands :
      • s : substitute : sed s/regexp/replacetext/ inputstream (where inputstream is typically a file).
      • d : delete everything within the pattern space
      • g : global (within the pattern space)
      • p : print pattern space (typically used with the -n (slient) option to print only those lines that match a condition)
    • Syntax examples :
      • syntax : sed s/regexp/replacetext/g : substitute (s) regexp with replacetext globally (g).
      • syntax : sed -e '/regexp1/s/regexp2/replacetext/g' : on lines that matches regexp1 substitute (s) regexp2 with replacetext globally (g).
      • syntax : sed -e '/regexp1/,/regexp2/d' : delete (d) from line match regexp1 until and including line match regexp2. Here regexp1 & regexp2 is also called address1 & address2 respectively. This will run globally without the g flag.
    • Command examples :
      • shell> sed -e '1,5d' : delete (d) line 1 and 5.
      • shell> sed -i '/^[ ]*\#/d' dovecot.conf : removes all comment lines from the dovecot.conf file if comments starts with a # (using [ ]* to match any spaces before the comment symbol)
      • shell> sed -i '/^[ ]*$/d' dovecot.conf : removes all empty lines from the dovecot.conf file
      • shell> sed -i 's/abc/($)/ : the $ represents what is matched by the regexp, so here the matched, abc, will be encircled with parantesed.
      • shell> sed -n '/abc/p' : silent (-n) so only lines that have a match will be printed (p).
      • shell> sed '/abc/d' : delete all lines that contains the string abc.
      • shell> sed -e 's/a/A/g' -e 's/b/B/g' : change all lowercase a & b to uppercase A & B.
      • shell> sed '3 s/a/A/g' : change all lowercase a to uppercase A on line 3 (if the condition is a number it means a line number).
      • shell> sed '3,5 s/a/A/g' : change all lowercase a to uppercase A on line 3,4 & 5.
  • awk : (awk user guide) : particularly used for selecting records in a file and perform operations on them.
    • syntax : awk 'program' file1 file2 ... : where program is a series of rules (each rule consisting of one pattern and one action).
    • syntax : awk 'program' file1 file2 ... : start awk and use the 'program' to process the input files (program is enclosed in single quotes to avoid the shell interpret special characters)
  • Debugging :
    • tail : displays the last lines of a file - the most important debugging command, very easy and full of information
      • shell> tail /var/log/mail.log : will display the last 10 lines of the mail.log file.
      • shell> tail -30 /var/log/mail.log : will display the last 30 lines of the mail.log file.
      • shell> tail -f /var/log/mail.log : will display the last few lines of the mail.log file and continue to print new lines from mail.log as they arrive.
    • strace : can trace another programs system calls - helps debugging (while strace produces a huge amount of info, the interesting part tends to be at the bottom)
      • shell> strace ls : output ls system call to standard output (screen).
      • shell> strace -o strace.ls.txt ls : output ls system calls to strace.ls.txt text file.

Appendix : Powerful One-liners

Sometimes complicated tasks can be handled easily by bundling together commands, here I collect the ones I use the most.

  • Replacing text in files :
    • shell> grep -lr "OldText" * | xargs sed -i 's/OldText/NewText/g' : will search (grep) all files (*) recursively (-r) from current folder and list (-l) these containing the text OldText, then pipe (|) the list to xargs and upon each filepath apply the sed command which will substitute (s) all (g) occurrences of OldText with NewText.
    • shell> find . \! -path '*.sh' -type f -exec sed -i -n '1h;1!H;${;g;s/AAAA.*BBBB//g;p}' {} \; : recursively (find does that) search all files of type file (-type f) from current folder (.) except (!) all sh files (-path '*.sh') and execute (-exec) sed to replace all text starting with AAAA and ending with BBBB even the text is over multiple lines.
  • Deleting messages in Postfix email queue :
    • shell> mailq | grep "SomeUsername" | awk '{print substr($1,0,12)}' | postsuper -d - : list all messages in Postfix queue and select only those of a specific user but print only the first 12 chars from the message identifier and then delete the messages using postsuper.
  • Remove k1r4, emp3ror malware from php files : (this is not enough to remove the full malware attack, this only removes the malware injection in php files)
    • shell> find . -name \*.php | xargs sed -i '/^<?[ \t]*\$s=substr/s/.*/<?PHP/' : find all php files recursively from current directory with relative path and execute the sed script on each file.
    • shell> grep -lr " " * | grep ".php" | xargs sed -i '/^<?[ \t]*\$s=substr/s/.*/<?PHP/' : here using grep instead of find (note that current directory is not searched).


click to top