Faq unix
C&CZ's Not Frequently Enough asked questions
with answers!
Last modified: 2018-03-21
For questions: <>
Further reference: http://wiki.science.ru.nl/cncz/
About this FAQ
https://wiki.science.ru.nl/cncz/index.php?title=Faq_unix
Contributors
Patrick Atoon | |
Kees Keijzers | |
Jos Alsters | |
Peter van Campen | |
Simon Oosthoek |
Linux shells
What is a shell?
The shell is a program that interprets the user's commands and executes them on the Operating System. There are many kinds of shells, which have different interpreted languages in which to write commands.
A shell can be used interactively, a human user typing commands on a terminal prompt, or in a shellscript. The shellscript is also called a batch file. A shell script that asks for human input can only be run from an interactive shell.
Examples of shells
On the linux machines you can use the shells:
/bin/sh :
the default "Bourne" compatible shell, on Ubuntu, this is /bin/dash/bin/dash:
a light-weight Bourne/POSIX compatible shell./bin/bash:
the Bourne Again Shell. A full fledged linux shell and the default./bin/zsh :
the last shell you'll ever need, very configurable/bin/tcsh:
a csh compatible shell. We strongly advise not to use this for scripts!
Note that there are shells for microsoft windows, e.g. cmd.exe and powershell. Recently the bash shell is becoming available on windows as well.
What is man
Most commands have a manual page, you can invoke the manual page by giving the command: man <command>
in an interactive shell.
prompt> man bash
will give you the manual page of the bash shell. Manual pages have a few common headers which you can find in nearly all man pages. Once you get used to this, you can quickly find out how to use the command more effectively.
prompt> man -k something
Will search for the keyword something
in the summary of each man-page on the system. This may help you find the right command to do what you want, without knowing the exact command name. This is not a fool-proof solution and often a search engine will be more effective!
Typical Linux commandline
If you work in a terminal (window) you can give commands via the shell. The shell interprets your command, most importantly it splits the line you provide into words separated by whitespace.
prompt> COMMAND [arg1] [arg2] ... <enter>
prompt> | printed by the shell, usually user@host $ , but it can be customised!
|
COMMAND | the command you want to execute (you type it) |
[arg1] | argument for the command |
[arg2] | another argument for the command |
... | further arguments for the command |
<enter>
|
You press the return key, or enter, to "give" the command to the shell. After this, your line will be interpreted and executed. |
examples
ls -l
|
long (-l) listing of all files and directories in the current working directory |
ls -a -l
|
long (-l) listing of all files and directories in the current working directory, also show hidden entries (-a) |
ls -l /tmp
|
long (-l) listing of all files and directories in /tmp |
ls -l /bin/bash
|
long (-l) listing of the file /bin/bash |
man man
|
show manual page of the man command
|
What is the environment?
The environment is a collection of shell variables that is passed to the commands you type. Check what is in the environment with the command:
$ env
In a Bourne shell, a variable can be set like this:
prompt> MYVAR=something
to show what the variable holds (Note the $ sign):
prompt> echo $MYVAR
To make it part of the environment:
prompt> export MYVAR
Variables that are usually present are
HOME
|
your home directory, used when you give the command cd without arguments to get to your home directory
|
PATH
|
the list of directories searched for the name of the command you give without a / in it. NB Do not put the current directory "." at the front of the path! |
EDITOR
|
The editor command used when you invoke a command that uses an editor to let you modify files. Typical editor commands are vi , nano , joe
|
PAGER
|
The pager command used when e.g. a manual page doesn't fit on a screen. Typical pagers are less , more
|
And many more...
What is input/output redirection
Whenever you start a command, the shell opens three "files" by default. These are stdin, stdout and stderr, also known by their numbers, 0, 1 and 2.
The (bourne) shell has the ability to redirect the flow of these files to somewhere else.
Without redirection, the three files are all connected to the terminal, so stdin to the keyboard device, stdout and stderr to the terminal window.
If you command the shell to redirect the output to a file, you can store the output of the command to a file, instead of seeing it on the window where you typed the command.
examples
ls > mylisting
|
output redirection creates or overwrites the file mylisting with the output of the ls command
|
ls >> mylisting
|
output redirection creates or appends the file mylisting with the output of the ls command
|
ls some-file 2>myerrors
|
standard error redirection creates or overwrites the file myerrors with the error message produced by the ls command
|
wc -l <$HOME/.bash_history
|
using input redirection count the number of lines in the file $HOME/.bash_history. Try the same command with and without the <! With the < the stdin is redirected from the file $HOME/.bash_history to the |
Instead of redirecting to a file, it is also possible to redirect the output of a command to the stdin of another command. This is called a pipeline. Another way to look at this is to see the command after the | symbol as a filter. Multiple filters can be used in sequence to massage the output of a command into a desirable format or selection.
prompt> cat /etc/shells | wc -l
How can I run multiple commands from one commandline?
Normally, commands are separated by a newline <enter>, however, you can also use a ; semicolon to seperate commands:
prompt> ls; sleep 10; echo bla
The above will list the directory, wait 10 seconds, then print bla
How can I run a command non-interactively and without having to wait for it to finish?
A common problem is that you want to run a command that takes a long time to finish and can do so without needing any input.
You can start a command to run in the background by adding a &
prompt> ls -l &
As you will see when you try this, it is still connected to the terminal. If you prepend the command nohup
you can disconnect it and even logout of the machine you run the command on. The output will typically end up in a file called nohup.out.
Alternatively: screen
If you want to run the program and be able to provide input or check on its progress now and then, you can run a virtual terminal program, like screen or tmux, these will continue running when you disconnect and log out, later you can reconnect to see how it's going. See man screen
or man tmux
.
What is filename expansion or wildcards?
The shell has some tools to save you typing long filenames. E.g. when you don't want to list all files in a directory:
ls -d ?
|
list all files and directories with a name of one character |
ls -d ???
|
list all files and directories with a name of three character |
ls -d a*
|
list all files and directories with a name starting with "a" and any number of unspecified characters after it |
ls -d .*
|
list all files and directories with a name of three character |
ls -d *[0-9]
|
list all files and directories ending with a number |
ls -d *[a-z]
|
list all files and directories ending with a lowercase letter |
ls -d *[A-Z0-9]
|
list all files and directories ending with an uppercase letter or a number |
ls -d ~/a*
|
list all files and directories in my homedirectory starting with the letter a |
ls -d *\** or ls -d *'*'*
|
list all files and directories with names containing a real *. Use the \' or single quotes to escape the character following it. i.e. prevent the shell from using the * for wildcard expansion. |
characters to escape | < > [ ] ( ) { } * & ! ~ ; "space" "quotes" and many more... |
The option -d of ls makes sure to list the entry itself and not the contents of the directory by that name.
More information about wildcards can be found in the bash manual.
NB A special case is when you escape the newline or <enter> character. The shell will not start interpreting the command and you can add more to your command line on the next line.
prompt> ls \<enter> -ltr<enter> ... prompt>
Will be the same as "ls -ltr
"
How do I customise my interactive shell session?
In this section we assume you are using the Bourne Again SHell (bash). You can configure your login shell via DIY. (If you are running this on your own Linux machine, you can use the chsh
command to change the login shell.)
What happens when I login?
When you login, bash checks a list of possible locations for configuration files (which are full of shell commands).
Bash looks for the following files
/etc/profile ~/.bash_profile ~/.bash_login ~/.profile.
When you start a new interactive shell on the same machine, bash looks for other files:
/etc/bash.bashrc ~/.bashrc
People often want to have the same settings in all shells, so a lot of times you see the following in ~/.profile:
if [ -f ~/.bashrc ]; then source ~/.bashrc fi
And all configuration can exist in ~/.bashrc
In either the system or personal bashrc files, variables and aliases are set to preference:
- PATH is set to find systemwide programs, you can expand this search path to include directories in your homedir
- PS1 is set to a useful prompt
- PAGER
How can I change the way my prompt looks?
The variable PS1 is the prompt you normally see when you can enter a command to the shell. Various ways exist to make this more useful.
- Set PS1 directly
- Set PROMPT_COMMAND to something that sets PS1
To make it permanent, define PS1 or PROMPT_COMMAND in your ~/.bashrc file.
examples
PS1='\u@\h:\w\$ ' PS1='\u@\h:\w [$?]\$ ' PS1='[\e[32m]\u@\h:[\e[33m]\w[\e[32m]$[\e[0m] '
Other customizations
Scripting
Often you want to be able to reproduce a sequence of commands or even create a small program to do a lot of work in the shell.
For this purpose, you can create a batch script, shell file, or shell script. We recommend you do this in the bourne shell (sh) or (bash). Using (t)csh for shell scripts is not recommended, see Csh Programming Considered Harmful.
Basic shell script
echo start of script ls -l echo end of script
If you put the above 3 lines in a file myscript
, you can call bash myscript
and run the commands in the file.
In order to make the file a command, you can make the file executable by giving the command: chmod a+x myscript
Then you can call the script using ./myscript
Choose interpreter #! hashbang
How to provide multiple lines of input: Here Documents
How to use the output of command as parameters: Command substitution
prompt> CMD-1 `CMD-2` or prompt> CMD-1 $(CMD-2)
Call CMD-1 with arguments created from the output of CMD-2
example
Rename a file to have a different extension or a date in the name.
- Get the date in a particular format (see man date)
prompt> date "+%Y%m%d-%H:%M" 20180327-12:41
- Get the first part of the name
prompt> basename pietje.jpg .jpg pietje
- Use both commands to create a new name for move (rename)
prompt> mv -v pietje.jpg $(basename pietje.jpg .jpg)-$(date "+%Y%m%d-%H:%M").jpg 'pietje.jpg' -> 'pietje-20180327-12:45.jpg'
- The `CMD-2` will be executed before `CMD-1`.
- Command substitution can be invoked using `CMD-2` or $(CMD-2)
- The advantage of $() is that it can be nested $(CMD-2 $(CMD-3))
TODO
1.17 Hoe vindt een shell een commando: -------------------------------------- De shell kan een commando vinden aan de hand van: - een absolute naam, een naam die met een '/' begint: wn4> /usr/ucb/vi wn4> /usr/local/datasearch/bin/telgids - een relatieve naam, een naam die niet begint met een '/', maar wel een '/' bevat: wn4> bin/testprogramma wn4> ../petervc/bin/backup-photon - een commando of programma naam. Om te weten wat er dan opgestart moet worden, gebruikt de shell de waarde va de PATH (sh en bash) of path (csh) environment variabele. Deze moet een lijst van directories bevatten waarin de naam van het betreffende commando opgezocht kan worden: wn4> vi wn4> telgids wn4> testprogramma De lijst wordt in volgorde afgewerkt en het eerst gevonden programma wordt opgestart. Omdat er een Unix programma met de naam 'test' bestaat, en wanneer je zelf een programma maakt met ook de naam 'test', bepaalt de volgorde van de directories in de path variabele welk programma gebruikt zal worden! De huidige directory ('.') moet ook in het path staan wil een commando in de huidige directorie gevonden worden (dit in tegenstelling tot MSDOS bijv.) Je kunt het path meestal het beste veranderen door het aan de voor- of achterzijde uit te breiden met de gewenste directorie(s): Voor sh/bash: wn4> PATH=$PATH:$HOME/bin Voor csh: wn4> path=(/usr/local/X11R5/bin $path) 2.7 Wat is een mailinglist? ----------------------------- Een mailinglist is een soort email-discussie, die per email wordt verstuurd naar de mensen die op de lijst staan. Je kunt je opgeven voor zo'n lijst bij degene die de lijst bijhoudt (de administrator). Iedereen die deelneemt aan de disussie stuurt zijn bericht naar de administrator, en die zorgt ervoor dat iedereen die op de lijst staat, ook dat bericht toegestuurd krijgt. Dit betekent vaak dat je bergen post in je mailbox zult krijgen. Hoe vind je uit welke mailinglists er zijn? Welke lists er precies allemaal zijn is niet uit te vinden, meestal kom je ze tegen `van horen zeggen'. Als je door je nieuws bladert lees je soms iets over zo'n mailinglist. Daarnaast zijn er de nieuwsgroepen die onder bit.list-serv te vinden zijn, dit zijn kopieen van mailinglists. 3.2 Hoe kan ik mijn bestanden ontoegankelijk maken voor anderen? ---------------------------------------------------------------- Daarvoor is zowel op Unix als op MSDOS met PC-NFS het 'chmod' commando. (Het bekijken van de protecties gaat met 'ls -lg' ook op MSDOS!) Chmod heeft een aantal varianten om de 'bitjes' te zetten: H:> ls -lg bestandje -rwxr-xr-x groep eigenaar 112 bestandje ||| -> eigenaar ||| -> groep ||| -> wereld H:> Dus de protectie van directories en bestanden worden steeds door 3 'bitjes' gegeven: rwx met de 'waardes' 4,2,1 Het chmod commando wordt als volgt gebruikt: wn4> chmod u=rwx,g=rx,o= bestandje wn4> ls -lg bestandje -rwxr-x--- groep eigenaar 112 bestandje Nu is het bestand alleen schrijfbaar voor de eigenaar, en lees en executeerbaar voor de eigenaar en de groep. De rest mag niets. Ander voorbeeld: H:> chmod g=,o= geheim H:> ls -lg geheim -rwx------ groep eigenaar 12 geheim Nu kan alleen de eigenaar het bestand lezen, wijzigen en executeren. Alle anderen kunnen niets. Ook kun je in chmod expliciet de bitjes opgeven, in octale notatie, zoiets als: wn4> chmod 700 bestandje wn4> ls -lg bestandje -rwx------ groep eigenaar 112 bestandje Hiermee is het bestand alleen nog te lezen, schrijven en executeren door de eigenaar. wn4> chmod 444 bestandje wn4> ls -lg bestandje -r--r--r-- groep eigenaar 112 bestandje Nu is het bestand nog alleen nog te lezen door iedereen. Verder zie 'man chmod' en 'man ls' op Unix, voor PC-NFS staat e.e.a. in de manual. Bijbehorend is de umask variabele: deze maskeert bitjes die niet vanzelf aan moeten gaan voor nieuwe bestanden: Als de umask 022 is dan zullen de bitjes van de eigenaar niet beinvloed worden, het '2' bitje (schrijven) voor de groep en de wereld wordt afgezet voor een nieuw bestand. Ook dit is weer waar voor zowel MSDOS als Unix. Onder MSDOS met PC-NFS zet men het umask als volgt: C:> net umask 077 Dit zet alle toegang tot nieuwe bestanden voor anderen dan de eigenaar af. Onder Unix gaat het bijna identiek: wn4> umask 022 Nu zijn voor alle nieuwe bestanden de 'schrijf bitjes' uit. Tenslotte: de protectie van de directorie is ook belangrijk! Het verwijderen van een file is een directorie operatie! Dus al heb ik op de file zelf geen rechten, als ik wel mag schrijven op de directorie kan ik het bestand gewoon verwijderen! 3.3 Help, ik ben een file op Unix kwijt! Kan ik die terug krijgen? -------------------------------------------------------------------- Zie http://www.sci.kun.nl/cncz/procedures/backups.html