SSH or Secure Shell in simple terms is a way by which a person can remotely access another user on other system but only in command line i.e. non-GUI mode. In more technical terms, when we ssh on to other user on some other system and run commands on that machine, it actually creates a pseudo-terminal and attaches it to the login shell of the user logged in.
When we log out of the session or the session times out after being idle for quite some time, the SIGHUP signal is send to the pseudo-terminal and all the jobs that have been run on that terminal, even the jobs that have their parent jobs being initiated on the pseudo-terminal are also sent the SIGHUP signal and are forced to terminate.
Only the jobs that have been configured to ignore this signal are the ones that survive the session termination. On Linux systems, we can have many ways to make these jobs running on the remote server or any machine even after user logout and session termination.
Understand Processes on Linux
Normal Process
Normal processes are those which have life span of a session. They are started during the session as foreground processes and end up in certain time span or when the session gets logged out. These processes have their owner as any of the valid user of the system, including root.
Orphan Process
Orphan processes are those which initially had a parent which created the process but after some time, the parent process unintentionally died or crashed, making init to be the parent of that process. Such processes have init as their immediate parent which waits on these processes until they die or end up.
Daemon Process
These are some intentionally orphaned processes, such processes which are intentionally left running on the system are termed as daemon or intentionally orphaned processes. They are usually long-running processes which are once initiated and then detached from any controlling terminal so that they can run in background till they do not get completed, or end up throwing an error. Parent of such processes intentionally dies making child execute in background.
Techniques to Keep SSH Session Running After Disconnection
There can be various ways to leave ssh sessions running after disconnection as described below:
1. Using screen Command to Keep SSH Sessions Running
screen is a text Window Manager for Linux which allows the user to manage multiple terminal sessions at the same time, switching between sessions, session logging for the running sessions on screen, and even resuming the session at any time we desire without worrying about the session being logged out or terminal being closed.
screen sessions can be started and then detached from the controlling terminal leaving them running in the background and then be resumed at any time and even at any place. Just you need to start your session on the screen and when you want, detach it from pseudo-terminal (or the controlling terminal) and logout. When you feel, you can re-login and resume the session.
Starting a screen Session
After typing ‘screen’ command, you will be in a new screen session, within this session you can create new windows, traverse between windows, lock the screen, and do much more stuff which you can do on a normal terminal.
$ screen
Detaching a Screen
Just when you want to log out of the remote session, but you want to keep the session you created on that machine alive, then just what you need to do is detach the screen from the terminal so that it has no controlling terminal left. After doing this, you can safely logout.
To detach a screen from the remote terminal, just press “Ctrl+a” immediately followed by “d” and you will be back to the terminal seeing the message that the Screen is detached. Now you can safely logout and your session will be left alive.
Resuming Detached Screen Session
If you want to Resume a detached screen session which you left before logging out, just re-login to remote terminal again and type “screen -r” in case if only one screen is opened, and if multiple screen sessions are opened run “screen -r <pid.tty.host>”.
$ screen -r $ screen -r <pid.tty.host>
2. Using Tmux (Terminal Multiplexer) to Keep SSH Sessions Running
Tmux is another software which is created to be a replacement for the screen. It has most of the capabilities of screen, with few additional capabilities which make it more powerful than screen.
It allows, apart from all options offered by screen, splitting panes horizontally or vertically between multiple windows, resizing window panes, session activity monitoring, scripting using command line mode etc. Due to these features of tmux, it has been enjoying wide adoption by nearly all Unix distributions and even it has been included in the base system of OpenBSD.
Start a Tmux Session
After doing ssh on the remote host and typing tmux, you will enter into a new session with a new window opening in front of you, wherein you can do anything you do on a normal terminal.
$ tmux
After performing your operations on the terminal, you can detach that session from the controlling terminal so that it goes into the background and you can safely logout.
Detach Tmux Session from Terminal
Either you can run “tmux detach” on running tmux session or you can use the shortcut (Ctrl+b then d). After this, your current session will be detached and you will come back to your terminal from where you can log out safely.
$ tmux detach
Resuming the Closed Tmux Session
To re-open the session which you detached and left as is when you logged out of the system, just re-login to the remote machine and type “tmux attach” to reattach to the closed session and it will still be there and running.
$ tmux attach
3. Using nohup command to Keep Running SSH Sessions
If you are not that familiar with screen or tmux, you can use nohup and send your long-running command to the background so that you can continue while the command will keep on executing in background. After that, you can safely log out.
With nohup command we tell the process to ignore the SIGHUP signal which is sent by ssh session on termination, thus making the command persist even after session logout. On session logout, the command is detached from controlling terminal and keeps on running in the background as a daemon process.
Executing a command using nohup in background
Here, is a simple scenario wherein, we have run find command to search for files in background on ssh session using nohup, after which the task was sent to the background with prompt returning immediately giving PID and job ID of the process ([JOBID] PID).
# nohup find / -type f $gt; files_in_system.out 2>1 &
Resuming the session to view if a job is still running
When you re-login again, you can check the status of the command, bring it back to the foreground using 'fg %JOBID' to monitor its progress and so on. Below, the output shows that the job was completed as it doesn’t show on re-login, and has given the output which is displayed.
# fg %JOBID
4. Using disown Command to Keep SSH Sessions Running
Another elegant way of letting your command or a single task run in the background and remain alive even after session logout or disconnection is by using disown.
Disown, removes the job from the process job list of the system, so the process is shielded from being killed during session disconnection as it won’t receive SIGHUP by the shell when you logout.
The disadvantage of this method is that it should be used only for the jobs that do not need any input from the stdin and neither need to write to stdout, unless you specifically redirect jobs input and output because when a job will try to interact with stdin or stdout, it will halt.
Executing a command using disown in background
Below, we sent ping command to the background so that ut keeps on running and gets removed from the job list. As seen, the job was first suspended, after which it was still in the job list as Process ID: 15368.
$ ping rootadminz.com > pingout & $ jobs -l $ diswon -h %1 $ ps -ef | grep ping
After that disown signal was passed to the job, and it was removed from job list, though was still running in the background. The job would still be running when you would re-login to the remote server as seen below.
$ ps -ef | grep ping
5. Using setsid Command to Put SSH Sessions Running
Another utility to achieve the required behaviour is setsid. Nohup has a disadvantage in the sense that the process group of the process remains the same so the process running with nohup is vulnerable to any signal sent to the whole process group (like Ctrl + C).
setsid on other hand allocates a new process group to the process being executed and hence, the process created is totally in a newly allocated process group and can execute safely without fear of being killed even after session logout.
Execute any command using setsid
Here, it shows that the process ‘sleep 10m’ has been detached from the controlling terminal, since the time it has been created.
$ setsid sleep 10m $ ps -ef | grep sleep
Now, when you would re-login the session, you will still find this process running.
$ ps -ef | grep [s]leep
