Category: Linux

Stop Entering Your SSH Passphrase All The Time

Stop Entering Your SSH Passphrase All The Time

You use SSH every day. You check out Git repositories, connect to servers and you Rsync a backup of your priceless home directory to remote machines for archiving and syncing. Because you know the importance of security you never use password authentication but an at least 4096 bit long RSA public-private keypair, encrypted by a passphrase only you know.

You are great. But you also waste time and patience, because you always enter the same single passphrase every time your computer wants to use SSH. Is it more secure, if you need to enter it every time? Probably not. The chance of someone figuring out (or keylogging) your passphrase is high enough, but I also heard of people who keep their passphrase in the clipboard for fast entry. Wow.

There is a simple, secure and elegant way to save your passphrase in-memory for your whole working session. Authentication agents like Pageant (Windows) or ssh-agent (OSX / Linux) can safely store your password and provide it to the SSH application when it requests a passphrase for your key.

„No Talk. Just Solution.“

Here is how I use it on my Linux and OSX machines. It requires you to install something called an SSH Agent Frontend – so basically a software that in turn talks to the ssh-agent – but in turn it provides a very elegant solution that manages the ssh agent, gpg agents and works even outside of environment scope (for cron jobs, etc.).

Using the frontend is optional and you can use the plain ssh-agent if you make sure to check for, inherit and run ssh-agent processes when needed. I strongly recommend using Keychain, though.

Installation & Configuration for Linux, OSX with Bash

I assume you already have installed SSH together with an SSH Agent, which is the case on most systems. I also assume that you use the bash or can transfer this article to other shells of your choice.

  • Install keychain by Funtoobrew install keychain yum install keychain apt-get install keychain pacman -Syu keychain (Or download and install manually from
  • Edit your ~/.bashrc and append the following line:eval `keychain --agents ssh --eval id_rsa` (Where ssh is the agent you want to use and id_rsa is a list of paths to your private key(s))
  • Re-open your shell or terminal emulator.

Keychain will ask you to enter your passphrase once and save it to the ssh-agent. You no longer need to enter your passphrase

Use keychain --stop all to stop all agents. The next time you start bash or your terminal emulator (and effectively keychain) you will be asked to unlock your private key with your passphrase again.

Installation & Configuration for Windows

Because Windows has no stable built-in SSH client we use an open-source tool called PuTTY. It comes bundled with an authentication agent called Pageantwhich works the same way as the OpenSSH SSH agent for Linux and OSX.

  • Install PuTTY by Simon Tathamchoco install putty (Or download and install manually from
  • Open it and select Add... to load and unlock your private keys with your passphrases.
  • Use PuTTY like you would normally. It automatically detects the running Pageant authentication agent.

See Also

iPXE Boot With Shoelaces

iPXE Boot With Shoelaces

Hey guys,
today I’d like to tell you about Shoelaces from Thousandeyes. Shoelaces may be used to automate Server installation with iPXE Boot.
If you are already familiar with the basics about PXE, iPXE and how to set up the environment you can go ahead to Shoelaces Setup.


Before I dig into the details I prepared an Image to show you the Software Stack and also the whole server set up.

Software Stack and Server Setup

All commands are executed on a server named pxe-server.local with IP running on CentOS 7.4.


The DHCP Server is the first line of the stack. As soon as the Client boot in PXE Mode, it requests an IP Address and receives an IP address with the command where it can find the Image to boot from.
I prepared an example configuration file below.

[email protected]:[~]: cat /etc/dhcp/dhcpd.conf
subnet netmask {
  option subnet-mask;
  option routers;
  option domain-name-servers;
  if exists user-class and option user-class = "iPXE" {
    filename "${netX/mac:hexhyp}";
  } else {
    filename "undionly.kpxe";

As you can see the DHCP Server manages the IP Range and also sends the next-server option which tells the client to boot from (TFTP Server) option. If the Client supports iPXE it boots from${netX/mac:hexhyp} which is Shoelaces. If the client doesn’t support iPXE it boots a microkernel which supports iPXE.

You can test your DHCP Settings with nmap

[email protected]:[~]: nmap --script broadcast-dhcp-discover
Starting Nmap 6.40 ( ) at 2018-08-06 12:58 UTC
Pre-scan script results:
| broadcast-dhcp-discover:
|   IP Offered:
|   DHCP Message Type: DHCPOFFER
|   Server Identifier:
|   IP Address Lease Time: 0 days, 0:05:00
|   Subnet Mask:
|   Router:
|_  Domain Name Server:
WARNING: No targets were specified, so 0 hosts scanned.
Nmap done: 0 IP addresses (0 hosts up) scanned in 0.07 seconds


The TFTP Server is needed to serve the undionly.kpxe when the client doesn’t support iPXE.

To “install” the microkernel you need to download the file from and save it into /var/lib/tftpboot.

[email protected]:[~]: wget -P /var/lib/tftpboot


You need Xinetd to actually serve the microkernel.

Here is the config file we used to glue tftp and Xinetd together:

[email protected]:[~]: cat /etc/xinetd.d/tftp
    socket_type     = dgram
    protocol        = udp
    wait            = yes
    user            = root
    server          = /usr/sbin/in.tftpd
    server_args     = -s /var/lib/tftpboot
    disable         = no
    per_source      = 11
    cps             = 100 2
    flags           = IPv4

As soon as you created the config file you need to restart xinetd and check if the tftp module is running.

[email protected]:[~]: systemctl status xinetd.service
● xinetd.service - Xinetd A Powerful Replacement For Inetd
   Loaded: loaded (/usr/lib/systemd/system/xinetd.service; enabled; vendor preset: enabled)
   Active: active (running) since Mo 2018-08-11 15:19:41 UTC; 3s ago
  Process: 8060 ExecStart=/usr/sbin/xinetd -stayalive -pidfile /var/run/ $EXTRAOPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 8061 (xinetd)
   CGroup: /system.slice/xinetd.service
           └─8061 /usr/sbin/xinetd -stayalive -pidfile /var/run/
Aug 11 15:19:41 pxe-server xinetd[8061]: removing daytime
Aug 11 15:19:41 pxe-server xinetd[8061]: removing discard
Aug 11 15:19:41 pxe-server xinetd[8061]: removing discard
Aug 11 15:19:41 pxe-server xinetd[8061]: removing echo
Aug 11 15:19:41 pxe-server xinetd[8061]: removing echo
Aug 11 15:19:41 pxe-server xinetd[8061]: removing tcpmux
Aug 11 15:19:41 pxe-server xinetd[8061]: removing time
Aug 11 15:19:41 pxe-server xinetd[8061]: removing time
Aug 11 15:19:41 pxe-server xinetd[8061]: xinetd Version 2.3.15 started with libwrap loadavg labeled-networking options compiled in.
Aug 11 15:19:41 pxe-server xinetd[8061]: Started working: 1 available service
[email protected]:[~]: netstat -tulpen | grep xinetd
udp        0      0    *                           0          235681     8061/xinetd

When you execute the status command at the end of the log should be at least ‘1’ available service printed. The Port 69/UDP should be open as well.


Now we’re running the basic iPXE Environment. At the moment we are able to boot a client with PXE boot, obtain an IP Address and boot the iPXE microkernel if the client doesn’t support iPXE.

But what happens if the client boot with this line (which we configured in the dhcpd section)?${netX/mac:hexhyp}
Before answering the question I’ll guide you through the shoelaces installation.


Shoelaces is a program written in Go Lang. Therefore we need to install go at first.

# Determine your needed go version at first. I need the amd64 binary which is version 1.10.3 at the moment.
[email protected]:[~]: wget
[email protected]:[~]: tar -xvf go1.10.3.linux-amd64.tar.gz
[email protected]:[~]: mv go /usr/local
[email protected]:[~]: echo "export GOROOT=/usr/local/go" >> /etc/bashrc
[email protected]:[~]: echo "export GOPATH=/opt/go" >> /etc/bashrc
[email protected]:[~]: echo "export PATH=$GOPATH/bin:$GOROOT/bin:$PATH" >> /etc/bashrc
[email protected]:[~]: src /etc/bashrc

To install shoelaces you need to get the repository with go get then you need to build the shoelaces binary.

[email protected]:[~]: go get
[email protected]:[~]: cd $GOPATH/src/
[email protected]:[~]: go build


Currently, we’re only able to start the service on demand with a command.

[email protected]:[~]: cd $GOPATH/src/
[email protected]:[~]: ./shoelaces -config configs/shoelaces.conf

But if you want the process permanently running you need to create a systemd file.
Luckily I already created one which needs a user to be created.

[email protected]:[~]: useradd shoelaces
[email protected]:[~]: cat /lib/systemd/system/shoelaces.service


# Restart Policy

# Execution
ExecStartPre=/bin/chown -R shoelaces /opt/go/src/
ExecStartPre=/bin/chmod u+x /opt/go/src/
ExecStart=/opt/go/src/ -config configs/shoelaces.conf

# Environment

# Logging


After you added the content to the file /lib/systemd/system/shoelaces.service execute systemctl daemon-reload; systemctl start shoelaces.service; systemctl status shoelaces.service to start the service and check the status. You can always take a look into logs if you execute journalctl -f -u shoelaces.service.


I guess you already noticed the only missing part. Configuring Shoelaces.

It’s straight forward and fairly easy. Just go into the shoelaces directory and take a look at the default config file.

[email protected]:[~]: cd $GOPATH/src/
[email protected]:[/opt/go/src/]: cat configs/shoelaces.conf

For production I changed the config file to the following one …

[email protected]:[/opt/go/src/]: cat configs/shoelaces.conf

… and moved configs/example-templates-configs to configs/templates-configs.


Recently we changed the config file. Therefore you need to restart your service.

Now open your shoelaces URL (http://<domain>:<port>) with your favourite browser. If everything went well you should see the following website. If not, take a look into the logs what’s wrong with shoelaces.

Shoelaces Startup Image

Deep Dive

Back to the question before the shoelaces installation.

But what happens if the client boot with this line (which we configured in the dhcpd section)?${netX/mac:hexhyp}

The client connects to shoelaces and kept in a waiting loop until the user executes a “Boot!” command or shoelaces is able to bootstrap the server with ‘mappings’. This state looks like this:

Server Registered

As soon as you click on ‘Select an iPXE script’ you are able to choose an iPXE script located in $GOPATH/src/

I clicked on centos.ipxe and filled out both arguments. If you check $GOPATH/src/ there are two strings which look like {{.<STRING>}}. These strings are the arguments shown in the UI.

[email protected]:[~]: cat $GOPATH/src/
{{define "centos.ipxe" -}}
set hostname {{.hostname}}
set release {{.release}}
set base${release}/os/x86_64

echo This automatically overwrites data!
echo CentOS ${release}
echo Installing ${hostname}

kernel ${base}/images/pxeboot/vmlinuz initrd=initrd.img repo=${base} ks=http://{{.baseURL}}/configs/centos.ks?hostname=$hostname&release=$release
initrd ${base}/images/pxeboot/initrd.img

As soon as I click on the big red button “Boot!” the client leaves the waiting loop and starts to execute the iPXE script. It depends on your kickstart file how long it takes until your client is back again but normally it should’nt take longer than 10 minutes.


Shoelaces is a very small footprint software written in go which supports bootstrapping a huge amount of server. Even in a small environment, it helps a lot. Take a look at their Github Project Site if you want to discover all further features.

For example you may use the mapping feature if you want to bootstrap server automatically based on IP or Hostname pattern.

Leave a comment or contribute to the project if you like the software.

Save your SD cards: Raspberry Pi on a network file system

Save your SD cards: Raspberry Pi on a network file system

If you work a lot with your Raspberry Pi, you probably have burned one or another SD card. While SD flash cards are great for storing photos or music, they suck at being written to very often. This is exactly the case if you run a whole operating system on them. Hundreds of thousands of read and write operations wear your SD cards out and if you do not have a solid backup concept you have probably lost some data, too.

In my home, a couple of systems are running on the cheap, embedded hardware by the Raspberry Pi Foundation because they are both small and power-saving at a very affordable price point. And so are the accessories like cases and add-on-boards. At the time of writing, these are the jobs that are done by my Raspberries:

  • SSH server (as gateway and tunnel from the internet into the LAN)
  • Monitoring
  • OpenHAB smart home controller
  • TV Server
  • Backup
  • Ambilight


When living alone, having all this run on SD card-based Raspberries is fine and you know what to do when something breaks. As soon as you have a partner to share your home with, you probably need to agree on an SLA to ensure a high WAF:

SLA (Service Level Agreement):

If you want to have a smart home controlled by a computer, you have a very short acceptable “TTR” if something breaks. Our Time-To-Recover is usually as short as 1 day. Honestly, it is more like same-day. Do not go to sleep until it is fixed.

WAF (Wife Acceptance Factor):

If you stick to the SLA, you are free to do all this crazy stuff; your partner accepts it and maybe even likes it.


And that is basically the reason I am writing this. It is completely OK if the ambilight stops working, it is also acceptable if the backups do not run for some days. But if the smart home controller fails, … well, you better have a backup plan. You do not want to deal with someone who spent two hours trying to turn the lights up or the television on and ultimately failed because it simply does not work anymore without the central controlling instance.

So we should reduce these outages to a minimum:

The Objective

  • Increase device uptime / reduce device outages
  • save data on a central storage for easy backups or snapshots


There are so many other things that can fail. The processor may overheat and hang, forcing a reboot by the watchdog, the power consumption could be too high and disable the USB bus (including the Ethernet Port), the cheap phone charger you use for powering may fail as well, but the single biggest problem is – without a doubt – the SD card.
So today I am showing you how simple it is to boot and run your Pi completely from the network.


The Requirements

  • always-on network storage with NFS server software
  • stable network connection to the storage (wired whenever possible)
  • 16+ MB SD card for the Raspberry Pi bootloader and config file that points to your network storage
    (Note: Starting with the Raspberry Pi 3 there are ways to modify the internal bootloader to boot from network without an SD card. I am not going to cover this here; this method will work with all the Raspberry Pi versions)
  • 3 beers of time

Everything I do is based on a Linux computer (I use Arch Linux and Manjaro) but you can also do this on OS X and maybe on Windows with a lot of tools. I really recommend you to use a Linux computer or a linux VM.

The Network Storage

You need a network storage capable of running an nfs-server. You can even use a Raspberry Pi as sever but that would not really solve the problem here, would it? 😉

Ensure the NFS service is globally enabled

Most commercial NAS systems support NFS and so does my Seagate BlackArmor NAS 4000. I highly recommend to create a seperate share for the filesystems of your Raspberries to be able to add stricter permissions later.

You may need to enable the NFS service first, because a lot of people do not need it and use CIFS only.

Create a share where you will put the filesystems for your Raspis

The resulting config, that my NAS generates, looks like this:

~ $ grep rootfs /etc/exports
/nfs/rootfs *(rw,async,insecure,no_root_squash,no_subtree_check)

Please note that the options rw, no_root_squash and no_subtree_check are obligatory for what we want to achieve. For a detailed description of all possible options, please consult the man page on your system or here.


The Extraction Of The SD Card Image Content

You have your NFS Server up and running and configured.

I already stumbled across some distributions of Raspberry Pi SD card images, that depend on and wait for partitions on the local sd card – even if you boot them from NFS. If you see your Raspberry Pi failing at boot, because it waited for a local partition or device for too long, simply flash the sd card image onto that sd card like you would normally. Then, change the settings to have it boot over NFS. Your local SD card still will not be touched, but the boot scripts and dependencies will work now.

This topic can be split into two subtopics: Using a freshly downloaded image to start from scratch, or converting a normal SD-card-based Raspberry Pi to a NFS-based one. Skip to Convert SD-Card-Based Raspberry Pi To NFS if you already have an SD card image, which you want to convert.


Create A RootFS From Scratch (New Download)

Now you can download any SD card image like you would do normally. But instead of flashing it onto an SD card, you will mount it and copy the content to your NFS share.

In this example I will download the official Raspberry Pi Foundation Raspbian Stretch Lite Image over the Torrent protocol on my Manjaro Linux machine.

[[email protected] Downloads]$ transmission-cli State changed from "Incomplete" to "Complete"

Be nice and give something back to the torrent community by leaving this open seeding (uploading) for the other downloaders. Meanwhile in another terminal window:

Check the checksum to check if the data was downloaded successfully and was not modified:

[[email protected] Downloads]$ sha256sum 

The checksum equals the checksum from the website. Great! Let’s unpack it and have a look inside the partition table. It is a complete device image, so it contains everything from (or for) the SD card, including partition table.

[[email protected] Downloads]$ unzip 
 inflating: 2017-09-07-raspbian-stretch-lite.img
[[email protected] Downloads]$ fdisk -l 2017-09-07-raspbian-stretch-lite.img 
Disk 2017-09-07-raspbian-stretch-lite.img: 1.7 GiB, 1854590976 bytes, 3622248 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
                                ^^^-----<<< write this down! <<<-------
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x11eccc69

Device                           Boot Start  End      Sectors Size Id Type
2017-09-07-raspbian-stretch-lite.img1 8192   93813    85622   41.8M c W95 FAT32 (LBA)
2017-09-07-raspbian-stretch-lite.img2 94208  3622247  3528040 1.7G 83 Linux
                                      ^^^^^-----<<< and this! <<<-------

In the output of fdisk -l you see some important information. As we are not interested in putting the boot partition onto our NFS server (but the rootfs partition, the second partition) instead, we need to know the offset where the interesting partition starts. To get this offset you simply multiply the sector size (usually always 512 bytes) with the start sector of the second partition (both are marked above).

[[email protected] Downloads]$ echo $(( 512 * 94208 ))

Let’s create a loopback device with this information so we can mount it and steal the data from it.

[[email protected] Downloads]$ sudo losetup -f --show -o $((512*94208)) 2017-09-07-raspbian-stretch-lite.img 

You need to be root to create loopback devices. -f found us a free loop device, -o took the offset we calculated and –show made sure we get to know which loop device losetup used. Now we can go and mount it like any other partition.

[[email protected] Downloads]$ mkdir /tmp/fresh-pi-rootfs
[[email protected] Downloads]$ sudo mount -v /dev/loop1 /tmp/fresh-pi-rootfs
mount: /dev/loop1 mounted on /tmp/fresh-pi-rootfs.
[[email protected] Downloads]$ cd /tmp/fresh-pi-rootfs
[[email protected] fresh-pi-rootfs]$ ls
bin boot dev etc home lib lost+found media mnt opt proc root run sbin srv sys tmp usr var

Now let’s copy that data to your NFS share. Refer to the code snippet in Convert SD-Card-Based Raspberry Pi To NFS, but copy from /tmp/fresh-pi-rootfs instead of /.

Afterwards, burn the SD card image to your SD card like you would normally. This ensures that you have the proper boot partition (the first partition in the partition table) on your card. If you feel fancy or have a tiny SD card, you can also extract the partition data like we did above and put it onto a fresh FAT partition on the SD card with the same start sector.


Convert SD-Card-Based Raspberry Pi To NFS

If you already have a running Raspberry Pi, you probably do not want to rebuild everything from scratch but move your existing data to the network share. All you need to do is to log into your Pi, install rsync (or any other application that reliably copies filesystem attributes like scp), and copy all your stuff over. Do not forget to stop your applications to have them in a defined (stopped) state.

[email protected] ~ $ mkdir /tmp/rootfs
[email protected] ~ $ mount nas:/nfs/rootfs /tmp/rootfs
[email protected] ~ $ rsync -Phax --numeric-ids / /tmp/rootfs/openhab

There are about 31.000 files to copy, so this might take a while. Go brew some coffee or grab a beer.


The Configuration

All your data are belong to the NFS share. The last step is to tell the Raspberry Pi to boot from the share instead of the local SD card. One widespread method that works on almost every linux device is to modify the kernel command line. This is usually done in the bootloader (Syslinux,  Grub, U-Boot). The Raspberry Pi got a configurable U-Boot bootloader starting with version 3. In the versions before you have to have a partition on your SD-Card, containing the boot files and a file called “cmdline.txt” that contains the kernel command line. This also works with Version 3 and newer, if you do not want to bother with U-Boot and you are fine with using an SD card for that.

Modify the “cmdline.txt” file on the boot partition. You can do this directly on your Raspberry Pi, or by plugging the SD card into another computer. When using another computer, you’ll find the file on the first Partition of the card, usually called “boot” partition. When editing the file directly on the Pi, it is located at “/boot/cmdline.txt”.

This is an example cmdline.txt I use in production:

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs nfsroot= ip=dhcp elevator=deadline

This is basically the default cmdline.txt with additions of the root, nfsroot and ip options. If you want to define a static IP address, use something like



Please keep in mind, that there must only be exactly one line in the cmdline.txt file. No comments, nothing else. Just one line.


The First Boot

Save it, unmount the boot partition (Apple and Microsoft call this “eject securely” or similar), put the SD card into your Pi and fire it up. If you have a screen attached, you will see the communication with the DHCP server and the handover to the NFS filesystem. This usually takes only a second. My boot time increases by 50% which is most likely the case because of my very old and slow NAS. Results may vary.

Congratulations, you just separated hardware from software resources and made your IT a lot more agile. Now, if your Pi explodes, you can just swap it with a new one from you Raspberry-Pi-drawer and be up again in no time. I am not responsible for exploding Pis, though.


The Remaining Questions

People asked me about this a lot so maybe this is interesting for you as well:

Q: Wait! What happens, if my NAS crashes, or is offline in some other way?
A: As soon as the NFS server doesn’t answer anymore, your devices will just stop completely. The root filesystem is essential so your Raspberry Pi will just wait until it is reachable again. It will not crash and it won’t reboot or panic. It just waits until your server is back. I think this is quite nice.


If you have any other questions, feel free to ask in the comments section below!


Read Also