In this post, I document how to create a bootable rescue USB stick on FreeBSD. Having such a stick is handy in case you want to quickly inspect a broken or suspect system.
There are many ways to Create a rescue USB on FreeBSD. One can also use the FREEBSD official installer to choose to get into a live shell.
I use mfsBSD as the contents of this rescue disk. Normally, one can directly use mfsBSD as a rescue risk but below I specifically create an MBR based USB image in order to boot on those older hardware. Many a time, older hardware is unable to boot using GPT.
WARNING: USE ALL INSTRUCTIONS BELOW WITH CAUTION. THEY CAN SCRAP YOUR COMPUTER IF JUST EXECUTED WITHOUT CHECKING.
Preparing the device
Download the mfsBSD ISO of interest. In this case, it is mfsbsd-13.0-RELEASE-amd64.iso
Have the USB stick ready. In this example we use an archaic 256MB stick
Ensure that the USB stick does not have any data. This example assumes that it doesn’t have any partitions as well.
Caution: Use the right device id. Identify the device id – you can find this after plugging in the USB stick and checking dmesg. In this example, we use da1
Setup the device partitions on da1. We keep a second partition which can be used as a writable scratch area which would be useful during recovery.
One of the top things taken for granted is Time on the internet. Think about it. We have millions, billions, gazillions of devices – servers in data centers running the Googles and the Facebooks, laptops etc. needing to have the right time.
Why? So that financial transactions have accurate timestamps, or that you can see the right time when you want to. The needed accuracy depends on the nature of the application – be in a few sec, few ms or sub-ms. How does all of this work seamlessly in the internet?
Almost all of the internet synchronizes using the Network Time Protocol (NTP). NTP is a protocol or a method for computer clocks to synchronize over a network – including the internet. NTP is one of the oldest protocols active on the internet. Implementations of NTP started coming into use well before 1985 when RFC 958 was published. NTPv4 was published in 2010 with RFC 7822 bringing in NTPv4 field extensions as recently as 2016.
In the NTP protocol, there is a client and a server. The client is the one seeking time, the server serves it.
How does it work? In simple terms, the client sends a request to the server asking for the time. The client believes that the server has a more accurate time. The request contains the client side timestamp. The server receives the request, and responds back with the timestamp on the server side. The client factors in the delay for packet transit and finally computes the offset (difference between the servers’ time and its own). This allows the client to optionally adjust its own clock. The client polls the server periodically – but not too frequently so that the server can cater to a large number of clients if it would like to do so.
OK, I admit the above is an oversimplification; but the intent of this write-up is not to explain the protocol.
When we talk about time synchronization on the internet, we have many servers providing time using NTP to hundreds or thousands of clients on a regular basis. These are run on a voluntary basis by government bodies, corporations or organizations and individuals.
ntp.org is the home of the Network Time Protocol project. The NTP Pool project is one which enables and provides the framework of NTP pools.
What is a NTP pool? The NTP pool project creates a pools of servers and groups them in geographical zones (with country as the lowest granularity). As an example, if a client from Japan requests for time from the NTP Japan pool, the client is assigned one of the servers who are participating in the Japan pool. This allows clients to be connected with servers that are in their proximity thereby allowing for faster synchronization.
State of affairs
As can be seen in the image above, certain areas like Europe have good participation in the pool whereas Asia, South America have extremely poor participation.
If you have a server on the internet with a static IP address, the best way to help is to join the pool. You can join the pool by following the instructions here.
Running an NTP server on the internet does not require a larger amount of maintenance. Some things to sort out before you embark as a timekeeper on the internet.
Selecting the server software – ntpd, ntpsec, chrony etc. There are slight differences between each of these though the overall concept and configuration is the same.
Ensuring your server has a static address – one that is fixed for the long term (think years!)
Ensuring you have the right connectivity (bandwidth & a stable connection)
In terms of hardware resources – both CPU and memory, the ask of an NTP server is minimal. However if you are handling a tens of thousands of clients, CPU starts to become a consideration
Which hardware – there are different opinions about virtual machines vs dedicated hardware – virtual machines can very much hold their own in the NTP Pool being extremely stable. You just need to choose a suitable provider
Mental note: At some point, I will put together a brief comparision of ntpd, ntpsec and chrony in their default configuration.
How many sources is good enough?
The NTP Pool project suggests to setup 4-7 servers to synchronize time with. I love this quote.
A man with a watch knows what time it is. A man with two watches is never sure
Once you configure the 4-7 servers to synchronize with, your NTP server will start polling them to maintain its own time.
After joining the pool, the NTP Pool project starts monitoring your servers time via NTP. Your server is assigned a score based on the offset and anything more than 100ms is not acceptable. An unacceptable score starts degrading your score and eventually you are kicked out of the pool available for clients. If your server recovers, your score gradually improves and once over 10, you are again added into the pool available for clients. You can monitor your server as in the below image.
The burden of serving time
Don’t underestimate the network traffic requirements of running a NTP server. If you are not careful, your bandwidth can easily run into many TB per month. Depending on your bandwidth usage plan for the server, this might result in huge additional costs that you did not factor in. One good set of articles available to check this in detail are here, here and here.
The NTP Pool project allows you to manage your servers and tune the bandwidth you want for your server. This is not an exact setting since the NTP Pool project only connects clients to you (via DNS) but clients can continue to be attached to you from a few hours to days to even years.
Individual clients do not pose a bigger challenge as long as they are using standard NTP client software; but additional challenges come with misbehaving clients who will poll too frequently or pool continuously – even many many times per second. There are ways to address these – using rate limitation rules in your NTP server software or implementing a firewall.
Finally, once you have your server configured and setup exactly how you like, a picture like this will bring a big smile on your face.
Running a NTP server is a great project to contribute to the Internet infrastructure as well as to learn about running a production system on the internet. There are a number of considerations before embarking on such a project, but it could be lead towards a very satisfying experience.