Trrrrrriiiinnnnnnnggggg… the school bell rang, signalling in the lunch break. Raghu and Anirudh slipped out of the classroom before the class teacher allowed the children to leave. Unlike other children who would sit in or around the classroom and eat their lunch, these two would visit a different spot in the school each day.
Their classroom was in one of the older buildings of the school. It housed the 4th to 6th standard classes. Both of them were waiting to be promoted to the 7th standard – when they would move to the new building. Everything was so new and clean, the smell of fresh paint, new furniture and the classrooms even had cabinets for the children.
“Let’s go to the playground near the church,” Raghu said.
St. Johns’ English Medium School was one of the older schools in the locality. Three school buildings and two grounds around a small church formed the school.
Anirudh was Raghu’s best friend and they were always together during school. When one of them did not come to school, the other felt out of place. Their lunch time together was special because they could talk, crack jokes and make fun of others. The lunch break was forty-five minutes long, and the boys usually took an extra five minutes to get back to class.
They found a shaded spot on the ledge on the wall near the playground where there weren’t any other children. Though it was lunch time, there were kids who were still on the playground. Some playing catch-catch with a tennis ball, some cricket and some just walking around in small groups without purpose.
Raghu had his dabba with him. He wondered what his Amma had packed for him. It usually was a few idlis, or dosas, and sometimes a rice mix. He loved to eat rice and pickle, but she didn’t send it frequently enough. He opened the small steel box and saw four idlis.
“Oh no, Not idlis again”, he said aloud. He wondered what Anirudh had got for lunch today. He often got special lunch; sometimes fried rice, parathas, chole bature etc.
“Where’s your dabba?”, Raghu asked seeing that Anirudh was empty-handed.
“Mom could not send one today. Would you share yours with me?”
“No, we should not share food and should get our own dabbas”, Raghu said while turning to look towards the playground.
During lunch, while Raghu ate his idlis, Anirudh tried not to look at his dabba. He instead looked at a small group of children who were playing cricket on the far side of the ground. The boys then talked about the half-yearly school fair that was to be held in the coming weekend. It was something that all the kids looked forward to since they could come to the school with their parents and play games and eat from the many food stalls that would be setup. The school bell rang and they realized their lunch time was over.
As they walked back towards their classroom, Raghu wondered if he did the right thing by not sharing his food with Anirudh. Anirudh got his dabba most of the times, but he wouldn’t once or twice a month.
“Yes, it’s the right thing to do. That’s the only way he would get his dabba everyday”, he thought to himself and put his arm around Anirudh’s shoulder.
One of the top things taken for granted is Time on the internet. Think about it. We have millions, billions, gazillions of devices – servers in data centers running the Googles and the Facebooks, laptops etc. needing to have the right time.
Why? So that financial transactions have accurate timestamps, or that you can see the right time when you want to. The needed accuracy depends on the nature of the application – be in a few sec, few ms or sub-ms. How does all of this work seamlessly in the internet?
Almost all of the internet synchronizes using the Network Time Protocol (NTP). NTP is a protocol or a method for computer clocks to synchronize over a network – including the internet. NTP is one of the oldest protocols active on the internet. Implementations of NTP started coming into use well before 1985 when RFC 958 was published. NTPv4 was published in 2010 with RFC 7822 bringing in NTPv4 field extensions as recently as 2016.
In the NTP protocol, there is a client and a server. The client is the one seeking time, the server serves it.
How does it work? In simple terms, the client sends a request to the server asking for the time. The client believes that the server has a more accurate time. The request contains the client side timestamp. The server receives the request, and responds back with the timestamp on the server side. The client factors in the delay for packet transit and finally computes the offset (difference between the servers’ time and its own). This allows the client to optionally adjust its own clock. The client polls the server periodically – but not too frequently so that the server can cater to a large number of clients if it would like to do so.
OK, I admit the above is an oversimplification; but the intent of this write-up is not to explain the protocol.
When we talk about time synchronization on the internet, we have many servers providing time using NTP to hundreds or thousands of clients on a regular basis. These are run on a voluntary basis by government bodies, corporations or organizations and individuals.
ntp.org is the home of the Network Time Protocol project. The NTP Pool project is one which enables and provides the framework of NTP pools.
What is a NTP pool? The NTP pool project creates a pools of servers and groups them in geographical zones (with country as the lowest granularity). As an example, if a client from Japan requests for time from the NTP Japan pool, the client is assigned one of the servers who are participating in the Japan pool. This allows clients to be connected with servers that are in their proximity thereby allowing for faster synchronization.
State of affairs
As can be seen in the image above, certain areas like Europe have good participation in the pool whereas Asia, South America have extremely poor participation.
If you have a server on the internet with a static IP address, the best way to help is to join the pool. You can join the pool by following the instructions here.
Running an NTP server on the internet does not require a larger amount of maintenance. Some things to sort out before you embark as a timekeeper on the internet.
Selecting the server software – ntpd, ntpsec, chrony etc. There are slight differences between each of these though the overall concept and configuration is the same.
Ensuring your server has a static address – one that is fixed for the long term (think years!)
Ensuring you have the right connectivity (bandwidth & a stable connection)
In terms of hardware resources – both CPU and memory, the ask of an NTP server is minimal. However if you are handling a tens of thousands of clients, CPU starts to become a consideration
Which hardware – there are different opinions about virtual machines vs dedicated hardware – virtual machines can very much hold their own in the NTP Pool being extremely stable. You just need to choose a suitable provider
Mental note: At some point, I will put together a brief comparision of ntpd, ntpsec and chrony in their default configuration.
How many sources is good enough?
The NTP Pool project suggests to setup 4-7 servers to synchronize time with. I love this quote.
A man with a watch knows what time it is. A man with two watches is never sure
Once you configure the 4-7 servers to synchronize with, your NTP server will start polling them to maintain its own time.
After joining the pool, the NTP Pool project starts monitoring your servers time via NTP. Your server is assigned a score based on the offset and anything more than 100ms is not acceptable. An unacceptable score starts degrading your score and eventually you are kicked out of the pool available for clients. If your server recovers, your score gradually improves and once over 10, you are again added into the pool available for clients. You can monitor your server as in the below image.
The burden of serving time
Don’t underestimate the network traffic requirements of running a NTP server. If you are not careful, your bandwidth can easily run into many TB per month. Depending on your bandwidth usage plan for the server, this might result in huge additional costs that you did not factor in. One good set of articles available to check this in detail are here, here and here.
The NTP Pool project allows you to manage your servers and tune the bandwidth you want for your server. This is not an exact setting since the NTP Pool project only connects clients to you (via DNS) but clients can continue to be attached to you from a few hours to days to even years.
Individual clients do not pose a bigger challenge as long as they are using standard NTP client software; but additional challenges come with misbehaving clients who will poll too frequently or pool continuously – even many many times per second. There are ways to address these – using rate limitation rules in your NTP server software or implementing a firewall.
Finally, once you have your server configured and setup exactly how you like, a picture like this will bring a big smile on your face.
Running a NTP server is a great project to contribute to the Internet infrastructure as well as to learn about running a production system on the internet. There are a number of considerations before embarking on such a project, but it could be lead towards a very satisfying experience.