Linus Torvalds & Dirk Hohndel

The biggest draw of this event was Linus Torvalds and his panel discussion with Dirk Hohndel. When asked about deprecating old hardware support from the kernel, he responded with “the cost of maintaining becomes too much at some point and it calls for deprecation”. Further, he attributed the recent uptick in traffic during a merge window, partly to extensive AI use by developers. AI has been finding bugs that the maintainer feels should go as part of the imminent release, not later. During a merge window that is typically 2 weeks long, he merges ~200 requests, and avoids travelling during this time period which he considers a sacred focus block. He further went on to mention quite candidly that he’s not a programmer anymore but that of a tech lead, so he naturally tries to understand the problem that a particular request is trying to patch. This is one of the reasons he emphasizes good commit messages as part of a request as that’s the most critical part of the request for him, he’s not reading much code lately.

Rust is now optional in git and will be a requirement in the next version of git. Linus is of the opinion that Rust in itself is not going to change the world, and that he’s fairly biased towards C which he thinks is a much simpler language to write. But more importantly, Rust in Git is not an epoch moment as the internet would have you believe. In fact, there used to be a version of Git in Java that was developed for enterprise environment where Java was omnipresent.

Linus was also surprisingly not-hostile towards AI as has been the trend lately for some weird reason. He personally uses AI for his toy projects and thinks the kernel is much better off with AI finding security bugs that we should’ve found 20 years ago. Of course, it goes without saying that on the other side of this coin lies the AI slop problem which many open source projects are lately dealing with.

Digiyatra

I’m a big fan of Digiyatra. It’s one of those conveniences that you appreciate whenever you fly [domestically] and so I was quite interested to understand what they have to do with Open Source. Because in all honestly, I was expecting corporate drivel but it was an insightful panel discussion.

Digiyatra for those who don’t know is a system for streamlined security checks at major Indian airports primarily for domestic travel. The concept behind this idea was “my face as a boarding pass” or “face as a single token”. They initially had a centralized storage system for all the customer’s PII. But that changed during the global pandemic when they switched to a decentralized storage system where all the user data resided on the customer’s phone. The customer would add its flight details close to the flight departure and the app would push the relevant data to the specific airport. This data is then removed shortly after the user’s flight has departed. They also practice selective disclosure – exposing/sharing only the data that’s requested. For instance if there’s a requirement to know the age of the user, just the birth date would do.

Device Tree

This talk was an introduction to Linux Device Trees. It’s a data format that describes non-discoverable hardware and is used by the linux kernel to understand the hardware it is running. The author explained Device Tree Source (DTS) and Devicetree Blob (DTB) before further delving into somewhat practical examples of how and when should one write DTSs. I would’ve loved if the talk had bridged the gap between what a DTS is and how I can use it, perhaps a minimal demo because the second half of the talk felt a little rushed but maybe that’s just my lack of knowledge on the topic.

Open Source Is Not the Same Anymore

This was an interesting–and slightly sensationalist imo–talk that claimed that the rules of the game have changed. In that Open Source is not the same it used to be a couple years back. They gave examples of curl shutting down its bug bounty program because of lots of AI slop, and Synadia taking back NATS from CNCF in hopes of relicensing it into a proprietary project, and finally the HashiCorp and Terraform license debacle. In due course though, Synadia backed off and OpenTofu was forked by the community. The point being “license stopped being the contract”; in the past ~5 years, Elastic, Terraform, Redis and NATS have all tried to or have moved their license. Which led the presenter to their next, bolder claim that “the community stopped being a community”.

So they’ve proposed a set of 5 rules that new contributors can follow in order to gauge whether a project is worthy of their contributions. These rules included reading the license, effectively grepping for certain blacklisted terms, understanding if the bus or elephant factor of the project is not dangerously low, or sensing how quick the turnaround time of the maintainers on PRs or issues are, among 2 others. This idea of “gauging” a project’s worthiness before beginning to contribute to it is something I found I didn’t agree at all. I did have a discussion with the author after the session where I argued that if contributors had taken a look at the bus/elephant factor of Terraform before they started contributing, there might not have been an OpenTofu. Or on a more personal note, had I checked for Podman’s bus/elephant factor, I wouldn’t have contributed to it and eventually grown into a maintainer role. I believe contribution to open source projects should be motivated for the love of the game, not for second-order tangible outcomes such as one’s future in the project.

I do agree that the five rules certainly help tell you about the hygiene of a project but that should never be your [only] heuristic in deciding whether you should contribute to the project or not. If anything, I believe this hampers Open Source innovation. I should note that I had a constructive discussion with the author while discussing the same.

Guide To Become Linux Kernel Maintainer

This was a full house, and ended up being a great simple, no-bs talk on how to best contribute to the Kernel. Drivers or subsystems are the best places for someone new to start with, there are countless orphaned subsystems that are in need of contributors, reviewers and maintainers. The kernel, in general, needs more and more reviewers especially now, with the advent of AI, the amount of contributors have grown significantly leading to increased burden on the maintainers.

Kubernetes Networking: A packets journey across pods and nodes

I’m a big fan of talks that are practical in nature and try to demystify a technical concept or component; I’ve personally prepared and presented two such talks, namely Build a Container Image from Scratch and Build a Container Registry from Scratch. So I was quite excited about this talk and it absolutely didn’t disappoint.

The crux of the talk was deciphering how pods connect with other pods either on the same node or on different nodes. We went through standard bridge+veth pairs to using TUN, to learning about ARP spoofing and flooding, overcoming that with VXLAN and finally calling it a day with using BGP via BIRD daemons. It was an extremely informative and interesting talk.

Accelerating Innovation Through Open Source at Jio

I hadn’t planned to attend this session but as this was a keynote that happened in the biggest hall where I also happened to be sitting at the back doing some work, I ended up getting some insights into the role that Open Source software plays at Jio. First and foremost, I appreciate a corporate as big as Jio on stage talking about how important Open Source is for them, because fluff or not, I believe that statements such as these at a keynote in one of the larger Open Source conferences must carry some impact.

On the technical front, I was surprised to know that Jio relies heavily on Container Images, they use some form of UBIs apparently but of course tuned and configured to suit their own specific needs. They have a Linux team who tunes the kernel config, slims down the kernel based on their requirements. It wasn’t clear whether they contributed back to the kernel or not, at least I couldn’t find Jio in the list of contributors on the kernel stats page at LF, please correct me if I’m wrong here.

Rust & Linux

Keynote session by Greg Kroah-Hartman where he expounds the state of Rust in the Linux ecosystem, mainly the kernel. The basic premise is that while C is unforgiving, Rust makes it easy for developers to not make small mistakes and that directly translates to reviewers and maintainers focusing on more important bugs or issues, linking back to the same idea in one of the earlier talks where a need for more reviewers was discussed. Apart from the direct impact Rust is having, it’s also changing the way C is being written in the kernel, adding a guard() method for instance for taking care of cleanup, somewhat similar to Go’s defer I presume. Greg ended the talk with the idea that Rust will help make code reviewing easier in the future and that if the kernel were to be written in Rust, ~80% of the CVEs would be impossible. Throughout the talk I couldn’t help but think about how different Linus’s and Greg’s views on Rust in Linux are. Linus sees nothing magical about Rust without being dismissive but at the same time Greg is extremely optimistic about the future of Rust in the kernel.

Pruning Kernel CVEs With Code Reachability Analysis

This was a fun one as well. There has been a significant increase in the number of reported vulnerabilities in the Linux kernel. But not all of them affect the particular version of the kernel on that hardware with that set of config and kmodules, etc. For instance if you are notified of a vulnerability in the bluetooth subsystem but you don’t build with bluetooth support, then that vulnerability doesn’t really apply to you.

The author introduces a tool that attempts to evaluate for false positives by static analysis. It does so by evaluating Build time, runtime and system call reachability. There are plans to support VEX output and potentially expanding the scope to supporting language runtimes as well.

Closing

Despite being more or less technical in theme, AI was the buzzword at the conference, for better or worse. Linus talked about it, Greg dismissed it, many presenters focused on it. Not that I wasn’t expecting it but I was glad AI didn’t really take the center stage at the conference, it was always on the sidelines save for a few talks.

This was the third Open Source Summit I’ve attended. I’ve spoken previously at the Japan and Europe editions in 2024 and 2025 respectively. I was an attendee this time around so I had ample time to attend all the sessions I wanted to and to engage in discussions with quite a few folks, which was “enlightening” in its own right.

I can’t really compare this conf with either KubeCon or FOSDEM or any other major conferences for I’m yet to attend those. But all the sessions I attended were informative and insightful. I had one gripe with the whole event and that was the Solutions Showcase. It was bland at best, nowhere close to OSSEU or OSSJ and I hope future iterations of OSSI will have a more engaging, diverse and larger solutions showcase.

So it’s been roughly 7 years that I’ve been budgeting; for the first year, I tried out various apps and tools. I downloaded and created multiple spreadsheet templates to find the one that suited my needs. And in May 2020, I came across YNAB (You Need a Budget), and found that it offered what I needed in a budgeting app. It uses a method of budgeting called Zero-Based Budgeting (ZBB). And the idea is to give every dollar of yours a job. This job could be paying your utility bills, your insurance, investments, or your monthly app subscription. So every month, when inflow in the form of either your salary or business returns, lands in your account, you distribute each and every single penny across various categories/buckets. And once you do that, you don’t have any money “lying around” or any dollar/rupee without a purpose. If you’re interested in YNAB specifically, there’s a good book on it by its founder or find more information on its website.

As of today, I’ve tracked ~8500 transactions over the the past 6 years, that’s roughly 1400 transactions per year. People sometimes comment on the value of budgeting and sticking with it for so long. But the thing with budgeting is that its first order effects are intangible and hence easy to ignore. They’re not apparent until after you’ve stuck with it for some time. This threshold obviously is variable but in my opinion, it does exist. Many times in the past, I’ve made the mistake of reaching for tangible outcomes when talking about budgeting, such as having a log of transactions, being able to visualize my spending trends, etc. Those are all second-order benefits which I’ll talk about shortly. Budgeting, first and foremost, is about awareness. Simply being aware of my finances is extremely valuable in itself. Having a bird’s eye view of my budget, areas I spend on, my assets, my expenditure leaks, etc, have made me acutely aware of my finances in general.

The very act of budgeting–keeping track of all transactions you make–helps with awareness. With the advent of digital payment options such as UPI and Credit Cards, the weight of spending is disappearing. All you need to complete a transaction is to either tap your card or enter your PIN; the amount doesn’t come to mind for the second time, and before you know it, you’re down a couple thousand bucks by the end of the day, if not the week. Compare this with the traditional cash-heavy model of payments where you are conscious about the amount of money spent because the higher the amount the higher the denomination of notes, or the number of notes, or both. So making the same purchase with both cash and a card, the former is bound to make you more aware of how “heavy” the purchase was. Thankfully, I’ve found that manually entering transactions is the next best thing when it comes to feeling the weight of the transaction similar to if I were using cash. There’s also the added benefit of recognizing leaks in my budget, once I was required by the process to track each and every transaction across all my accounts, I quickly realized expenses which were not yielding any value or transactions I wasn’t even aware of that were on auto-pay, and was eventually able to cut them out.

Having a clear picture of my finances directly led to reduced anxiety and stress. A lot of this stress existed because I had little clarity on where my money went by the end of the month. This is also perhaps the provenance of the joke “I’ve no clue where my money went!”. But once I knew where my money was going month-on-month, and of course after fixing some of the issues, there was surprisingly very little to worry about. In the past, when I had no system, no framework, when I would deal with purchases on an ad-hoc basis, I used to stress about finances because I couldn’t possibly keep my entire budget in my mind. But once I started budgeting, all I had to do was to consult my budget for the month and I’ll have a clear answer to questions such as “Would I be able to travel in August?” or “Can I buy this?”. Note that I used the term “buy” instead of “afford” but that’s a different, more psychological discussion not suited for this essay. I can now spend guilt-free on a lot of things because I kept money aside in the past few months leading up to the purchase or travel. The corollary is that I’ve not stressed about my finances in the past 6 years and that’s primarily in part due to budgeting. Sure, there have been occasional hiccups and that’s fine as long as I’m able to optimize for at least the remaining 80%.

Before I talk about the second-order benefits, it’s imperative that I talk about how my personal finance system has evolved and matured over the years. For starters, ZBB has changed the way I think about purchases, big or small. Everything relies on me tracking each and every transaction that I do across all my accounts, and then I can simply consult the particular category before making any purchase. But this also means telling myself sometimes that no, I can’t afford an outing this month. Funnily enough, there’s a term for this in the community–YNAB Broke. It’s when you objectively do have money but you still can’t spend on a particular category because there’s simply not enough money left in that category for that month. So you wait for next month to refill that category and go about your business as usual.

On a more practical note, I spend roughly an hour on the 1st or 2nd of every month on what I call the “Budget Day”. This involves allocating my monthly inflow into various categories, reconciling the transactions between my banks and the app, updating investment numbers, and paying off debt, if any. Spending an hour on the budget every month is almost therapeutic; in that I can see how terrible last month was and how I should course correct. Or on good months, It gives me a chance to think ambitiously about my financial goals for the future.

Once I have funded all the categories that I could or want to that month, I’m technically set for that month. Now I could only spend the money available to me for that category, no more. It might seem a bit restrictive at first but you get used to it in no time. A few months in and I learnt about leaks in my budget and came across categories I really want to spend or splurge on. With that information, I could distribute the allocation to suit my wants and needs for the upcoming months. For example, I love to travel and I very quickly realized that I should be allocating more to the travel bucket and effectively reduce allocation towards eating-out/ordering-in categories, it’s a zero-sum game after all. So now, whenever I travel, I can freely (and responsibly!) spend the amount from that category because it’s meant to be spent on travel. There are no qualms about whether I’m spending too much or whether I have enough left for some emergency waiting right around the corner. For annual purchases or recurring purchases, I use what YNAB calls annual goals, which is essentially simple arithmetic that consists of dividing a one-time annual purchase in 12 monthly parts that you allocate to the category every month so that when the time rolls around for that purchase, the money is sitting right there. I use this extensively for insurance payments, annual subscriptions, and personal finance goals.

This all might seem a bit much, and clearly this is additional overhead that’s not desirable for many. It was, for me. But then I wanted to ensure this overhead doesn’t scale with time. In pursuit of this, I reduced the number of credit cards I used from 3-4 down to 1, effectively consolidating all my spendings on just one card. This made reconciling the transactions between my card and my budget much easier than reconciling 3-4 different cards every month. Additionally, I got my credit card statement’s date changed such that it overlaps with my Budget Day. Now, when I sit for Budget Day, my credit card statement is ready for the previous month so I can reconcile the statement with my budget then and there. This makes budgeting slightly less time consuming than it was. In the same vein, I also reduced the number of funds I invest into from ~6-7 down to 3, with the same goal of reducing overhead when I reconcile transactions between my budget and investment portfolio every month. Less is more.

Now finally coming to second-order benefits. The most obvious one is visualization. This is the reason why many people are attracted to budgeting. Those fancy line graphs, colorful pie charts. They are real! My favorite of these reports is the Net Worth report. I mostly use it every year to set personal finance goals (reach certain NW by the end of 20XY year). The NW graph also helps with positive reinforcement–you see the graph rising, and you want to see it continue to rise! The other graph that I quite frequently refer to is the monthly spending trends trend. It’s a histogram that shows how you’ve been spending your money over the past few months or years. For instance, when I moved to Delhi from Bangalore last year, my spending graph declined because a lot of the expenses I used to bear when I lived alone were simply gone now and so I could see that I was, in fact, saving slightly more money than before. I also routinely use this graph to filter out spends on subscriptions over months to see if a sneaky subscription has come up increasing my baseline spends on monthly subscription fees. There are many other graphs available but I don’t use them as much as I use the two I mentioned.

The other second-order benefit worth talking about is “lookbacks”. I’m not sure if this is a term but it’s loosely based upon TLBs. Lookbacks simply means a vast inventory of old logs and information that you can refer to at any point in the future to derive insights or value. They have been immensely helpful for me in recent times. Having a vast inventory of transactions where I have noted down each and every spend and purchase, has come in handy when I want to recall how much I paid for an item, let’s say 2 years back. This might sound minor but has helped me numerous times while I’m negotiating a price for something. For instance, when I was purchasing a new battery for my car that I had replaced two years back from the same vendor, I was able to quote him the old price for the same battery and was able to secure a small discount which would’ve been otherwise not possible, or if possible, it would’ve been too cumbersome for me to even try. Or negotiating the price for a gym membership fee by telling the guy at the reception that I’ve been paying X for the past Y years and perhaps it’s time for a loyalty discount? It’s a small but very satisfying win.

When I show how I budget to friends and family, they often dismiss it as too much overhead for an intangible return. And rightly so, personal finance is not the same for everyone, much like everything else in life. I have many friends who are financially doing very well and have not budgeted once in their lives. Which is to say, budgeting is not something that would magically make your financial hygiene better. It would expose it, sure and perhaps that’s exactly what some people need, but that’s about it. And once exposed, that’s when you need to put the effort in. By prioritizing spending, cutting back on superfluous purchases, planning months into the future. That’s when you start seeing noticeable results.

Being able to distribute containers efficiently, securely, and at scale is one of the primary reasons why containers are so ubiquitous. There are countless registries available online that store and distribute pre-built and custom container images. Docker Hub and Quay.io are a few of the popular ones. Registries allow us to distribute our container images to other users, and at the same time, allow us to use container images authored and built by others, reducing the usual overhead that comes with writing Containerfiles.

This article tries to answer the following question—”What happens when you do podman push/pull?” We’ll do so by creating a simple container registry–we’ll call it tiny-registry—using pure Go under 200 lines that will respond to push and pull commands from any container engine (we’ll be using podman for our use case).

What makes a registry?

OCI Distribution Spec

The OCI Image Spec defines the structure of a container image, while the OCI Distribution Spec defines how clients interact with a registry.

In simpler terms, the OCI Distribution Spec defines a set of API endpoints that a registry needs to handle for it to be considered compliant with the OCI Distribution Spec and act as a functional container registry that clients can interact with.

Conformance with the spec is defined for various categories, but at the bare minimum, a registry should support all the API endpoints that correspond to an image PULL. Our tiny-registry will support a few more endpoints than just PULL to support podman pull and podman push against our registry. The list of endpoints we will support is as follows:

GET     /v2/ — version check (clients probe this first)
GET     /v2/{name}/manifests/{reference} — pull manifest (handles both tags and digest refs)
GET     /v2/{name}/blobs/{digest} — pull blob
POST    /v2/{name}/blobs/uploads/ — initiate upload session
PATCH   /v2/{name}/blobs/uploads/{session} — stream blob data
PUT     /v2/{name}/blobs/uploads/{session} — finalize blob (rename tmp → blobs/)
PUT     /v2/{name}/manifests/{reference} — store manifest

Let’s get to the implementation. Since this is a demo registry, our goal is to ensure our registry can support push and pull from container engines. This means we should be able to podman push a container image to our registry, and subsequently podman pull that same image and be able to run a container off of it.

1. Push — to the registry

1.1 Storage model: storing the image

If you recall, an OCI image consists of 4 key components, the image index, an image manifest, the image config, and the layer blobs. All of these are content addressable objects, except for the image index in an OCI layout. The registry doesn’t have to deal with index.json although it can store them for multi-arch images. All it needs to do really, as we saw before, is to handle the routes as stated in the OCI Distribution Spec.

This means we’re concerned with storing only the manifest and the blobs on the host filesystem. The distribution spec doesn’t mention anything about how you store these components on the filesystem, it’s only concerned with how you serve them to the client—by following the API endpoints it defines.

For the tiny-registry, we can simplify things by having the following directory structure:

registry/
├── blobs/
│   ├── sha256:216659b5...   ← manifest (also stored as blob)
│   ├── sha256:a3b655d4...   ← layer blob
│   └── sha256:af3f0f48...   ← config blob
├── {name}/
│   └── manifests/
│       └── latest           ← pointer file, contains the manifest digest
│       └── v1               ← pointer file, contains the manifest digest
└── tmp/                     ← staging area for in-progress uploads

• blobs/ - Leveraging the benefits of content-addressability, we have one single directory where we’ll store all the blobs of all the images ie the layers, manifests, and configs.
  • This helps us avoid duplication since if there are two images sharing layers, we’ll only store one copy of it on the filesystem and serve that file when a pull request comes in for either of those images.
• {name}/ - We create a new directory for every image that would store the manifests’ sha256 digest. This corresponds to an image tag, and for every additional tag for this image, we’ll create a new file.
• tmp/ - This is a temporary directory used as part of the PUSH flow; we’ll use this for rudimentary session handling our tiny-registry will implement.

Production-grade registries typically use object storage (such as S3) instead of a local filesystem, and maintain metadata in a database. The underlying idea, however, remains the same.

1.2 Upload protocol: the push endpoints

Now let’s understand the API endpoints corresponding to the PUSH workflow. A client can push a blob in two ways, either chunked or monolithically. Podman typically performs chunked uploads. It has obvious advantages over monolithic especially when it comes to pushing large blobs, most important being not having to start the push from scratch if your connection to the registry was interrupted for some reason.

For our registry to support chunked push, we have to support the following routes:

GET     /v2/                                    handleVersion

POST    /v2/{name}/blobs/uploads/               handleInitUpload
PATCH   /v2/{name}/blobs/uploads/{session}      handlePatchBlob
PUT     /v2/{name}/blobs/uploads/{session}      handleFinalizeBlob
PUT     /v2/{name}/manifests/{reference}        handlePutManifest

A chunked push has essentially 4 parts to it that are handled by 4 different HTTP handlers in our code:

1. handleVersion - This is to confirm whether the registry is Distribution Spec compliant. We return 200, and the client continues. The client runs this everytime it attempts to engage with the registry as part of a workflow, push or pull for instance.
2. handleInitUpload - This initializes the upload by creating an empty file under ./registry/tmp/<sessionID> where sessionID is a random 16-digit hex code.
3. handlePatchBlob - The client streams the blob data, we append it to tmp/, and return 202 Accepted.
4. handleFinalizeBlob - Client says “I’m done sending the blob, here’s the expected digest”. We rename registry/tmp/<sessionID> to blobs/sha256:..., and return 201.
5. handlePutManifest - Once all the blobs are uploaded, client sends the manifest JSON for the image as the body. We compute the digest of the manifest, store it under blobs, and write that digest under {name}/manifests/{tag}.

1.3 Implementation

Here’s what our Go pseudocode looks like when it’s handling the PUSH workflow:

// routes
mux.HandleFunc("GET /v2/", handleVersion)
mux.HandleFunc("POST /v2/{name}/blobs/uploads/", handleInitUpload)
mux.HandleFunc("PATCH /v2/{name}/blobs/uploads/{session}", handlePatchBlob)
mux.HandleFunc("PUT /v2/{name}/blobs/uploads/{session}", handleFinalizeBlob)
mux.HandleFunc("PUT /v2/{name}/manifests/{reference}", handlePutManifest)

func handleInitUpload(w http.ResponseWriter, r *http.Request) {
    rand.Read(b)
    sessionID := hex.EncodeToString(b)
    os.Create("registry/tmp/" + sessionID)
    w.Header().Set("Location", "/v2/"+name+"/blobs/uploads/"+sessionID)
}

func handlePatchBlob(w http.ResponseWriter, r *http.Request) {
    data, _ := io.ReadAll(r.Body)
    f, _ := os.OpenFile("registry/tmp/"+sessionID, os.O_APPEND|os.O_WRONLY, 0o644)
    f.Write(data)
}

func handleFinalizeBlob(w http.ResponseWriter, r *http.Request) {
    digest := r.URL.Query().Get("digest")
    os.Rename("registry/tmp/"+sessionID, "registry/blobs/"+digest)
}

func handlePutManifest(w http.ResponseWriter, r *http.Request) {
    digest := fmt.Sprintf("sha256:%x", sha256.Sum256(data))
    os.WriteFile("registry/blobs/"+digest, data, 0o644)
    os.WriteFile("registry/"+name+"/manifests/"+reference, []byte(digest), 0o644)
}

The above should be considered pseudocode as I’ve tried to simplify by only showing the routes and the happy paths. It would be wasteful to paste the complete source here. You can always take a look at the source code here.

The methods and handlers we defined above would allow our registry to respond to a client’s push request, a podman push for instance, and store the image blobs and manifest on the host filesystem. We’ll bring it all together in a later section and see how it all works.

2. Pull — from the registry

Since we already defined the filesystem layout, implementing support for the PULL workflow is simpler in comparison. For starters, we need to support the following API endpoints:

GET /v2/                                 handleVersion

GET /v2/{name}/manifests/{reference}     handleGetManifest
GET /v2/{name}/blobs/{digest}            handleGetBlob

1. handleGetManifest - When pulling, the client first requests the manifest, and uses it to further request the blobs by digest.
   • If you recall, we store the manifest by its content-addressable name under blobs/ and store that digest under ./registry/{name}/manifests/{tag}
   • So we first read the digest and then return the actual contents from ./registry/blobs/sha256....
2. handleGetBlob - Once the client has the manifest, it will request the config and layer blobs by referring to their digests.

Here’s what the Pull implementation looks like:

// routes
mux.HandleFunc("GET /v2/", handleVersion)
mux.HandleFunc("GET /v2/{name}/manifests/{reference}", handleGetManifest)
mux.HandleFunc("GET /v2/{name}/blobs/{digest}", handleGetBlob)

func handleGetManifest(w http.ResponseWriter, r *http.Request) {
    digest, _ := os.ReadFile("registry/" + name + "/manifests/" + reference)
    data, _ := os.ReadFile("registry/blobs/" + string(digest))
    w.Write(data)
}

func handleGetBlob(w http.ResponseWriter, r *http.Request) {
    data, _ := os.ReadFile("registry/blobs/" + digest)
    w.Write(data)
}

3. Execution — tracing a push/pull

3.1 Tracing an image push

Once we’re done with the implementation, let’s run our registry and test it:

$ tree tiny-registry
.
├── go.mod
└── main.go

Looking at our tiny-registry source directory, we can see it doesn’t have a lot going on: a go.mod file, and our main.go file that houses all the logic we’ve discussed so far. Let’s now try to push an image to our registry:

$ go run main.go &

$ podman push --tls-verify=false busybox:latest localhost:8080/busybox:latest
GET     /v2/

HEAD    /v2/busybox/blobs/sha256:a3b655d4356d3eafbfce127e94844901a68074eb6fde905780d46cabf0caf342
ERROR   getting blob: open registry/blobs/sha256:a3b655d4356d3eafbfce127e94844901a68074eb6fde905780d46cabf0caf342: no such file or directory

POST    /v2/busybox/blobs/uploads/
PATCH   /v2/busybox/blobs/uploads/82d84bdff2813060a23991fde157fcf5 name=busybox sessionID=82d84bdff2813060a23991fde157fcf5
PUT     /v2/busybox/blobs/uploads/82d84bdff2813060a23991fde157fcf5 name=busybox sessionID=82d84bdff2813060a23991fde157fcf5

HEAD    /v2/busybox/blobs/sha256:af3f0f48a24edb84e94aff6f44f5d089203453719d3b2328486d311e61db9b09
ERROR   getting blob: open registry/blobs/sha256:af3f0f48a24edb84e94aff6f44f5d089203453719d3b2328486d311e61db9b09: no such file or directory

POST    /v2/busybox/blobs/uploads/
PATCH   /v2/busybox/blobs/uploads/d659ef9bb0583136ec814301a63d1eae name=busybox sessionID=d659ef9bb0583136ec814301a63d1eae
PUT     /v2/busybox/blobs/uploads/d659ef9bb0583136ec814301a63d1eae name=busybox sessionID=d659ef9bb0583136ec814301a63d1eae

PUT     /v2/busybox/manifests/latest name=busybox reference=latest

You can note the order of execution from the registry’s viewpoint.

• GET: Client does the API version check first.
• HEAD: Client checks if the layer blob exists or not.
• ERROR: It doesn’t; so the client initiates the upload.
• POST-PATCH-PUT: The client streams the blob to the registry.
• POST-PATCH-PUT: The client repeats the same for the config blob.
• PUT: Finally, the client uploads the manifest, marking the end of the flow.

3.2 Tracing an image pull

Similarly, let’s take a look at the pull flow:

$ podman pull --tls-verify=false localhost:8080/busybox:latest
GET     /v2/
GET     /v2/busybox/manifests/latest name=busybox reference=latest
GET     /v2/busybox/blobs/sha256:af3f0f48a24edb84e94aff6f44f5d089203453719d3b2328486d311e61db9b09 digest=sha256:af3f0f48a24edb84e94aff6f44f5d089203453719d3b2328486d311e61db9b09
GET     /v2/busybox/blobs/sha256:a3b655d4356d3eafbfce127e94844901a68074eb6fde905780d46cabf0caf342 digest=sha256:a3b655d4356d3eafbfce127e94844901a68074eb6fde905780d46cabf0caf342

This is relatively straightforward. Client does a version check, followed by pulling the manifest, and from the manifest, it pulls the config and the layer blob; marking the pull complete.

The sequence above mirrors the upload and download flow we described earlier, but now shows actual requests and responses between a client and a registry. Lastly, as a sanity check, we can test if the image we pushed to our registry and pulled from it, works as expected:

$ podman run --rm localhost:8080/busybox:latest echo "it works"
it works

There we have it, our tiny-registry responds to the podman push and pull commands, successfully storing an image that was pushed, and then serving it back to the client.

Since podman by default refuses to talk to registries over plain HTTP, --tls-verify=false allows us to play around with our registry on localhost without the overhead of provisioning a deployment or TLS cert.

3.3 Inspecting our local storage

Lastly, let’s take a look at how our registry/ directory on our filesystem looks like after the above interactions with the registry:

$ tree ./registry
.
├── blobs
│   ├── sha256:216659b5706c007bcf2bcec124a6d32ad2ecbfd9b1921c695e022b7a4a27a0d9
│   ├── sha256:a3b655d4356d3eafbfce127e94844901a68074eb6fde905780d46cabf0caf342
│   └── sha256:af3f0f48a24edb84e94aff6f44f5d089203453719d3b2328486d311e61db9b09
├── busybox
│   └── manifests
│       └── latest
└── tmp

$ cat ./registry/busybox/manifests/latest
sha256:216659b5706c007bcf2bcec124a6d32ad2ecbfd9b1921c695e022b7a4a27a0d9

As we discussed earlier, the manifest, the layer, and the config.json for the busybox image we had pushed are stored under blobs/, with tags pointing to the corresponding manifest digests.

At a high-level, a registry is a thin HTTP wrapper over a content-addressable storage. Adding features or making it robust makes the registry more appealing but the idea remains fundamentally the same.

Limitations

As the name suggests, this is a demo registry. So, while the Distribution Spec defines additional workflows such as Content Discovery and Content Management, we’ve implemented only the Push and Pull workflows. Even in those workflows, we’ve implemented basic functionality for the sake of demonstration and skipped features such as multi-arch support, authentication, content-type validation, etc.

Projects such as distribution implement the full OCI Distribution specification, and act as a full-fledged registry solution.

Conclusion

We built a minimal registry that implements just enough of the OCI Distribution specification to support push and pull commands from any container engine. In doing so, we saw how a registry stores image data—manifests, configs, and layer blobs, and how it serves them via a small set of HTTP endpoints.

The next time you run podman push or pull, you can mentally map the requests to API endpoints and the content-addressed blobs being transmitted and stored by the registry.

I’ve tracked my command line history in a MariaDB instance for the past 5 years. The original idea was to simple keep track of everything I do on the command line so that I can refer to them later on. The utility of the history command was immediately apparent to me and so the logical next step was to store the history in a more sophisticated setup.

I remember I looked around the internet looking for answers, and found this repository. I set up the database as per the instructions with a few changes, and then forgot about it for 2 years. Later, as the year 2022 came to a close, and inspired by the year-end reviews by various applications and services such as Spotify Wrapped, Strava Year in Review, etc, I thought about doing a similar year end review but for my command-line usage using the data I’ve gathered.

The result was surprisingly good. It had good reception with readers, and most of all, I enjoyed the whole process of deriving insights, generating charts from the data, quite a lot. I also recently wrote a 2025, and have potentially commited to a every-5-years cadence to this series.

You can checkout the analyses I’ve done in the past:

• Year in Command Line (2022)
• Year in Command Line (2025)

I thought about adding a Methodology section to the 2025 analysis post explaining how to use the data to create a year-in-command-line based on your own data, if you decide to. But I believe a separate post would be ideal, and I can perhaps go into a little more detail. Let’s get into it.

0. Prerequisites

The setup is quite straightforward, make sure you have the following installed:

1. mariadb
2. zsh or bash
3. python (for visualization)

1. Setup Database

Create a database in mariadb using the queries below. The schema consists of, among other fields, the command, current working directory, and the timestamp.

create database if not exists bash ;
use bash ;
CREATE TABLE `history` (
  `oid` bigint(20) NOT NULL AUTO_INCREMENT,
  `command` TEXT,
  `arguments` TEXT,
  `cwd` TEXT,
  `created` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `tag` TEXT,
  PRIMARY KEY (`oid`),
  KEY `created` (`created`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci ;

3. Shell Integration

The preexec Hook

This is a special function in zsh and is executed after a command is read and just before executing it. The idea with using this hook is to store the read command in our database and then continue on with the execution of the said command. The following snippet achieves this:

Add the following to your .zshrc:

preexec() {
    BASH_COMMAND=$1
    if [[ -z "$HISTORY_TAG" ]]; then
        HISTORY_TAG=''
    else
        echo TAG: $HISTORY_TAG
    fi

    [ -n "$COMP_LINE" ] && return
    [ "$BASH_COMMAND" = "$PROMPT_COMMAND" ] && return

    local cur_cmd=$(echo $1 | sed -e "s/^[ ]*[0-9]*[ ]*//g")
    cwd=$(pwd)

    # Optional: ignore certain commands
    [[ "$BASH_COMMAND" =~ historyMysql* ]] && return
    [[ "$BASH_COMMAND" =~ _pyenv_virtualenv_hook* ]] && return

    printf -v BASH_COMMAND_ESCAPE "%q" "$BASH_COMMAND"

    mariadb -ucli-history -e "INSERT INTO sh.history (oid, command, arguments, cwd, created, tag) values (0, '${BASH_COMMAND_ESCAPE}', '', '$cwd', NOW(), '$HISTORY_TAG' )"
}

You can also tag certain commands if you wish to, for instance export HISTORY_TAG=macOS would make it easier to filter out commands that you ran on your Mac machine.

source ~/.zshrc and that’s all you need to get started. If you’re a bash user, you can use this snippet. I haven’t tried bash but it should work just the same.

To test whether the setup is working, run the following command to verify:

$ mariadb -ucli-history sh -Nse "SELECT command FROM history ORDER BY created DESC LIMIT 1"
mariadb -ucli-history sh -Nse "SELECT command FROM history ORDER BY created DESC LIMIT 1"

The above command runs a query in the history database that we created earlier and shows the most recent command that was stored in it. Since preexec work just after parsing the command but before executing it, we should see the same mariadb command as the output.

4. Backup

Before we get to analyzing the data we will gather, it’s also equally important that you keep this data safe. The most important bit here is our database dump. Keeping a regular backup–I recommend daily–of your database dump file will ensure you won’t lose your terminal history. At its core, dumping your DB is as simple as:

$ mariadb-dump -ucli-history --skip-lock-tables sh> sh_history_dump.sql

You can store this dump anywhere you’d like. If you already have a backup pipeline in place using tools such as restic or rclone, you can simply plug in this command have this dump backed up regularly. If you want something simpler, you could wrap this command in a bash script, and coupled with either cron or systemd-timers, you can automate a daily dump push to any cloud service, or even a Github repo.

Analysis & Visualization

Once you have enough data, ideally an year’s worth, the next step is to analyze and visualize the data you’ve gathered.

For the 2022 analysis, I wrote throwaway python scripts to generate the charts. I thought about the kind of charts that would make sense, the kind of insights I could draw from those charts and any potential actionable items that would be worth talking about.

In 2025, I delegated the task of writing throwaway script to an LLM agent that are clearly suited for such tasks. I’ve packaged that into a simple script in a repository–danishprakash/year-in-command-line–that can be used as a plug-n-play script to generate a lot of charts that can get you started.

Using the script is as simple as having pythong installed, you can then:

$ git clone https://github.com/danishprakash/year-in-command-line
$ cd year-in-command-line

$ python3 -m venv venv
$ source venv/bin/activate  # On Windows: venv\Scripts\activate
$ pip install -r requirements.txt

$ python analyze.py --year 2025

$ deactivate

Once run, the script will create a charts/ directory and generate all of the charts inside that directory. You can refer to the README in that repository to see the different kinds of charts that it creates, and the customization options available for it.

Other uses

History recall

Apart from making for a nice end-of-year retrospective article on your command line usage, the data can be used for other things as well. One really good use-case is for history recall. Paired up with fzf to read the history from your MariaDB instance, you can fuzzy search across years of shell history. It’s extremely efficient if you work a lot on the command line. I used the following snippet that allowed me to fuzzy search through my entire shell history for the past 5 years:

fzf-history-widget() {
selected=( $(mariadb -ucli-history -B -e "select distinct command from sh.history group by command order by max(created) desc, command;" |
    FZF_DEFAULT_OPTS="$FZF_DEFAULT_OPTS -n2..,.. --tiebreak=index --bind=ctrl-r:toggle-sort $FZF_CTRL_R_OPTS --query=${(qqq)LBUFFER} +m" fzf) )
  local ret=$?

  BUFFER="$selected"    # populate input prompt
  CURSOR=$#RBUFFER      # set cursor position at the end of the command inserted
  zle reset-prompt

  return $ret
}
zle     -N   fzf-history-widget
bindkey '^R' fzf-history-widget

But clearly that’s not as efficient and doesn’t scale as you gather more and more data. I now have a modified version where I apply a few filters and limits before fzf shows me the data.

Debugging

On those fateful days when you pushed something that you weren’t supposed to and need to figure out the command to blame? You could search for all the commands you ran between a given time period:

SELECT command, created FROM history
WHERE created BETWEEN '2025-12-15 14:00:00' AND '2025-12-15 15:00:00'
ORDER BY created;

Conclusion

The entire setup takes less than 5 minutes, but the value compounds over time: by the end of 2026, you’ll have a year’s worth of data to analyze, visualize and even optimize your workflow. Or quite simply, recall that obscure command from 8 months ago.

If you end up using this for something interesting, I’d love to hear about it.

:wq

Fifth anniversary of the Year in Review series, and I’m happy I stuck with it for so long. The structure remains the same, barring the change in ordering of the sections that I feel is now more appropriate.

Education

My favorite books from this year, in no particular order, are CODE, City of Djinns, To Kill a Mockingbird, and The Anxious Generation. I also finally found audiobooks to be a viable reading format. I think the reason I had not been able to stick with audiobooks was the kind of books I was choosing.

I’ll divide writing into two. Public and personal writing. Public writing, the articles I share here, followed the trend for the past 3 years–handful of articles. I’ve amassed over 70 topics I would like to write about, and a lot of them are even drafts with either a rough outline or writing points that I’ve gathered over time. I’ve pondered about this while I was writing this article and doing my internal goal review for 2025 and it all comes down to this–“If you want to write, write.”. That might seem simple but it’s the one suggestion I’ve heard repeatedly from people who are good at writing and seemingly great at churning out pieces. I did manage to find both time and motivation to write 2 additional articles. My favorite posts from 2025 are Kinesis Advantage2 and Nepal (2022).

My personal writing, on the other hand, is a completely different picture. I’m now writing more than I have ever before. There are more observations in my journal, more notes in my book notes, more entries in my auxiliary notes. Personal writing has been indescribably therapeutic, so much so that I now pull out my phone to capture a moment, not only by the camera, but by words. The next step would be to carry over this comfort to public writing.

The other pillar of how I learn, beyond reading and writing, is how I retain information, and that has been a bit of a hit or miss. I managed to accumulate a massive Anki backlog when I travelled to Europe in August and have been trying to clear it to this day. This sucks because it means that while there’s a backlog to be cleared, I’m not learning new information. Which in turn means, the notes I take while I’m reading, even though they help solidify my understanding of a concept or an argument, I’m not making an effort at remembering important information I want to remember because I’m not actively injecting anything into Anki.

Health

This year has been the worst when it comes to consistency surrounding health and fitness. I tried to understand the cause of decline in fitness this year and this is what I found. Lack of routine and travelling; I have a hard time maintaining my physical fitness routine when I travel, not to mention sleep takes a massive hit. This year I travelled comparatively more than previous years and that showed up in the various metrics I track.

Second was dealing with injuries and moving back to Delhi. Things were going well till the air quality dropped down to hazardous levels in October. I had to stop running, productivity dropped, focus took a hit. It didn’t help that I met with a lot of injuries as the year came to a close, thanks in part to Delhi winters.

It was shocking, though not surprising to see the metrics tell the same story. The graph above shows the Resting Heart Rate (RHR) Baselines. So while I was in the fair category and was well on the way to improve my cardiovascular health by getting into regular running, life happened and here we are. But I’m hopeful for what’s to come in this new year.

Lifting was also somewhat consistent except for when I was travelling and was recovering from injuries. I gained considerable muscle this year, and as is the norm, a lot of fat along with it as well. On the nutrition front, I stopped tracking my macros as rigidly as I was these past 3 years but will shortly get back to the grind once I start losing weight. I think I have a strong baseline for nutrition entrenched that allows me to stay the course even when I’m injured or when I’m travelling.

Career

First half of 2025 wasn’t as exciting as the second half, unfortunately. I spent a lot of time doing meta work, which even though is necessary, wasn’t challenging. I managed to spend slightly more time in the second half on a few long-pending upstream bugs I had been working on which was a welcome respite. I wish and have planned to do more of this in 2026, in a more structured manner.

It’s natural I speak about AI in this section. Adopting AI for technical work has been the biggest change I’ve ever done to my workflow. There was a time this past year when I had to use Cursor, and while its agentic capabilities were good, I was disappointed to work outside the terminal, after roughly a decade of working within tmux sessions, so naturally, I rejoiced when terminal based agents particularly Claude Code came out and started gaining traction. Since then, my workflow has changed dramatically, and I’ve found myself using these tools more and more. I’m still programming, still debugging, still solving problems, but with some help in the gruntwork and research department. It also doesn’t hurt that my organization is bullish on AI for developer productivity, I’m excited to see what the continued and rapid pace of AI means for my career growth, and how this section evolves over the years.

Travel

If you’ve been following along so far, you might’ve noted that travelling has had a slightly negative impact on my educational and health goals. I don’t normally follow routines or rules while travelling and so it’s inevitable that travelling would throw off some goals that require deliberate effort and focus. Perhaps the idea is to figure out how to not completely derail while travelling, or more importantly, after coming back from a vacation.

It so happened that I condensed all my travels in the second half of the year and as much fun as it was, it was also slightly exhausting. I visited Mangalore, Bhutan, Thailand, Kazakhstan, Europe, and Jaipur.

I had always planned to write more travelogues but one way or another, they tend to get deprioritized and eventually out of my to-write-about-topics queue altogether. For the time being, here are some photos as consolation:

Focus

Moving back to Delhi in a new environment, instead of improving, the idea was to not regress. With that in mind, there weren’t many drastic changes I made this time around in order to improve focus and productivity. Things remained more or less the same as I left them in 2024. I still maintain the practice of undirected consumption on Saturdays, and that’s about it. Though I must confess–as I’ve done before–that this type of consumption is a complete waste of time even if it’s limited to just one day of the week. A more ambitious goal would be to eliminate it entirely.

I’ll continue to use and rely on the techniques and processes I’ve learned over the years–pomodoro, checking messages only twice a day, cognitive-heavy morning routine, etc. But at the same time, as a self-proclaimed self-improvement junkie, I’ll continue to look for new ideas and additions because honestly, I still struggle with lack of motivation, productivity and focus quite frequently.

Conclusion

This wasn’t a groundbreaking year by any measure but it was a year of change, and, as they say, change is inevitable. Of course, there are many changes that happened this past year that don’t quite fit here but they all shaped me, and I’m hopeful for what 2026 will bring. I’m still working on my goal setting for 2026 and the idea is to incorporate incremental and realistic improvements that build upon the foundation, and to maintain existing systems.

It’s been roughly 5 years since I started capturing my shell history in a MariaDB (MySQL originally). I had a vague plan to derive insights or optimize my shell usage from all the data I would capture. And sometime in 2022, I decided to sit down, write some SQL queries and rather forcefully, derive insights from my shell usage.

This is the second iteration of Year In Command Line. I hadn’t originally planned for this particular post but a friend suggested I make Claude do the heavy lifting by writing SQL queries and generating graphs, and I can focus on writing the actual article. So let me preface this article by saying that all the queries and graph generation scripts were written by claude, and while I did review most of them, there might be some issues though unlikely. All in all, it turned out to be fairly convenient, and fun.

The structure of this post remains the same as earlier. I will go through some high-level insights, and then dig deeper into some of the most used commands.

Data

Starting in January 2021, I now have 5 years worth of shell history in a MariaDB (MySQL originally) database. As of December 2025, I’ve gathered:

• Total commands (all time): 145,415
• Total commands in 2025: 29,120
• Distinct commands in 2025: 7,411
• Total commands in 2022: 21,475
• Distinct commands in 2022: 5,060

This post will only use data for the year 2025. A new section comparing the insights between the previous and the current iteration is added later in this post. Let’s start.

Usage

Weekly Usage

The year didn’t start off CLI-heavy but then it picked up. There’s a positive upward trend throughout the year. The most notable dip just before October was when I was travelling. I mostly did meta work while I travelled, preparing slides for a conference I was attending, and spent little time on the command line.

One thing that surprised me is how the adoption of AI didn’t seem to show up as a data point, or even an outlier. I started using agentic editors sometime in June 2025 and even though my usage mostly revolves around editing, reviewing and understanding code, I was prepared to see a sharp dip and a subsequent downward trend for the rest of the year.

Day of the week

Looks like the blues have been hitting me hard. I recall Monday being the highest in weekly frequency but it seems to have taken a slight dip. Though I will chalk that up to meta work and catching up on communication in the past week.

The fall on weekends is comparatively sharper, Since 2022, I’ve stopped spending as much time on the command line on weekends as I used to. This is a bittersweet revelation because I genuinely enjoyed hacking on personal projects over the weekend. Perhaps life happened, or I just involuntarily wanted to step away from the machine after spending the whole week in front of it. Or it could simply be me spending more or less the same amount of time, but not on the command line. Though I’m afraid it’s very much the former.

Hourly Usage

This is fun, mostly because the graph clearly depicts the change in my routine between 2022 and 2025. Starting January 2023, about the same time I started at SUSE, I also moved to an active-evening routine. This means I no longer hit the gym in the morning; instead, I spend my time on cognitive tasks–mostly reading, but also researching and often getting a head start on something important and interesting at work.

So the graph shows that I now start working much earlier than I used to back in 2022. And then after a short dip around 6 (that’s when I now hit the gym or go for a run), I’m again back at work till dinner. Not ideal but the post-activity productivity rush is seemingly real, for me.

Activity Heatmap

This graph is a new addition. It shows the activity heatmap and shows the day-time combo when I was the most active on the command line. I’m slightly skeptical of the insights here because Wednesday is the day I have most, if not all, my meetings and so for Wednesday noon to be the most active seems weird.

Saturday is also interesting, that’s the day I spend on catching up on pending personal digital chores. Writing an article, book review, or hacking on a side project, etc. It’s nice to see the graph aligning with that.

Top Commands

Moving on to the actual commands, the following treemap shows the top 10 commands of 2025, this doesn’t include coreutils or built-in commands, those are discussed in the next section.

There are quite a few things going on here:

1. git remains the most used command for presumably 5 years in a row (though I haven’t checked that specifically), and I guess this will be the case for years to come–the Lindy effect.
2. The biggest change is the disappearance of kubectl. It used to be the second most-used command after git. I used to work extensively with Kubernetes at my last job, but I’m now focusing more on container technologies and while I’ve played around with k8s now and then, I’ve effectively moved away from actively using it.
3. Replacing kubectl is osc–the command line interface to work with Open Build Service. At SUSE, I maintain quite a few container packages and the lifecycle of those RPM packages are managed via OBS, and interfaced via the osc tool. This data also includes isc along with osc, the private OBS instance we use at work.
4. I’ve been working with Podman quite extensively at SUSE, contributing to upstream, and maintaining the package downstream. So it’s natural for podman to make an appearance in the top commands list. I still work with docker so it’s also right up there along with podman.
5. claude makes an appearance but I’m surprised at the mere invocation of a long-running interactive process like claude being triggered so many times that it made the list. I will talk more about the role of AI in this context in a later section.
6. Finally, skiff–a container image inspection utility I co-developed with Dan. It can help you find duplicate layers, inspect the diff between two images, or find the largest files by size in an image, and more. Feel free to check it out.
7. I’m a little embarrassed at the presence of brightnessctl in this list. Whenever I’m not using my laptop in clamshell mode, I use brightnessctl to configure the brightness of the laptop display. I admit that I couldn’t be bothered to figure out how to make the fn keys work.

Top Coreutils

There’s little to be said for this graph, however there are some interesting insights:

1. vi for some reason exploded in usage, I’m not quite sure what’s going on here. I’ve vi aliased to nvim and have been using neovim for the past 9 years. But I wasn’t expecting any real change here. One hypothesis is that I’m spending relatively more time dealing with ephemeral tasks such as package maintenance, doc writing, along with writing code and that usually begins with me firing up neovim.
2. I’ve mentioned this before but I’m a big fan of z utility. It helps you switch to directories based on weightage and frequency and is smart enough to not require the whole directory name, let alone the full path. It’s extremely convenient, so much so that you can see it as the 4th top command in the previous section. So with that being said, it’s interesting to see cd taking the 4th spot in the standard commands.

Not much else seems to be going on apart from the fact that the reliable coreutils are still alive and kicking, what wonderful pieces of software.

Git

There are 3 new commands that made it to this year’s list, namely:

1. format-patch: Some projects that I contribute to now collaborate over email. So I generate patchsets via format-patch to share over send-email and to backport some of them for downstream work.
2. worktree: This is purely a function of AI introduction in my development workflow. It allows you to manage multiple working trees (separate directories) that are parented to the same repository. So you can work on multiple features/bugs in the same repo in parallel.
3. switch: One of my friends introduced me to this command. It was introduced to clear the confusion around the multi-use checkout subcommand, and can only be used to switch and create branches.

I still incessantly revert to hitting git status on a repository as a kind of mental reset. In the same vein, git diff and git l are also almost reflexive when I’m working and they provide me with a bird’s eye view of what’s what in the current repository. I guess I can shorten git diff to git d and save those precious ms.

It’s funny how workflows change over time, I thought I would only use a handful of git subcommands forever but git is such an exceptional and complex software that there are increasingly new features and concepts that I’m being made aware of every now and then. I still remember the time I learnt about git rebase during my first internship and it was a game-changer. I think of worktree with the same level of appreciation and admiration now. The powerful ability to work on multiple branches in the same repository is underrated, especially with the advent of AI. So while the basic git commands still dominate its usage, it’s exciting to learn new ways to work with git repositories in the future.

Docker

Docker usage has reduced a little, given that I’m focusing more on Podman. Docker still finds use as a lot of applications still default to docker as their container engine of choice. More often than not, this comes by way of Docker Compose since it makes it easier to distribute a container stack for applications. Additionally, I use docker as a reference as I develop for and contribute to podman. It’s also funny how a few typos also made it to the list.

Podman

Podman has been one of my primary focuses this past year. I’ve developed various features, debugged issues, backported security patches, etc. run takes the lead because that’s been the entrypoint to testing most of the features or bugs I work on, --push, -p, etc. Please note that podman here also includes ./bin/podman or the development build.

Beyond run, the distribution is fairly linear. The workflow is quite simple, run the containers, play around with them, check status using ps, and clean them up via rm.

Switching Directories

There are many ways to switch directories on a command line. There are inefficient ways, and then there are more efficient ways. I’ve mostly grown habitual to using z to switch directories majority of the time, it’s a nifty little bash program that does its job really well, highly recommended.

I’m also a big fan of swapping directories back and forth using cd -. A lot of other tools also support this, for instance, git switch - allows you to switch back and forth between recently used branches, it’s a great way to save time.

Lastly, modern shells also support switching to the parent directory using .. and pathname tab expansion. These are helpful at times but as you can see in the split above, z takes care of almost half of the directory switching.

Evolution and Reflection

Looking at the year-on-year trend from when I first started tracking the data back in 2021 to now, we can see the dip in 2022, that was the time I got into a little bit of meta or leading the project work, so there were lots more discussions, planning, and delegating. End of 2022 was me switching jobs so there’s that. Post that, there’s steady increase in the total commands graph.

It’s interesting that the distinct commands and average commands per day remains more or less same(there’s slight fluctuation in both but are dwarfed due to the scale of the graph) through the past 5 years. The distinctions in the metrics only starts happening once you step into them individually as we’ve discussed earlier in the article.

2022 vs 2025: What Changed?

I think the biggest change is the disappearance of kubectl and the adoption of podman. You can see the spike in usage of podman, and at the same time, how docker and podman converge in numbers.

Open Source

The major change in my workflow after 2022 came in 2023 when I switched to SUSE. I ended up doing a lot more Open Source work–podman, openSUSE package maintenance, etc. This is reflective of tooling changes such as the high usage of osc.

AI

Lastly, adopting AI in 2025 has been the biggest change I’ve had in my workflow in the past 10 years. I was initially not sold on the idea but it really shines if you know how and when to use it. I had fully expected AI to reduce raw command count but that wasn’t the case. This review would’ve been quite different if it weren’t for AI. For instance, git usage wouldn’t have dropped or vi usage might’ve been higher.

Conclusion

It’s a lot of fun for me to go through data and charts to derive insights into one’s workflow. While there aren’t many actionable optimizations to extract, it’s interesting to see how my workflow has evolved over time with changing roles and requirements–a shift from Kubernetes to containers in this case, the surprising absence of AI’s impact on metrics, and a bittersweet decline in weekend hacking.

Doing this review every 5 years seems to be the sweet spot, and it will be interesting to see what changes might show up in 2030 with the development landscape changing every few months these days, thanks to AI. I also plan to write a small post on the methodology behind this activity soon.

Beyond the numbers, this exercise serves as a time capsule, a snapshot of not just what tools I used, but also how I worked, when I worked, and what I worked on. I’m a big believer of the concept of compounding, and I’m excited to see the trends over the course of decades.

:wq

I got the Kinesis Advantage2 Quiet LF (the Cherry MX Silent Linear Red version) from a friend of mine who was visiting the US. It’s not cheap by any standards especially if you plan to have it shipped outside of the US from Kinesis themselves.

First impressions

The keyboard felt extremely light and hollow which is a common sentiment amongst KA2 users. It is divided into two splits, the left and right wells, as they call it. These keywells allow you to keep your hands in a naturally rested position without any undue strain on your wrists. There are two thumb clusters that make use of your thumbs more efficiently, unlike traditional keyboards where both the thumbs tag team on the single long spacebar key. This allows you to quickly hit Return, Backspace or Modifier keys without moving your thumbs much. It’s a great concept that is used in various different kinds of keyboards, some implementing it even more efficiently than Kinesis, the Glove80 for instance.

The keyboard is also tented at 20 degrees allowing a more comfortable and natural position for your arms to rest. There are two wrist pads provided but I have since stopped using them. The keyboard connects to your device via a USB-A connection and connects directly to the controller inside the keyboard. You get a few replacement keycaps and a keycap puller alongside.

Onboarding

The first two days were brutal. I was down to 30 WPM from ~100 but it was extremely frustrating though I knew it would get easier, and so I stuck to it. I was back to 80 WPM by the end of the first week and completely at home by the end of the second week with 100 WPM. Moving to a split keyboard meant getting used to a new way to use modifier keys. I had to move my frequently used workspaces to the left well numeric keys of the keyboard allowing me to switch the workspaces with only the left hand–I had also remapped the modifier key to be on the left thumb cluster.

It was a bit jarring to hit the parentheses and tilde characters that were ever so inconveniently placed at bottom left and right corners of the keywells respectively. My pinky finger still struggles at times to press them as part of a long sentence or statement that I’m writing, forcing me to pause and take a note of where the key is and whether I’m hitting the right key or not.

By the third week, I was hitting typing PRs. I did 121 WPM with 100% accuracy. This, however, might not be representative of whether the KA2 helped in increasing both my typing speed and accuracy. I believe it was purely because I was trying out the new keyboard, I was more inclined to do typing tests more often throughout the first few weeks. Had I been doing them regularly over the years, I would’ve achieved these results even on an older board. But that being said, I can confidently say that typing was extremely comfortable on the KA2 compared to my older boards and the one on my Lenovo ThinkPad.

Programming the keyboard

The KA2 doesn’t support QMK for programming, it ships with SmartSet. You can remap keys, define macros and have it configured and stored on the keyboard itself. It’s more than enough for daily use, and I’ve been using the following config for the past 2 years:

...snip

[escape]>[lctrl]
[caps]>[rctrl]
[kp-V]>[cbrack]
[kp-C]>[obrack]
[kp-right]>[kpshift]
[lctrl]>[escape]
[delete]>[rwin]
[bspace]>[enter]
[rctrl]>[rwin]
[enter]>[bspace]
[rwin]>[enter]
[pup]>[delete]

...snap

Notable changes include:

1. Remapping Caps Lock to RCtrl, that’s one mapping I do on every system and keyboard regardless. It allows me to hit escape in vim (and in applications with vim bindings) quickly with (Ctrl+[) and often acts as a secondary modifier key for a lot of programs I use e.g. Firefox, Obsidian, Slack, Spotify, etc.
2. I swapped around the keys in the thumb clusters as well eg. Backspace <> Enter. This was purely subjective, I just found it easier and more intuitive to hit Enter with my left thumb rather than my right, YMMV.
3. Activate keypad layer and use HJKL as arrow keys. Since the arrow keys on the board are integrated in the bottom-most layer, it helps to move around while keeping my right hand on the home row.

It took me a while before I arrived at the above configuration which has served me well so far. Clearly, I wish it had QMK support so I can go a step beyond what SmartSet allows.

Long-term experience

I’ve been using this keyboard almost exclusively for the past two years at the time of writing this article and so I think it’s a good enough time to evaluate on some aspects of this keyboard. I’ve seen quite a few reviews when I was planning to purchase this keyboard that reviewed the keyboard in just 3-6 months of usage and I feel that’s not enough time, so here goes.

RSI and comfort

For starters, I’ve not had any strain related issues so far. If anything, it has gotten more comfortable as I slowly got used to the split and keywell layout. I know it’s not for everyone, and in fact, there might be better and slightly cheaper options available but I think I’m settled with this keyboard for the foreseeable future. So much so that I’m thinking of getting a secondary backup board in the unfortunate scenario where my primary board dies, more on that later.

Build quality/durability

As I mentioned earlier in the article, the board when I first held it, felt extremely light and hollow. Anybody I show this board to has the same reaction. But it hasn’t really been a bother when it comes to using the board. Sure, the hollow sound feels cheap and flimsy but there are workarounds for that which we’ll discuss shortly.

I also haven’t dropped it or banged it against something yet to tell you how sturdy it is but I’ve seen people using this keyboard for 10-20 years so I guess it’s sturdy enough.

Portability

This keyboard is not meant for travelling. There are cases available somewhere and people do carry it around but it’s simply not meant for that. Even though it’s light, the keyboard itself is huge enough to be a nuisance to carry around. I’ve never traveled with this keyboard, never intended to in the first place and probably never will. I’m more than happy if I get to spend 80% of my time with this keyboard, and manage the rest 20% with the laptop keyboard whenever I’m traveling.

I was afraid I won’t be able to use the standard keyboard at all after switching to a split, but that’s not the case. It’s like riding a bicycle, even after a long hiatus, the older standard keyboard layout just clicks, and you’re good to go. As I said, that doesn’t mean I would switch completely to a standard keyboard layout.

Price value

For me personally, the keyboard is value for money, even after considering the cost of getting it to India. If your job relies on you typing and your standard keyboard is giving you wrist issues, it might be worth giving a split keyboard a shot. It’s a shame that it’s not possible to try such keyboards before you can buy them, because that’d be a lot more convincing for a lot of folks who are on the fence about buying a split keyboard.

Modding: KinT and QMK

Sometime at the end of the two year mark, I started experiencing ghost typing with my KA2. At random, it would enter a barrage of seemingly random characters at lightning speed, rendering the keyboard useless. I couldn’t type or do anything, and would scramble around to save the work I was doing, or at times, prevent the sentient keyboard from sending messages to my work Slack on my behalf.

I reached out to Kinesis and they suggested a few troubleshooting steps. I tried thoroughly cleaning the board inside out, resetting it, and reseating the ribbon connectors connecting the keywells and function layers to the main controller, but all in vain. It was concluded that the motherboard was at fault and changing it would cost me upwards of $150 excluding shipping and RMA costs. I had earlier known about the KinT mod by Michael Stapelberg for the KA2. It’s an open source replacement board for the KA2. I quickly did some napkin math to confirm that the cost of all the parts needed to mod my KA2 with the KinT mod would be slightly cheaper than getting a new motherboard from Kinesis. It was also the week SUSE held its annual hackweek, so the timing was just right.

KinT

I purchased all the required parts mentioned in the project BoM, and soldered them onto the PCB. It had been a while since I had done soldering, so while I was able to solder the majority of the board (MOLEX, etc.), I had to take a little bit of help to solder the Teensy microcontroller onto the PCB. I inserted the new controller into the keyboard, and after a little bit of resoldering, tightening the screws and reseating ribbon connectors for the umpteenth time, I managed to flash QMK on the Teensy and voila! I’ve been using the keyboard with the new controller for roughly 2 days and haven’t had any ghosting issues yet.

QMK

The KinT project uses QMK as the firmware of choice. I had heard about QMK and the amount of flexibility it allows when it comes to configuring your keyboard. I was quickly able to recreate my Kinesis SmartSet configuration remappings into QMK-supported configuration. QMK supports quite a few super nifty features such as Mod-taps, Tap dance, HRMs, and key combos. Looking forward to trying some of these soon.

I’m extremely happy with the mod. I would never have attempted it if the original board hadn’t gone bust. I’m glad it did. My second backup board would also see this mod as soon as I get my hands on it, the QMK support alone is worth it.

Challenges & Future improvements

Even though this keyboard is perfect (for me), there are still challenges that I’ve faced with it and improvements I wished it had.

Support

If you’re based outside of the US, getting this keyboard fixed would cost a fortune. I was bit by this problem and it wasn’t as easy dealing with it compared to other keyboards I’ve held in the past. From what I’ve read and heard, support is not a problem in the US where Kinesis takes care of everything for you if your board is still under warranty. Even if it is not, getting the parts and paying Kinesis for the RMA would still come out cheaper and reasonably affordable imo.

Switches

As much as I like the linearity of the Cherry MX Silent Red switches, I don’t like how they sound at all. In the past I’ve tried Cherry MX Browns but while they sound decent, the actuation force on them isn’t for me. I might experiment with Gateron Reds or even a clicky switch. The KA2 doesn’t have hot swappable switches and from what I gather it uses diodes for the switches, so it might not be as easy as other keyboards. Unfortunately this also means not being able to experiment much, if only there was a hotswappable mod for the KA2.

At the same time, I also plan to add some weight and dampening to the board. It’s an easy hack that would help improve the sound profile of the keyboard.

Keycaps

I have remapped most of the modifier keys and have been touch typing for as long as I remember, so I don’t see the point of having printed keycaps, they’re just a distraction at this point, and besides, blank keycaps look good. Apart from the blank keycap set from Kinesis, the DSA profiles fit almost perfectly, so that might be the next best option. Though I am a little worried about what happens when I clean the keyboard by taking the keycaps out at some point in time.

Bluetooth Mod

I like a clean table and have pondered about modding the KA2 with the amazing bluetooth mod. It can perhaps make for another post detailing the procedure.

Closing

I’ve been meaning to write this post for some time but the recent issues with the board, and the subsequent revival excited me enough to finally write it. My experience so far with the Kinesis Advantage2 has been great, it’s the most comfortable keyboard I’ve used and I wish to write more about it, perhaps after hitting another milestone.

Why did you start blogging in the first place?

I was fascinated with the idea of having a personal space on the internet. Early on when I started programming and getting active on the internet, I saw a couple of highly personalized websites from some smart folks and that led me on a mission to set up a website of mine. Next thing I know, I had a barebones html/css makeshift page up and running.

At some point, I got interested in how a shell works, so I read various articles about how they are implemented. Once I had a fair bit of understanding of what it was, I simply thought why not write an article about it and publish it on my website? And I did. Over time, some folks reached out to me over email and otherwise asking me about the post and the website, and that sort of got the ball rolling.

What platform are you using to manage your blog and why did you choose it? Have you blogged on other platforms before?

Not counting the makeshift CSS/HTML page I first had because it wasn’t suited for blogging, my first actual blog started off with Jekyll because it had the lowest barrier to entry for a static website generator back then and was quite popular. Coupled with Github, getting the site up and running took me under an hour.

Sometime in 2022, I wrote a static site generator and moved from Jekyll to kizai for this site. I realized Jekyll offers so much that I don’t really need and so I made the switch in order to simplify my setup. I also moved from Github Pages to Netlify, and later to Cloudflare sometime last year.

I’ve tried Wordpress in the past but used it to share warez links(a story for another time perhaps) and some old design stuff I used to do when I was in high school. But it never really stuck with me. There was also a time when I tried to experiment with ghost but abandoned that pursuit relatively quickly.

How do you write your posts? For example, in a local editing tool, or in a panel/dashboard that’s part of your blog?

I use Obsidian for ideation. Whenever I have an idea for a blog post, I create a page for it in Obsidian and jot down my thoughts there. Obsidian has a mobile app that has good sync functionality. This lets me jot down ideas anytime—while traveling, reading, or even at the gym..

Once I feel I have enough substance, I move to my blogging setup. I create a new markdown file in my local repository for my site, and start structuring the first draft. I use (neo)vim for writing, and have a light monochrome theme I created for writing specifically. I keep the draft for a post in a separate branch so I can write multiple posts concurrently, if I’m feeling particularly passionate about writing. This has been working well for me so far.

When do you feel most inspired to write?

I generally like to write on weekends especially during the mornings with a cup of tea/coffee. And if the topic I’m writing about is engrossing enough, I might continue writing till late evening.

Do you publish immediately after writing, or do you let it simmer a bit as a draft?

I don’t publish immediately, far from it. Once I’m done with the draft, and given that I host my site on Github, I raise a PR and ask some folks to review it if it’s a post that would benefit from a review. For instance, a technical post or a travelogue with a friend who can take a look at the post and point out any inaccuracies in my recollections.

Many of my ideas simmer for years before I write about them. I have many ideas for posts that start with “X years of doing Y”. I have this notion of spending time on something and then writing about an experience. For instance, I have a post on budgeting that I have in ideation phase now for roughly 5 years which I wanted to only publish when I’ve personally done budgeting for 5 years at least. I feel skin in the game is important and that age gives credibility to an experience.

What’s your favorite post on your blog?

I don’t have any favorites, but If I had to pick one, I would pick Build a Container Image from Scratch because I spent lots of time on it to get it right and was quite happy how it came out in the end.

Any future plans for your blog? Maybe a redesign, a move to another platform, or adding a new feature?

For me, redesigns are inevitable. I have spent countless hours fiddling around with CSS trying to get that perfect design only to make further tweaks a few days later. I’ve written about my design philosophy for this blog in the past if you’re interested. Specifically, though, I’m focused on improving the typography on the site, so much so that I just made some typographical changes while I was writing this post, but I’m still not content.

Besides redesign, I have some plans for extending kizai with new features for convenience and for supporting new content on the site for instance, having support for private posts, and using YAML for data sourcing (markdown tables are an abomination). I also have some plans to tweak the reading page to add more metadata in order to get more insights into the reading I’m doing.

Finally, I’ve been planning to add a /cooking page for a long time now. Even though I don’t plan to make it accessible from the main site, I would like to use it for sharing recipes and other cooking-related “nuggets” of information that I want to share.

Who’s next?

Aditi, Anirudh, Arpit, Hari and Priyanka—you’re up!

:wq

Flew from New Delhi to Kathmandu, it was a relatively smoother ride. We got some good mountain views as we approached Kathmandu. You get out of the airport and it’s breath of fresh air–quite literally–fortified on one side by the lush green mountains compared to that of Delhi’s smoggy skyline surrounding its airport. Immediately outside the airport, we got into a haggling match with a cab driver who then took us to our hostel.

Kathmandu -> Pokhara

First impressions of the city weren’t much different from that of let’s say Delhi or any other dense Indian city. The roads were narrow, cars smaller, people everywhere. We reached our hostel and then headed out for dinner in the evening. This is where I had the best momos I’ve had in life, jhol momos. A soupy delicacy that I just couldn’t get enough of during the whole trip. We made efforts to try the different kinds of Jhol momos at every place we found them at.

Our plan was to go to Pokhara first and then come back to explore Kathmandu. So the next morning, we went to the airport. Bought the tickets from the airport as if it was a bus depot—replete with passengers scrambling around to buy tickets, barely any security, extremely delayed flights and virtually no organisation whatsoever. The regional airport was as barebones as it could possibly be. Finally, after the usual airport process, I took one of the most scariest flights I have ever taken in life. Aboard the Yeti Airways YT-677 (ATR-72 IIRC), everybody excluding the locals held their breath as the pilot did dangerous maneuvers before finally hard landing us at Pokhara airport. It was intense. The airport was a relic in itself, you deboard the plane and you huddle together in a small shed close to the aircraft, collect your bags and take the taxi, all within roughly 500m.

You can see the airplane right in front, a stone’s throw away from the “airport”. Just behind me is the taxi stand, so in a way this was the quickest deboard-to-taxi experience I had but it was also arguably the most terrifying flight experiences I’ve ever had so far. Not more than 6 months from when I took this flight, a plane crashed at the same route before landing at Pokhara from Kathmandu, reading about that crash gave me chills.

Pokhara

After some haggling, we too a cab to our Hostel. Pokhara stood in stark contrast to the chaotic Kathmandu, people seemed relatively relaxed, pace was slow, roads were relatively empty and the air was fresh. Our hostel was close to the Phewa lake with paddy farming around our premises looking onto the Phewa lake. It’s funny how I’m able to recall how I exactly felt at the place looking at the picture below roughly 3 years later. The chill early morning breeze, the sound of the hills further out back, and the stillness of the water. It somewhat resembled the slow life Thoreau came around to appreciate while living next to Walden, you could spend your day looking out the window, or sitting by the lake and that would still constitute a day well spent. Which seems a bit hard to comprehend nowadays where there’s a risk of being chastised if even a single minute of your day goes by unproductive.

Guzzled down a traditional Nepalese lunch along with their local wine–Raksi–and then took the local bus to Baidam Road, downtown Pokhara. The area lined up with fancy cafes, souvenier shops and shops offering adventure sports to the visitors of Pokhara made it a pleasant walking and shopping street where all the action in Pokhara happened. We had planned to do a mountain bike tour in the hills surrounding the Phewa lake. After getting our bikes, and the relevant equipment, we took off for the route, and as luck would have it, the weather turned cloudy sparing us the wrath of the summer sun.

We biked on the road that ran alongside the lake till we reached the foothills and then meandered into the forest. The route was a mix of passing through complete wilderness–not untrodden, there was a clear trail but nobody around for the most part–and then passing through local houses and shops. We were trying to time our return but it so happened that it got dark and we were lost. If it weren’t for that ex-armymen who was out cycling that same route as his daily routine, I might’ve much more gory details to write about. But then we returned the bikes, had dinner on Baidam Road, did some shopping, and returned to our hostel.

Woke up the next morning amidst the clouds, it was nothing short of magical, so we tried to capture the perfect shot of the cloudy hills with the Phewa Lake in front, but in vain. But it was the perfect weather, and it rained shortly afterwards which was expected. We had planned to do sightseeing today, so we hopped on the local bus again, deboarded on Baidam Road and rented a scooty.

We headed to Devi’s Fall and Gupteshware Mahadev Cave first. It was raining through and through the whole time but that did not deter the crowds. The rains at least allowed us to take some photographs with slightly higher saturation.

We then headed further up to World Peace Pagoda, it was extremeley serene. The only “noise” was the raindrops sputterning around on the cold marble floor. Part of the charm was it was built at a high altitude lending to the cooler and pleasant climate, and partly because the place itself was quite and well maintained.

While riding back to the town, we got caught by the traffic police. The person we rented the scooty from specifically said Indian driving licenses are valid in Nepal and it said something similar on the internet too. But the policemen was adamant and just wouldn’t let go. He kept the registration certificate of the vehicle with him and asked us to tell the owner to collect it himself. Upon reaching the rental place, we got into a huge altercation (verbal) and after paying him for the “damages”, we were on our way, on foot back to Baidam Road.

Putting that single bad experience with the locals aside, we went to this place for dinner by the Phewa Lake–everything happens around Phewa Lake in Pokhara–that had good food, a good environment and live local live music. That was the end of our time in Pokhara, a city I would love to come back to to spend some quality time but by road and not by flight…or so I thought.

The next day as we checked out of the hostel, the adventure sports shop guy called us and asked us whether we want to partake in paragliding. So next thing we know, we were in a cab hailing us from Baidam Road to the top of a hill.

Pokhara -> Kathmandu

The paragliding was a good experience overall but it also meant we were late for the last flight out of Pokhara for the day. We did squander around the airport trying to see if another flight would be scheduled but nope. The next best option was to take a local cab from right outisde the airport to Kathmandu.

So we hopped onto the most normal looking cab and took our seats at the back. It was a 6-seater cab, but the resourceful cabbies loaded upto 10-12 people inside the cab. It was packed to the brim with adults, kids, and lots of luggage. We weren’t feeling so good and as luck would have it, a local doctor was also sitting next to us. He slipped us some tablets that he said were for headache. And next thing I personally remember is getting off in Kathmandu at midnight. I was dazed (and later confused) through the whole journey. It was only later that I came to know that the journey was arduous thanks to the terrible roads and that we even had an accident on the way which caused such a delay. Bottomline, there’s no good way to travel between Kathmandu and Pokhara, unless of course you’re driving on your own.

Kathmandu

Fun fact: my friend carried a frog in one of his spare shoes in his bag from Pokhara(probably Phewa Lake) to Kathmandu. Poor lad must’ve been overwhelmed from the chaos of the city.

Back in Kathmandu, we stayed in the old town area, and witnessed remnants of the destruction from the 2015 Nepal earthquake that rocked the country. The Patan Durbar Square was being renovated along with other structures in and around Durbar Square. But the residential parts of the old town area were exactly that, an old town. You could enter an alley, discover a small temple, while old folks are sitting next to the temple, having a chat or a smoke.

We spent the last day sightseeing the city; Swayambhu Temple, Buddha Stupa followed by shopping at the Bazaar near Durbar Square. And just like that, the day ended, and so did the trip. We took an early morning flight the next day from Kathmandu back to Delhi, of course we had Jhol Momos before we left.

Closing

I have a habit of jotting down observations when I travel, sometimes even in the most mundane of places and sometimes when there’s nothing to observe. For whatever time I spent in Nepal, I noted how everybody was happy for the most part. But as the old adage goes, as a tourist, you only experience the good parts of the place you’re visiting. But to lend some credibility to my observations, a taxi driver in Kathmandu told me that while you would find poor people here, you woulnd’t find people who are sad. My view of Nepal, shaped from the books and media coverage about the country I’ve come across admittedly conditioned me to have a dismissive view about Nepal as a country but It’s safe to say that’s not the case anymore, in fact, I’m much in awe of the country and the people and would hope to go back again.

It’s been roughly two weeks since I actually started the draft for this article. I guess I wanted some more time to gather what has happened this past year, summarise it and put it in words here.

Personal (Health & Fitness)

2023, in hindsight was quite good in this category. 2024 started out with a serious ankle injury while playing badminton which meant little to no movement for a good part of January and some of February. I did continue to lift throughout the year when I wasn’t travelling or out of town. I reached my bodyweight goal of 72kg but realized it’s not cutting it, and upped the goal weight to 75kg. But this meant that I wasn’t able to achieve my goal of reaching sub-15% bodyfat.

In terms of sports, I mostly played badminton and started running as part of my ankle injury rehab. Nutrition was quite a success, I continued to cook and track everything I ate and that has been quite helpful in the whole journey, I plan to continue doing that for as long as feasible.

Career

Work has been fulfilling, primarily because of the ownership and flexibility I have in my current role and the good set of people I get to work with. My work still primarily revolves around open source software, and occasionally I get to solve problems that are interesting. Another goal I was able to accomplish from last year was to speak at a conference, I spoke at Open Source Summit 2024 talking about container image internals.

That being said, I can objectively say that the amount of code I’ve written this past year hasn’t been much and that’s unfortunate. I’m already planning so that things are not the same in 2025.

Education

On the reading front, I continued to explore different subjects and authors. Some of my favorite reads from last year were Einstein, Linux System Programming, and The Last Lecture. I seem to have finally settled on a note-taking system for my reading, of course, it’s nothing as extravagant as incremental reading but it does the job and does it well for me. So far, all my reads from past 2 years have proper notes and every once in a while I create flashcards from those for long-term retention of certain topics.

Writing is a bit hard to describe or justify. Looking at the archive here, it’s clear I wrote 2 articles in 2024, one of them being the review for 2023. But I’ve done a lot of writing that I haven’t published yet. This includes my reading notes, personal notes, work notes, and other auxiliary topics that I find writing about fun. Then there are article drafts that I haven’t decided to publish yet because I plan to publish them after a certain period of time. For instance, writing an article about my experience with budgeting for the past 5 years would have to wait 5 years before I can write about it. And so I have long-term articles planned for 5 and 10 years intervals. I believe talking about a topic with which you have certain experience–skin in the game, so to say–carries a certain credibility automatically. So while there isn’t much published in 2024, I’ve written a lot and there are numerous items in the pipeline that I wish to publish on a semi-regular basis soon.

On learning side of things, I’ve continued doing Anki everyday this whole past year focusing on subjects such as Japanese, mathematics, nutrition, cosmology, etc. I didn’t read the Genki book I set out to last year and neither did I learn Rust. There are goals for 2025 that address these issues but I’m not being too adamant or aggressive about such goals.

Travel

I travelled quite a bit this year, as usual. Partly because 4 of my friends got married and I travelled to different parts of the country for the ceremonies, it was a lot of fun meeting old friends. I also travelled to Japan for Open Source Summit 2024 and spent some time there.

I travelled to Rishikesh, Dehradun, Palwancha, Jaipur, Japan, Tungnath, and Mysore. A good thing about travel I realized lately, especially solo, is the time you get to simply think about things. I spent so much time on some of my travels in 2024 taking personal–dare I say philosophical–notes that I was quite surprised at the amount of stuff I had jotted down in random places and about random observations.

Productivity

I think I’ve cured myself of the curse of social media a couple years back. There’s still the occasional reddit binge which I’m trying to also bring into a routine but I’m quite happy with what I have so far. Social media isn’t an impediment to the work that’s actually important to me anymore.

Looking back at some stats and retrospecting, I did however feel I could do better with how I structure my day by giving importance to creative or educational pursuits early in the day and then allocating time for physical fitness or other chores like cooking later in the day post work. I’m trying this out now in the new year and it’s been working well so far, a proper summary is due in next year’s review.

As I said last year, I don’t have an active goal towards reducing my social media usage in 2025. I’m focusing instead on applying the concept of directed consumption to certain services on my phone, namely messaging apps and email.

Conclusion

The idea of writing these annual reviews has become therapeutic and doesn’t feel like a chore anymore–it used to feel like it when I first started it while I was building the habit. There was a lot I didn’t achieve in 2024 but there are always learnings that I can carry over to the next year and try not to fall into the same traps.

For a developer, a Container image is essentially a collection of configurations required to run a container. But what really is a container image? You might know what a container image is, how it is made up of layers and that it’s a collection of tar archives. There are questions that still went unanswered, questions such as what makes up a layer, how are layers combined to form a complete filesystem or multi-platform images, etc. In this article, we’ll build a container image from scratch and try to answer all these questions to understand container image internals.

OCI Image

Little bit of history before we proceed. Up until roughly 10 years back, docker-format was the only format being used and with the emergence of all the tooling around containers, there was a need for standardisation. Around 2015, the Open Container Initiative was established–now a Linux Foundation project–with the aim to standardise all things containers. They came up with a specification for container images called the OCI spec. Modern tooling all conform and follow the runtime spec while dealing with container images and so for this article, and OCI image and a container image will be used interchangeably.

An OCI image consists of four core components–layer, config, manifest and index:

We’ll understand each of the above components and build a “hello” image from scratch to practically understand all these components. If we had a Containerfile for our “hello” image, it would look like the following:

FROM scratch

COPY ./hello ./

ENTRYPOINT ["./hello"]

The image itself is based on scratch, an empty base image. We then copy the hello binary to the image and set it as the entrypoint for the container. Let’s get going.

1. layer — what’s inside the image

Layers represent what’s inside a container image. Often referred to as the basic building blocks of container images. They consist of components such as your source code you copy to your image, the container filesystem, or virtually anything you add to your container image.

Technically, a container image is a filesystem changeset. A filesystem changeset is a diff between two filesystems serialized as a tar archive. Consider the following container image:

FROM Alpine

RUN rm -f /bin/ash \
    && apk add bash

We’ve used alpine as the base image, deleted /bin/ash and installed bash. Let’s use this as an example to build changesets and then to assemble the final filesystem of the container.

1.1 layer: create a changeset

In order to create a layer or more appropriately, a filesystem changeset, we start with a minimal root filesystem, one from alpine in our case since our example Containerfile starts off with the alpine base image:

$ wget https://dl-cdn.alpinelinux.org/alpine/v3.18/releases/x86_64/alpine-minirootfs-3.18.4-x86_64.tar.gz && tar -xvf ${_##*/}
$ tree
.
├── alpine-minirootfs-3.18.4-x86_64.tar.gz
├── bin
│   ├── arch
│   ├── ash
│   ├── base64
│   ├── bbconfig
│   ├── busybox
│   ├── cat
│   ├── chattr
...

In order to create subsequent layers, we create a copy/snapshot of the base filesystem–using cp -rp–and make changes to the snapshot by adding, deleting, or modifying files or directories.

A changeset consists of only files that are added, modified, and deleted. In order to create the changeset, both the filesystems(snapshots from previous step) are recursively compared. A tar archive is then created that contains only the changeset.

Given that we removed /bin/ash and added /bin/bash from our example image, our changeset/layer would look like:

./bin/bash
./bin/.wh.ash

Notice the .wh which means whiteout to denote deleted files. The changeset above tells the container engine that we added/modified bash and deleted ash on top of the previous layer/changeset–the alpine base in our case. And that would constitute a changeset, a layer.

1.2 layer: creating the filesystem

Now we have created a layer. A container engine takes multiple layers and creates a complete filesystem for the resulting container.

Let’s say we have 3 layers, an empty layer with no filesystem–the starting changeset. Then we create a changeset and add two binaries, /bin/ash and /bin/bash. Then a final layer deletes /bin/ash and modifies /bin/bash.

The container engine would apply all the layers part of an image on top of each other left-to-right to generate the resulting filesystem. In this case, the final filesystem would only have /bin/bash as a result of applying layer 1 on top of layer 0, and layer 2 on top of layer 1. This is how a container engine creates a filesystem from a collection of layers in the form of a container image.

1.3 layer: lifecycle

So far we’ve understood:

• How an image consists of one or more layers.
• How layers are created from a Containerfile.
• How a container enginer creates the final filesystem for the container using all the layers.

How does this all play together when you employ containers in your daily workflow? Consider the lifecycle of a container image, from a Containerfile to a Container image and finally to a running container:

You use podman build and pass in the Containerfile, podman would then create various layers, and package that as a “container image”. When you use podman run with this image, the engine would combine all the various layers part of the image and create the root filesystem for the container being run.

1.4 layer — our scratch image layer

Let’s create the layer for our “hello” image. For reference, this is how our image would look like had it been a Containerfile:

FROM scratch

COPY ./hello /root/

ENTRYPOINT ["./hello"]

Here, our image contains 2 layers. The first layer comes from the scratch image–a special “image” that tells the container engine to start without a filesystem layer for your image. As we saw in previous sections, this means starting off with an empty filesystem layer and then having subsequent instructions create layers. Almost every instruction inside a Containerfile generates another layer. So in the Containerfile above, the COPY instruction creates the second layer which includes filesystem changes to the layer before it. The change here is “adding” a new file—the hello binary—to the existing filesystem i.e. the alpine root filesystem.

Let us create the layer for our image. We start by creating a statically linked C binary for our simple “Hello world” program:

$ cat hello.c
#include <stdio.h>

int main(int argc, char *argv[]) {
    if (argc < 2) {
        printf("Usage: %s <name>\n", argv[0]);
        return 1;
    }

    printf("Hello, %s!\n", argv[1]);

    return 0;
}

$ gcc -o hello hello.c -static
$ tar --remove-files -czvf layer.tar.gz hello

$ sha256sum layer.tar.gz
36c412b23a871c4afbec29a45b25faad76197f3a9dbf806f3aef779af926790a layer.tar.gz
$ mv layer.tar.gz 36c412b23a871c4afbec29a45b25faad76197f3a9dbf806f3aef779af926790a

We then create a gzip compressed tar archive of the binary. This constitues our first and only layer.

2. config — how to run the container

The config represents how to run the container. It’s a JSON file that stores the configuration options used to configure the container. Options such as environment variables, entrypoint of the container, and volumes, etc. These options can be supplied via the command line while running the container, or as part of the Containerfile in which case, the config.json file is populated.

Consider the following snippet from a sample config.json:

$ vim sample_config.json
{
    "architecture": "amd64",
    "os": "linux",
    "config": {
        "Entrypoint": [
            "./bin/bash"
        ],
        "User": "danish",
        "ExposedPorts": {
            "8080/tcp": {}
        },
        "Env": [
            "FOO=bar",
        ],
        "Volumes": {
            "/var/logs": {}
        }
    }
}

You can see the various configuration options being set in the config section of the file along with additional metadata. You can set the entrypoint, user, ports, environment variables, etc. Let’s write the configuration for our image:

$ vim config.json
{
    "architecture": "amd64",
    "os": "linux",
    "config": {
        "Entrypoint": [
            "time",
            "./hello"
        ]
    }
}

You can notice it’s not too much. If you refer back to our image’s Containerfile equivalent, you’ll notice we set our “hello” binary as the entrypoint for our image, and that’s the only configuration option we need to set.

3. manifest — locate layers and config.json

The container engine uses the manifest to locate layers and config.json for an image. Consider the following snippet from a sample manifest.json:

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "config": {
        "mediaType": "application/vnd.oci.image.config.v1+json",
        "digest": "sha256:<DIGEST>",
        "size": XYZ
    },
    "layers": [
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "digest": "sha256:<DIGEST>",
            "size": XZY
        }
    ]
}

The manifest mentions the config and the layer in the above snippet along with additional metadata for each of them. But instead of referring to a path, the manifest addresses the components via a digest. That’s because the layers, config and the manifest together make up what’s called a content-addressable store.

3.1 Content addressability

In order to allow for efficiency and integrity, OCI requires components in the OCI image to be identified based on their content i.e. you can identify data based on their content rather than its location (filepath, etc). To achive this in the case of an OCI image, a unique identifier (generally a cryptographic hash) is used as the filename. This is known as content addressability.

It helps with deduplication, layer sharing leading to reduced memory and performance overhead, and ensuring data integrity.

We’ll use sha256 as the algorithm to generate the identifiers for the various components in this post. Let’s make our layer archive that we created in a previous step content addressable:

$ sha256sum layer.tar.gz
c37c06cdec9d6a0f2a2d55deb5aa002b26b37b17c02c2eca908fc062af5f53eb layer.tar
$ mv layer.tar c37c06cdec9d6a0f2a2d55deb5aa002b26b37b17c02c2eca908fc062af5f53eb

Directory structure

OCI also defines a layout for an OCI image and container engines expects an image to be packed in the specified format before parsing. It’s defined as follows:

$ tree
├── blobs/<alg>
│       ├── <content addressable config>
│       ├── <content addressable manifest>
│       └── <content addressable layer>
└── index.json

The content addressable content resides inside a subdirectory under the blobs directory which identifies the algorithm used to encode the content. Since we used sha256 to compute the digest for our blobs, we create the directory sha256:

$ mkdir --parents blobs/sha256
$ mv c37c06cdec9d6a0f2a2d55deb5aa002b26b37b17c02c2eca908fc062af5f53eb $_

$ tree ../..
└── blobs/sha256
        └── c37c06cdec9d6a0f2a2d55deb5aa002b26b37b17c02c2eca908fc062af5f53eb

While we’re at it, let’s also make our config content addressable and create the manifest.json file for our “hello” image:

$ sha256 config.json
99c9d2dcbdbc6e28277d379c2b9a59443b91937720361773963d28d5376252a9  config.json
$ mv config.json 99c9d2dcbdbc6e28277d379c2b9a59443b91937720361773963d28d5376252a9

$ tree ../..
└── blobs/sha256
        ├── c37c06cdec9d6a0f2a2d55deb5aa002b26b37b17c02c2eca908fc062af5f53eb
        └── 99c9d2dcbdbc6e28277d379c2b9a59443b91937720361773963d28d5376252a9

$ vim manifest.json
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "config": {
        "mediaType": "application/vnd.oci.image.config.v1+json",
        "digest": "sha256:99c9d2dcbdbc6e28277d379c2b9a59443b91937720361773963d28d5376252a9",
        "size": 149
    },
    "layers": [
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "digest": "sha256:36c412b23a871c4afbec29a45b25faad76197f3a9dbf806f3aef779af926790a",
            "size": 1372604
        }
    ]
}

From the top above, we make our config.json content addressable, we then tke a look at our current image layout, consisting of blobs/sha256 directory which contains our layer and our config. We then create our manifest file, and refer to to both the config and the layer by their digests along with additional metadata such as size and label.

And now that our manifest is ready, let’s make that content addressable too:

$ sha256 manifest.json
2e17c995558ebfa8faacfe64ff78c359ab9f28b3401076bb238fd28c5b3a648b  manifest.json
$ mv manifest.json 2e17c995558ebfa8faacfe64ff78c359ab9f28b3401076bb238fd28c5b3a648b

$ tree .
└── blobs/sha256
        ├── c37c06cdec9d6a0f2a2d55deb5aa002b26b37b17c02c2eca908fc062af5f53eb
        ├── 99c9d2dcbdbc6e28277d379c2b9a59443b91937720361773963d28d5376252a9
        └── 2e17c995558ebfa8faacfe64ff78c359ab9f28b3401076bb238fd28c5b3a648b

So far, our unpacked OCI “hello” image has a single layer archive consisting of our statically linked hello binary, the configuration for the container, and the manifest for the image, all encoded by their sha256 digest.

4. index — manifest for manifests

index.json file acts as an index for a set of images that can span different architectures and operating systems.

An example could be having images for two different architectures in one OCI image, a linux/arm64 and a linux/amd64 variant. In this case, the index.json would contain two entries pointing to the two variants of the image via their manifests that in turn would contain image specific metadata as shown in the diagram above.

With that in mind, we create the index.json for our “hello” image. We don’t have multiple architectures so our index.json file points to only one manifest.json in our image, by its digest:

$ cd ../..
$ vim index.json
{
    "schemaVersion": 2,
    "manifests": [
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "digest": "sha256:2e17c995558ebfa8faacfe64ff78c359ab9f28b3401076bb238fd28c5b3a648b",
            "size": 748,
            "annotations": {
                "org.opencontainers.image.ref.name": "hello:scratch"
            }
        }
    ]
}

Additionally, we’ve added an annotation representing the name and tag for our image referenced by the manifest. The index.json file isn’t required to be encoded, so it resides at the top-level directory of our image and not under blobs/ if you recall the high-level OCI image architecture we discussed in the beginning. With all the finished components, let’s do a quick recap:

$ tree
├── blobs/sha256
│       ├── c37c06cdec9d6a0f2a2d55deb5aa002b26b37b17c02c2eca908fc062af5f53eb
│       ├── 99c9d2dcbdbc6e28277d379c2b9a59443b91937720361773963d28d5376252a9
│       └── 2e17c995558ebfa8faacfe64ff78c359ab9f28b3401076bb238fd28c5b3a648b
└── index.json

(in the order shown above)

1. c37c06cdec9d6a0f2a2d55deb5aa002b26b37b17c02c2eca908fc062af5f53eb — the layer for our image that includes the hello binary.
2. 99c9d2dcbdbc6e28277d379c2b9a59443b91937720361773963d28d5376252a9 — config.json that tells the container engine to set our hello binary as the entrypoint to the container that will be run from this image.
3. 2e17c995558ebfa8faacfe64ff78c359ab9f28b3401076bb238fd28c5b3a648b — manifest.json file locates the config and the layer for the container engine.
4. index.json — a higher-level manifest that references multiple image manifests, allowing for the distribution of container images across different platforms or architectures.

Packing

We have all the parts in place for our image. Let’s create an archive of the components we’ve created so far, and test our image:

$ tree
├── blobs/sha256
│       ├── c37c06cdec9d6a0f2a2d55deb5aa002b26b37b17c02c2eca908fc062af5f53eb
│       ├── 99c9d2dcbdbc6e28277d379c2b9a59443b91937720361773963d28d5376252a9
│       └── 2e17c995558ebfa8faacfe64ff78c359ab9f28b3401076bb238fd28c5b3a648b
└── index.json
$ tar -cf hello.tar *

$ podman load < hello.tar
Getting image source signatures
Copying blob c37c06cdec9d done   |
Copying config cd12bca58e done   |
Writing manifest to image destination
Loaded image: localhost/hello:scratch

$ podman run localhost/hello:scratch world
Hello world!

$ podman image ls hello
REPOSITORY   TAG       IMAGE ID       CREATED   SIZE
hello        scratch    25e8b3bd9720  N/A       3.67MB

We can see that the image was loaded by podman, and we were able to run a container based on our image, correctly outputting hello world! to stdout. We know exactly what made up the image, what files we put in, the configuration we set, and the metadata that’s part of the image.

Using a Base Image

Let’s create a version of our image based on alpine instead of scratch. This introduces another layer in our image, so let’s see how to handle multiple layers. In this case, the Containerfile is as follows.

FROM alpine:latest

COPY ./hello /root/hello

ENTRYPOINT ["time", "./hello"]

Note that we changed the entrypoint to use the time utility. This is done to demonstrate how two layers work in the image.

So far, the image has only one layer that consists of the hello binary. As mentioned previously, a layer can be considered a filesystem diff. So if we use alpine as the base image, we need the respective root filesystem onto which we’ll base the second layer (that is, the layer with the binary). You can obtain the alpine root filesystem from the project’s official website.

$ cd blobs/sha256
$ wget https://dl-cdn.alpinelinux.org/alpine/v3.18/releases/x86_64/alpine-minirootfs-3.18.4-x86_64.tar.gz
$ mv alpine-minirootfs-3.18.4-x86_64.tar.gz \
    $(sha256sum alpine-minirootfs-3.18.4-x86_64.tar.gz | awk '{print $1}')

That rootfs downloaded is already gzipped, so all we need to do is encode this with its sha256 digest and it’s ready to go.

$ cd ../..
$ tree
├── blobs/sha256
│       ├── 99c9d2dcbdbc6e28277d379c2b9a59443b91937720361773963d28d5376252a9
│       ├── 8289bd1bdc2a1fae2d2d717b7c40baaedc4c5c4d9c9f4f1a1b045287067e9f2c
│       ├── c37c06cdec9d6a0f2a2d55deb5aa002b26b37b17c02c2eca908fc062af5f53eb
│       └── 2e17c995558ebfa8faacfe64ff78c359ab9f28b3401076bb238fd28c5b3a648b
└── index.json

Along with our previous three components–hello layer, config and manifest–we now have an additional layer for the alpine rootfs (8289bd1).

Next, we need to update config and manifest files with new information. After that, we also need to make sure that we encode them again with their respective updated sha256 digests.

$ vim config.json
{
    "architecture": "amd64",
    "os": "linux",
    "config": {
        "Entrypoint": [
            "time",
            "./hello"
        ]
    }
}

$ mv config.json $(sha256sum config.json | awk '{print $1}')

We modify our entrypoint to preped time to our hello binary so as to differentiate between our scratch image and alpine base image. No other modifications to the config.json are required.

In our manifest.json file, we update the digest for our modified config.json and add layer information for our alpine base layer. Layers are added top-to-bottom, so our alpine base layer is the first entry in the array, followed by our hello layer.

$ vim manifest.json
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "config": {
        "mediaType": "application/vnd.oci.image.config.v1+json",
----->  "digest": "sha256:75b148a9a5b61403d082f1930ccc779bfdac5acddb48ed8f90ccdc4219d51268",
----->  "size": 603
    },
    "layers": [
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "digest": "sha256:8289bd1bdc2a1fae2d2d717b7c40baaedc4c5c4d9c9f4f1a1b045287067e9f2c",
            "size": 3279768
        },
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
--------->  "digest": "sha256:c37c06cdec9d6a0f2a2d55deb5aa002b26b37b17c02c2eca908fc062af5f53eb",
            "size": 1372611
        }
    ]
}

$ mv manifest.json $(sha256sum manifest.json | awk '{print $1}')

Since we changed our manifest, we need to update index.json with the new digest and filesize:

$ vim index.json
{
    "schemaVersion": 2,
    "manifests": [
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
--------->  "digest": "sha256:8289bd1bdc2a1fae2d2d717b7c40baaedc4c5c4d9c9f4f1a1b045287067e9f2c",
--------->  "size": 748,
            "annotations": {
                "org.opencontainers.image.ref.name": "hello:alpine"
            }
        }
    ]
}

If we pack and load the image again with podman, we can see the modified image in action:

$ tree
├── blobs
│   └── sha256
│       ├── 99c9d2dcbdbc6e28277d379c2b9a59443b91937720361773963d28d5376252a9
│       ├── 8289bd1bdc2a1fae2d2d717b7c40baaedc4c5c4d9c9f4f1a1b045287067e9f2c
│       ├── c37c06cdec9d6a0f2a2d55deb5aa002b26b37b17c02c2eca908fc062af5f53eb
│       └── 2e17c995558ebfa8faacfe64ff78c359ab9f28b3401076bb238fd28c5b3a648b
└── index.json
$ tar -cvf hello-alpine.tar *

$ podman load < hello-alpine.tar
Getting image source signatures
Copying blob c59d5203bc6b done   |
Copying blob c37c06cdec9d done   |
Copying config 75b148a9a5 done   |
Writing manifest to image destination
Loaded image: localhost/hello:alpine

$ podman run --rm localhost/hello:alpine reader
Hello, reader!
real    0m 0.00s
user    0m 0.00s
sys     0m 0.00s

$ podman image ls hello
REPOSITORY   TAG       IMAGE ID       CREATED   SIZE
hello        scratch   25e8b3bd9720   N/A       3.67MB
hello        alpine    3d0268e9a91e   N/A       11MB

Conclusion

I hope this post helped you understand container images in slightly more detail and in a manner that was accessible and approachable. The examples used in this article is not something you’d use in your daily workflow. Their purpose is to demonstrate the inner workings of container images.

If you find any problems or improvements, open an issue or feel free to reach out via email.

Thanks to Aleksa, Dan, and Dmitri for all their help with this article.

:wq

The first I heard of short-form content was back in school, roughly 11 years ago. All of a sudden, people would send these extremely short yet hilarious videos with a Vine watermark on them. I was smitten by the creativity of these people(the term creators wasn’t common parlance back then), by how they were able to tell a story in a few seconds. During the next ten years, Snapchat mastered the “stories” concept which was later picked up by Facebook, Instagram and virtually every other social media application in existence. But I was never too drawn to social media, and thus a little out of loop on what was happening, especially with the shortform content craze.

Fast forward to 2022, I was reading Digital Minimalism, in which Cal Newport talks about the disadvantages of social media, and so thanks to frequency illusion, I found that shortform content is virtually everywhere. It has—primarily in the form of stories—spread like a plague in the past decade or so, especially with the newfound affordability and availability of smartphones and cheap internet. People of varying ages are hooked to their devices watching these short bursts of entertainment to keep themselves, well..entertained.

And I kinda understood why it was so appealing. It was simply convenient, and it wasn’t long before I found myself looking up information quickly. What’s the water to quinoa ratio while cooking or getting a form-check on a new exercise I want to try at the gym while I was in the gym. It was just too convenient and worthwhile. All of a sudden, I didn’t have to watch a whole video or read an entire article on a topic to find the information I was looking for in the moment. What’s not to like?

But I was astonished by the extent to which it spread and captured people’s attention, and took over their time. I was out on a walk recently and noticed a small kid, about 8-10 years old, sitting next to what I believed was a young full-time babysitter (quite common here in India). They both were sitting on a bench in the park, glued to their respective mobile phones and endlessly scrolling something that resembled Instagram stories or Youtube shorts, I’m not sure which. Now, I’m in no way a Luddite but that sight was extremely depressing. Yet another time, at a wedding, I noticed mothers thrusting a phone in each of their young children’s hands and then feeding them food. Upon some questioning, I was told that they just don’t eat without a phone anymore. These are not isolated incidents, these are just the ones I’m choosing to share. If you start to look, I’m sure you will easily find many more around you. All day everyday, as soon as someone has some time to kill, out comes the phone, and the scrolling begins. It has gotten to the point that people now do this without thinking, they pull out their phones at the slightest hint of boredom, their bodies and brains working like automatons.

It has been said time and time again that the brightest minds of our time are busy building applications which ensure that you spend as much time as realistically possible on your devices. The simple premise being: you spend more time on these apps, the apps learn more about you, show you more targeted adds and well…$$. So it’s no wonder that getting rid of this addiction is extremely hard, I’ve written about how I attempted to get over my problems with social media in the past. So this would explain why the problem is so rampant across all ages.

I wouldn’t have really blabbered about all of this if people were simply spending–better yet, wasting–their time on devices watching short videos. But that’s simply the tip of the iceberg imo. Last month, I was at a service center for my two-wheeler’s scheduled maintenance. Sitting next to me in the customer’s waiting area, was another customer, a middle aged gentleman. For a good 1 hour—and yes, I timed it—the individual spent scrolling through Instagram reels non-stop without earphones (that’s another discussion). I was fascinated by the sight and I couldn’t help but observing this man and the content he was watching(mostly because it was audible to the whole room). There was no coherence in the content he was watching, he was completely at the whim of the application’s algorithm. One moment, he was watching a funny animal video, and a provocative news byte, the next.

There are two problems with this. First, If you don’t choose the kind of content you want to consume, then you can seemingly easily mess up your state of mind. You could be sitting down with a hot cup of tea on a Sunday morning and the algorithm decides to serve you some depressing and aggressive content, how do you think it will affect you? Your head space is being governed by these apps, and unfortunately you don’t have a say in it. This is exactly what social media manipulation is. You pay the social media companies and the companies would make sure that content inspired by your ideology gets the front page for millions of people, enticing them, triggering them. As harmless as this sounds, a quick search will tell you how fatal this has been so far, and the untapped destructive potential this holds in our fragile society.

Second and the more pressing of the two problems is how it all impacts our intellect and rational thinking in the long run. Majority of people’s time is being spent mindlessly scrolling their feeds. Wishful thinking on my part but are there better uses of this time? Absolutely. Especially when you’re partaking in undirected consumption, the least you can do is opt for directed consumption. For instance, in the previous Sunday morning example, instead of opening any social media app and scrolling away, you can watch that video you saved a while back that you had planned to watch or search something about a topic that you wanted to learn more about, etc. It’s highly likely that your brain wouldn’t allow you to choose aggressive or distressing content now that you have a say in what you watch.

Yet more damning are the effects on younguns. It’s common knowledge that a young kid, during their developmental years, benefits immensely from exploring the real world, playing, getting feedback from the physical objects around her in order to grow healthy. Not always sitting indoors cooped up in front of a device for hours on end, unaware of what they are being fed(pun unintended). And even when indoors, there’s a case to be made against simply sitting down and passively watching something. When a child explores something on their own, for e.g. reading a novel or a comic, solving a puzzle, or playing with toys, it encourages critical thinking and allows them to learn important cognitive skills. Compare this to letting the algorithm on one of these apps serve you or your little ones curated videos that they watch and discard the very next second without any pause in between to think about what they just watched. Generally the content is so superficial, there’s hardly anything to think about and by the time the video approaches the end, they’re already invested in the next.

Finally, and this is a last-minute edit to the draft of this post, I’ve noticed friends and family members busy scrolling while they are sitting together in a group or are in the company of someone. It has become this tick that we simply can’t shake off. Imagine sitting across someone and they are reading tweets from random strangers on the internet in the hopes that one of these random bytes of information would come in handy…some day. Not only is this disprespectful, but I feel it makes it difficult for people to strike up a conversation. Purely happenstance, but I find it easier to start talking to strangers who are aged or not hooked onto their devices(you can tell) than with people who are busy with their devices, scrolling away.

Is there a way out? At the risk of sounding pessimistic, I’d say nothing’s going to change. The odds are all stacked against us with this one, to be honest. None of the companies behind these applications want you to spend any less time on your phones, hell, they want you to spend as much of your waking time as you can on them because that would mean more money for them. On the other hand, humans are primed by evolution to seek novelty and rewards. In prehistoric times, our ancestors had to be accutely aware of their surroundings in order to survive which made our brane attuned to releasing dopamine when something new and unexpected happened. Companies tap into this system by using mechanisms such as notifications and making it terribly easier to continue watching videos one after another simulating a slot machine. And so, we feel terribly anxious in a good way before scrolling between two bytes of information(video, image, etc.) in the hopes of hitting the jackpot, that one information that’s going to change everything, but it never happens, and that’s by design. But we don’t know it, and this is why, it’s extremely difficult for people to simply cold turkey quit using these applications or to self-regulate their usage. We’re just not built that way, it takes too much effort.

So bottomline, our time has been taken over, more often than not, we are no longer in control of our own state of mind, we’re losing out on crucial cognitive development benefits, and the very fabric of our social interaction is being changed ever so slightly. Situation is so dire, the kids are refusing to have food without their phones.

Stay tuned for a shortform version of this post.

3rd iteration in the series calls for some changes, namely, the category headers are changing but the content remains the same more or less. This is done in order to keep articles in this series in sync with my private goal planning. Let’s begin.

Personal (Health & Fitness)

One of my most important health goals this year was to reach an acceptable(to me) bodyfat percentage. I managed to get my bodyfat down from 23% at 72kg body weight to ~14% at 65kg body weight in the span of roughly 10 months. I did however made some mistakes in order to reach this target but it was all worthwhile at the end.

Bodyfat stats on the left, bodyweight on the right, with roughly the same x-axis scale denoting the time in weeks. That last blip at the end in the BF stats and the stagnation was when I went on a vacation and simulataneously went on a maintenance phase.

I also lifted consistently throughout the year, except for when I was travelling. There were times when I had to sit back due to an overdone knee or shoulder but thankfully neither was it as bad as it was the previous years nor was it long. One reason could be because I started focusing more on mobility and isometrics as part of my workout warmup. Badminton was also right up there with lifting in terms of consistency. I didn’t spread myself too thin by engaging in a number of activities this year and it was fun nonetheless.

For the upcoming year, I’m going to be focusing slightly more on bodyweight exercises and foray into calisthenics. My composition goal is to get to ~72kg body weight again with <15% bodyfat by the end of next year. Slightly ambitious but I’m hopeful.

Career

I started a new job with the year and it has been great so far with occassional hiccups which I’m used to by now. But I’m enjoying working for and in the open source domain and the people around me are smart and supportive. I primarily worked on container technologies and contributed to projects such as podman, buildah and nerdctl, both upstream and downstream.

There’s still a lot for me to learn in this domain, and one of my goals for the next year is to publicly document as much of the learning I can in the form of technical articles or even talks if and when the circumstances allow.

Education

Read a wide variety of authors and topics this year ranging from geography, nutrition, literature and science among others. Some of my best reads were The Demon-Haunted World by Carl Sagan, Home in the World by Amartya Sen and Where India Goes by Diane Coffe & Dean Spears. I stopped tracking the number of books as a goal two years back so there’s no goal to look for in the next year as such but I’m just going to continue exploring the domains I mentioned above.

Unlike reading, writing was a dismal affair. As much as I enjoy writing, or let’s say the thought of me writing, I didn’t write at all this year. I barely scraped by at the end of this year with 2 articles that I started sometime in June of this year but kept dragging them unfinished to the very end. This becomes more pronounced because I was able to bring this category back from the dead in 2022 but it fell through the cracks again.I’ll be trying things out to ensure this doesn’t happen again.

Last year, I mentioned I had been learning Japanese for a while now. To be more precise, I’d been learning vocabulary, this year I started learning grammar using Genki. The plan is to finish both editions of the book as soon as I reasonably can but to be realistic, this year’s goal is to complete Genki-I through and through while continuing to learn more kanji. Talking of languages, I had originally planned to learn Rust last year but I was lost on both time and motivation to do it. I’m not setting any concrete goals this year but If at all things seem to fall in place, I will start learning Rust this year, finally.

Finance

Can’t share the exact details here for reasons but I index invested consistently and donated monthly, except for a few misses here and there. My goals for next year is to simply increase the percentage contribution to both slightly and clear out an ongoing loan. I have more drawn out and specific goals internally but the big picture is to spend meaningfully and focus on time in the market.

Travel

It was a good year for travel for me, I travelled to Hampi, Coorg, Shantiniketan, Singapore, Malaysia and Indonesia. I learnt so much from all these travels, and perhaps I’ll be writing about my experiences in some of these places some day. I worked remotely on some trips, was with a group on some and solo on others, it was a healthy mixture of experiences. Despite sounding like a broken record, I’d like to emphasize again the fact that I’m grateful and lucky enough to be able to make travel plans and execute them.

Productivity

My only plan in this category was to cut down on social media. I brought down the consumption to only my laptop and that too only on Sundays. And in so doing, I could feel no tangible difference between when I was all over social media and this past year when I reduced it to, essentially 52 times a year, technically. I could safely say I didn’t miss anything major or life changing that one of my friends wouldn’t later talk to me about. I’ve written about this before and I still believe that social media is predominantly a waste of time, not even considering the other more psychologically damning issues.

For all the reasons stated and more, I can happily cut this piece out for good going forward. For the upcoming year, I plan to use it either once a month or not use social media at all.

Conclusion

That was about it for this year’s reflection. Before closing, I jumped on the AI hype train and asked ChatGPT to summarise this year for me, based on this post so far and here’s what it had to say:

In summary, 2023 marked a year of personal transformation and professional growth. Achieving significant strides in health and fitness, successful navigation of a new career path in open source technologies, diverse reading experiences, and enriching travel adventures defined the year. While writing saw a dip, plans are in place for a more consistent output. Language learning progressed, financial goals were met, and social media consumption was effectively curtailed. Reflecting on these accomplishments, I look forward to embracing new challenges and opportunities in the upcoming year with optimism and determination.

Not bad at all, thanks for reading and see you next year!

:wq

Before we start, let’s make it clear that I’m not trying to say that having options to choose from is a bad thing. Far from it, it enables people from different backgrounds or cultures to live a comfortable life. For instance, if we only had high-end SUVs for people to travel in, the world would be a very different and inconvenient place for a lot of us who can’t afford an SUV but thankfully that’s not the case. People have the option to choose, they can use the public transport(!), ride a bike or get a hatchback based entirely on their requirements and concerns.

I’m also not implying that the previous era when we didn’t have the luxury of choice offerred us better times, that’s factually incorrect. The world is definitely a better place and I’m not here trying to romanticize earlier times just because I seem to feel something is not right with our age. But at the same time, I feel like we lost something specifically with respect to certain activities or experiences when we agreed to or rather when we started racing to provide more options for every single thing. Let’s talk about some of these experiences.

Watching Movies

It’s extremely rare for me to watch movies with someone these days, be it family or friends, but I remember back when I was in 4th standard, our family used to watch movies quite often. On many occasions, cousins and neighbours would join in on a Sunday afternoon for a movie and everybody would get together and have a good time. But I feel we’re getting together less often to watch movies and choice might be the reason behind it.

Back in the day, you would not have access to hundreds of movies, there’d be only a handful of movies that you could choose to watch, either in the cinemas or by buying a DVD. Internet was not as widespread as it is now, neither was it fast enough to allow streaming videos, let alone movies. So whenever there was a chance to see a movie, nobody would let it go. Contrast that with today, we have every single title that is out there at the convenience of a couple clicks in our homes or as some would prefer, in their pockets. Not to mention the fact that I’d rather prefer a movie that is more suited to my taste than to watch something I don’t have an interest in. And with that, what you essentially have here is the availability of choices affecting real-life connections.

But on the flipside, it also feels good when there’s that one movie that everyone is eager to watch and therefore, the novelty of watching a film in theatres with your friends and family never wears off. But it’s hard to say that this compensates for everything else we might’ve given up in order for us to have more choices in the kinds of movies we want to watch. The best way out in this case is to perhaps choose a movie, either by voting or my mutual consensus every couple of months. The voting part is still going to be tricky but hopefully there’s a movie every so often that piques everyone’s interest even slightly.

Listening to Music

I vividly remember the latest popular song to come on the radio or on the handful of TV music channels that were available back then and everybody would go gaga over it and sing along. Here was a single source for music and since you didn’t have any control over it, you’d have to wait for your favorite song to come up. Sometimes, it’d come up sooner and other times, it wouldn’t come up at all but there was something fun about the whole process. The choices were extremely limited so most of the times, we’ll be jamming to whatever was on the television or on the radio.

Fast forward to the current day with services such as Spotify, YouTube Music, and Apple Music, and you have so many artists and tracks to listen to. To add to that, there’s very little friction for you to fire up one of these apps and start listening to whatever music you’re in the mood for. For instance, at the gym, rarely do I see people jamming to what’s playing in the background, howsoever terrible it might be, everyone’s plugged in, myself included. But I don’t think we can blame anyone either, because I’d rather listen to that UK drill single lest I mess up my workout which would be undesirable. I was recently discussing Spotify Wrapped with some friends over dinner a couple days back and hardly anyone had any common song or artist in their top 5. I thought it was surprising. But here too, we can talk about both the pros and cons. One one side, you can see people in the subway or the gym or even in the garden, plugged in and walking among themselves potentially losing out on forming new connections or engaging in discussion with their fellow passengers. And yet on the other hand, such platforms prove to be a boon for upcoming artists which further promotes creativity and originality, to a certain extent. They are able to reach a wider audience without having to spend a fortune and hence you nowadays see so many people listening to and supporting Indie artists which is a win-win situation.

Finding a way out for this is much more complex than watching a movie simply because listening to music nowadays is generally considered as an auxiliary or a background activity. You could be in the gym or doing chores or even studying or working while listening to music. And so finding someone to have that exact same taste while they are engaged in some other activity which might require a different kind of background music is not trivial. But similar to movies, there are moments such as going to that live show where your favorite artist is performing, that prove to be that much more fun. As much as I would want to find a way out, I think we can keep things the way they are for music.

Reading

Books, I personally feel, are an anomaly in this context. I haven’t yet had an experience where having more choices in books came out as not right. Of course, I’ve had my fair share of instances where I have some trouble deciding which book to read but it’s not strenous and much more importantly, I don’t see it having an effect on anyone else other than me. Reading has historically been a solo activity so there’s not much to lose in the way of forming or enhancing connections while engaging in the activity. Rather, once you’re done with the act of reading, you can then choose to engage in sharing with or receiving recommendations from others and for what it’s worth, having more options would only enhance this whole experience.

When I’m looking for what to read next, I can scrounge the internet for precisely what I’m looking for. This is thanks to the sheer number of titles that are easily available on the internet and are equally accessible. It’s an understatement to state that I really enjoy researching about what I want to read next. But at the same time, whenever I read memoirs set in the pre-tech era, there are always references to people receiving books from friends or families, which turn out to be enjoyable reads. And so this naturally got me thinking whether I’d also enjoy reading whatever I can get my hands on without doing any prior research on the exact topic? I’m not particularly inclined to try this out because I enjoy the status quo unless of course something completely new comes out to change the way we approach reading as an activity. So we seem to have come out unscathed when it comes to reading and advent of choices.

Eating Out/In

So freaking hard to decide on what you want to eat and once you manage to accomplish that task, you’re caught up in a never ending mire of where to eat. This is more pronounced when you’re ordering in because there the options open up more. When choosing for a place to eat out, there are usually tangible restrictions around such as the distance or whether the place has availability, do you have to take reservations, etc. You also need to think about the weather, the transportation and after clearing all these hurdles, you reach your destination to enjoy your meal, or rather, to think about what to order only to get caught up in the same dilemma again. Whereas while ordering in, all you need to do is to choose, and if you’ve been following me, that’s the crux of this whole essay. You have a a myriad of options and there are no physical, logistical or meteorological restrictions attached, you can order anything you want as long as it’s there on the screen. Anyone who has tried this can tell you how frustrating this is. It’s in fact so prevalent, that one of the major food delivery services in India has built a feature to help people overcome the cognitive impairment that comes with choice overload.

Back to the good old times, I fondly remember a handful of places we’d frequent whenever it was time for a celebration. Ordering-in was going to be a relic of the future, thank heavens. But I don’t really remember the elders looking famished trying to figure out where to go. For starters, we used to head out quite seldom and when we did, everybody would already have the same place in mind and discussions about where to go were extremely rare. There’s a good reason all this could be attributed to my background and that’s okay. The point isn’t to carve out a pattern but rather share an opinion or theory. And the theory is that choosing where and what to eat has become a cumbersome exercise, thanks to innumerable choices.

Gifting

This is a fun one. It was so much easier and fun to gift someone something. Limited choices made it easier to choose something and the likelihood of the receiver enjoying it was also significantly greater. Nowadays, it’s just impossible to gift someone something they’d like unless or until they had given you a list beforehand. Let’s just say you chose a perfume for someone, now given the variety in perfumes, it’s extremely unlikely that what you chose is what the receiver wanted. This completely rules out the fact that the person wanted a perfume in the first place, but having more choices or not doesn’t necessarily solves this so we’ll not get into that.

I thought one day about how I would like to receive a gift while I was wondering what to gift someone. I realized how with all the shopping options available online these days, I’d much rather prefer a gift card so that I can buy something I’ll use (I like to consider myself a minimalist utilitarian) rather than receiving a perfume for the umpteenth time. But..I still have to ask someone which gift card they’d like to have because there are just so many of them, which online store, which gaming console or which food delivery app, etc. This is still one hop more than I’d have preferred but at least the reciever is going to get something useful out of it at the end of the day. But thankfully, people are receptive to the idea of getting a gift card and so I feel this is the best way out so far.

Conclusion

This topic has been bothering me for a while and I have a lot more examples of activities or experiences that are seemingly affected by having to choose from a number of options. But the point of this essay isn’t really to keep blabbering but to put my thoughts out there giving the chance to the reader to think about the topic too. The topic of choice overload has generally been of interest for people but I rarely see it being discussed with respect to things that we come across daily and how they compare with the times when we weren’t required to choose from so many options.

This essay, on a broader level, discussed only the situations and not the solutions, even though there are a few suggestions. This is deliberately left as a thinking exercise for the reader because every single topic discussed in this essay includes high level of subjectivity and therefore proposing a solution wouldn’t have made any sense. So think about it and reach out if you feel there’s more to it than I have tried to cover and I’ll be happy to discuss further.

Humble Beginnings

The first iteration for my small corner on the interwebz looked like as shown in the screenshot below. Little background, feel free to skip to the good parts if you want. Sometime in my final year in college, I came across someone’s “personal” website, the term was fresh to me and I didn’t understand it completely although I knew internet-famous folks had their presence on the internet. But the mere fact that virtually anybody could have a piece of the internet reserved to themselves, felt somewhat liberating. I quickly sprung into action to figure out how I could do the same. Lo and behold, technology came to the rescue. Github pages had just launched and I got the coveted username.github.io up and running.

And that was that. Occassionaly, it felt good to have someone come up to me to say that they came across my website and that it looked cool. Positive reinforcement worked it’s magic and I got to work on new iterations for the design looking every which way for inspiration(more on the progression at the end of this post). And it slowly evolved to the point that I focused more on fretting about the design than the content, yak-shaving. I struggled quite a lot finding a balance between writing and aesthetic. To this day, I sit to write something and I’d snap open main.css to fiddle with the design here and there.

Disclaimer

Before I start harping about some of the decisions I’ve taken, I’d like to add a small disclaimer that I’ve written this post at a certain point in time when the site looked like the screenshot below. If you’re reading this in 2034 with a dark theme, please don’t label me a hypocrite. People change, decisions change, tastes change.

(inception^) With that legality out of the way, I’d like to talk about some of the things I had in mind while I was writing the draft for this post.

Why

Natural for someone reading this to ask why was this written this in the first place. Honestly, there’s no concrete reasoning behind this. I just thought it’d be fun to dump the often in hindsight reasons behind some of the design decisions I’ve taken with this blog. Another reason that nudged me to write this was how sometimes people would ping talking about the design of this site and even asking for the theme. I thought it’d be fun for such readers and anyone interested in design really to go through the thought process. I know I would, so if you’re on the fence on something like this, please write!

I’ll briefly talk about a topic and then move on to the next, there’s nothing profound about this so I’d rather not waste too much time if you decide to read the whole piece.

Dark Theme

The most obvious of them all. I want to create a more readable experience, I wanted to optimize for reading when someone lands on this blog. I’m of course biased–affect bias–because I’ve never personally liked reading white-on-black. Have you ever had to read white text on a black sheet? It’s not nice. Reading black-on-white takes me back to the good old days without technology where all you could find is black-on-white and I cherish that memory when I open such sites still. It is also partially inspired by Instapaper, that app is so visually appealing to read on and I wanted to create something similar, I’m sure you’d have noticed similarities by now. And also, let’s not forget Kindle. I do most of my reading on an old kindle and I cherish the warm and comfortable reading experience that the white on black provides.

Hyperlink Decoration

I come from the school of thought that too much visual confetti breaks comprehension. Again, affect bias would be the underlying theme for this post but you get the idea. For that reason, I’ve tried to keep hyperlink decorations to the bare minimum. Only so much as you’d get a hint that it’s indeed a hyperlink and that’s about it. No need for your peripheral vision to jump around to colorful elements popping out while you’re trying to focus on content. I don’t use syntax highlighting to comprehend code and I believe it’s easier for us to comprehend literature without syntax highlighting. Think about it.

The above image is taken from this post by Linus Åkesson. End of the day, this is personal preference and very much pointless to debate on it but it’s worth mentioning here because of the topic in question.

Center Alignment

The content on this site has been centrally-aligned since its inception. I tried a left-aligned variant briefly before reverting back.

I realized fairly quickly how “unnatural” left-alignment on a monitor is. It’s just natural for you to sit with your face aligned with the center of your display. And if you use multiple displayes, then for your eyes to align with the center, no matter what’s on the screen before adjusting itself to find content. To draw parallels with traditional media, we always keep the book or paper we’re reading in the front center, not left and not right and if it were to be like that for some reason, it’ll quickly get uncomfortable if you spend more than a few minutes in that position.

Header

By now the pattern must be discernible. Optimize for reading and keep the fluff to the minimal. The header tries to be as unassuming as it possible can while still providing the required details and navigation across the site. There’s no over the top hero image, contrasting color scheme or fancy floating animation.

I’ve come across websites in the past that make it extremely difficult to navigate the site or to skip to the main content of the site just because they have so much chaos going on in the header. I admire Google’s simplicity here, keeping it stupidly simple just works.

Future

There are a lot of microdecisions that I decided not to talk about here for the sake of keeping this post comprehensible. As for the future of this blog’s design, as I mentioned in the disclaimer, tastes change and it would inevitably lead to some drastic changes to this blog but I want to believe the things I’ve talked about here would remain intact one way or another. I plan to keep this post as a living post which documents the design journey of this blog.

Conclusion

No reasonable conclusion to this one but hoping this would be a poignant reminder to my future self to continue this ever-evolving obsession with improving this blog, so to speak, all the while continuing to strike a balance with writing more content here.

Second coming of the yearly review. I really enjoyed doing this the first time and I’m guessing it’ll be an yearly ritual I’ll follow from here on out. Especially after following last year’s writing goals, I’ve come to quite enjoy the cathartic experience that writing is. Again, no fixed structure to this post, but building upon last year’s post as we go.

Work

I felt I was more involved in product discussions and ad-hoc issues more than actual hands-on work that I was more inclined to do. I had known this and I found a really good opportunity at the end of the year. I started at SUSE this new year and I’ll be focusing on container technologies. I’ve always been passionate about open source and have always felt like trying out such an opportunity if I ever chance upon one, I’m super excited for what’s to come.

Projects

While reviewing the previous goals in my annual OKRs, this was an eyesore. I wrote about this in 2021 too and unfortunately, things haven’t changed around much. Although I did briefly work on a toy DB engine and a static site generator, the truth is that I didn’t find the inspiration to see either of those to completion. This year, I’m at least looking forward to complete the former and have this site being backed up by it. Despite the dismal disclaimer, as part of an issue I was struggling with at work, I wrote a small kubernetes controller which I then further hacked at our internal hackathon at HackerRank. It was fun.

Not so dreary were the open source contributions. I’m hoping, with the new role, I’d be in a better position to contribute more this year.

Reading & Writing

I had consciously chosen not to participate in the Goodreads reading challenge in 2022. I found that with the challenge in tow, I was trying to push towards quantity rather than quality sometimes. This year, I chose which books to read purely on the basis of my interest at that specific period of time. I especially enjoyed The God Delusion, Man’s Search for Meaning, Walden, Digital Minimalism and Annihilation of Caste.

Writing also, surprisingly went quite well compared to 2021:

$ find _posts/ -type f -name "2022-*" | wc -l
8

Not bad, not bad at all but I’m going to improve this as much as I possibly can. I’ve some ideas in the pipeline for both technical and non-technical posts and I’m going to strive for quality as well as quantity this time around.

Health

Continued lifting for the better part of the year but had to deal with recurring shoulder impingement throughout the year, it was frustrating but at the end of the year, I started seeing a physiotherapist with whom I’m working on strengthening the rotator cuff. Hopefully it’ll not be as big of an issue as it already has been going forward, but thankfully I was not away from the gym for the most part.

Halfway through the year, I moved back to Bengaluru which meant no cycling at all, which was a bummer. But I continued playing Badminton and also threw in swimming occasionally for good measure which helps keep things on track. On the nutrition front, I tried a few different diets with not-so-life-changing results as some online articles would have you believe but thanks to these experiments, I know the foods I feel good after eating, while simultaneously achieving my macros. I’ll still be experimenting for as long as I can though.

I have some ambitious health goals for 2023, such as reaching a certain bodyfat percentage and weight. Looking forward to them, will check back in ‘24.

Travel

I was able to overshoot my travel plans after so long. Early on, I went on a workcation to Goa and then to Nepal and ended the year with a trip to Vietnam. Travel, to me, is explorative and not just a means to turn the dials off. It is another excuse for me to keep learning about different cultures which I find satisfying and satiating. I’m grateful to be in a place where I can make travel plans and execute them.

Other

Learning Japanese was put on the backpedal(すみません), in that I didn’t learn a lot of new stuff, just kept on revising the stuff I had learnt already. Safe to say, I didn’t appear for the JLTP exam in 2022. I’m going to attempt this ambitious goal once again in 2023 with more vigour and see where we end up.

This year, I’m also focusing more on how I retain information and how I take notes. I’ve started using Obsidian and have been reading this series of posts about note-taking by Jamie Rubin. I intend to be more mindful and comprehensive while taking notes and working my way towards increasing my overall retention of topics I find valuable and useful. I’ll most probably write a post about this sometime in 2023 so I’ll keep it short here.

Finally, a thing I realized this past year is that I value solitude a lot. I tend to avoid a gathering of more than 2 individuals just because I think the discussions quickly spirals into drivel one way or another. So far, I had been under the impression that this was not acceptable or socially correct as people around you would quicly have you know. But I recently read Susan Cain’s Quiet which throws light on being quiet or being introverted is just another, even better in some ways, way of life. I’m curious to know more about how to optimize (for lack of a better word), such a way of life.

2023

I have some specific areas I’ve identified which I’m going to be focusing more on in 2023, most notably:

• Technical Education: the pipeline with which I learn most non-technical things in life seem to be well in place (books, videos, discussions, etc) but I want to focus more on education from a technical standpoint. I aim to be a T-shaped individual, I like going in-depth on a topic but to find said topic, I like to diversify the kind of topics I’m reading or learning about.
• Health: Focusing more on my nutrition and body composition while continuing to progressively overload in the gym. This will, of course, involve a lot more discipline and work and I’m devising a plan on how to proceed towards this goal.
• Career: I’m still looking for objective goals for the next year but not all things have to be intricately measure, and at this point, I’m just really looking forward to enjoying my new role, working on the technologies I like and in a domain I’ve always had an inclination towards.
• Lifestyle: I’m at a point in my life where I’d like to experiment more with my lifestyle, not in a materialistic way, but perhaps spiritualistic or stoic. As mentioned, I’d like to also play around with solitude. I don’t think I have a clear picture in my mind yet but I’ll whittle this goal out as I learn more.

That’s it for this year’s edition, thanks for sticking till the end.

Journey

We took a direct flight to Hanoi from Delhi, landed late in Hanoi and took a public bus to the famous/iconic Buffalo hostel. The next morning, we visited the Literature Temple, a famous Bun Cha place and tried egg coffee at a lakeside cafe. The city was teeming with young people rustling by on their bikes and scooties. It’s an overwhelming sight when you first ride on the main roads and streets. And this is coming from someone who’s ridden in some of the most congested of places in Delhi.

The first thing we had planned to do in Vietnam was the incredibly surreal Ha Giang bike loop. It’s exactly what it sounds like, a 4-day bike loop in the mountains of northern Vietnam. Our group, by the time we flew from Delhi, had morphed into a 6-member group. In Ha Giang, it further transmogrified into a 12-member group when we teamed upw with two other smaller groups to do the loop together.

The idea is to start from Ha Giang city, then ride a couple hundred kilometres and stay at a homestay in the valleys, repeat 3 more days. We all rode a semi-automatic bike with surprisingly good mileage. The loop is designed in such a way that even though it’s ~120km a day, it has enough stops in between that you cover the whole day and reach the homestays after sundown.

The pictures aren’t lying. It was just sheer greenery through and through with occasional patches of life when you’re passing by a valley. And that too was accompanied with extremely curious locals trying to get as much information about India as they can.

I have so many more photos I can attach here but then I might not be able to talk about the other destinations, so let’s stop here with Ha Giang and move on.

Next stop was Ha Long Bay. The UNESCO World Heritage Center site is famous for it’s limestone pillar in the inland ocean. The only way to explore these is to book a cruise which would sail further into the bay to give you a closer look at the formations.

You can also visit the caves formed naturally inside these formations and additionally, go canoeing around them.

Back to Hanoi, and off we went to Ninh Binh for a night. We rode bikes from Hanoi to Ninh Binh and the reliable partner that Google Maps is, it took us through deep residential pockets skipping the highways only meant for cars. But thanks to this route, we were able to see middle-aged ladies enjoying a session of tai-chi? in the evening in a community center verandah while a group of young kids played volleyball in a ground further ahead. Or in other words, a casual evening routine for the residents of outer Hanoi.

Ninh Binh was quiet, serene but due to the storm that just went by a couple of days before we arrived, it was raining quite heavily. We still managed to do the famous boat tour there. During the ride, Vietnamese tourists would shout out and greet us and ask us where we’re from, it was a jovial affair. We even had a couple of races with some of them, it was healthy.

We headed back to Hanoi for one last night where we explored the War Museum, it was an overwhelming collection of feelings, artifacts and information surrounding Vietnam’s history especially the infamous Vietnam war.

The next morning we took an early morning bus to Hue, the ancient capital of Vietnam. It was raining cats and dogs and while we touched upon some famous tourist spots in the city, we eventually ended up having coffee and just taking short walks around the bus stand waiting for our afternoon bus to Da Nang.

Da Nang is a typical urban expanse with lots of things to do. We made a day trip from Da Nang to explore Hoi An, which is a quaint little coastal town with lots and lots of tourists throughout the day. You can walk around the town capturing the old streets, shop for souveniers and have their local delicacy, Cao Lau. The town suddenly transforms after sundown, it’s a brightly lit cacophony of lights and lamps, a sight to behold.

The final leg of the trip was to head south to Ho Chi Minh City, formerly Saigon. As soon as we landed in Saigon, it was chaos. It looked like any other big city in South East Asia but with lots of people. There were bikes all around and coming from Da Nang and the quieter North, Saigon was the polar opposite.

On the flipside, I had the best coffee here coupled with getting some really good street shots.

We visited the Ho Chi Minh City Museum where, along with the Vietnamese history, we witnessed some intense wedding shoots. Yes, inside the museum. I also witnessed the city’s wildest nightlife at the Bui Vien street, it was crazy and mere inches away from being considered a stampede. Throngs of people out on the street drinking, smoking and just being plain crazy. I was glad that stretch was over when it finally was.

Ho Chi Minh is where we did most of our shopping at the famous Ben Thanh market. There are eateries all around the place and it gets especially lively at night when all the street eateries put their baby-chairs out for the locals.

Food

If you’re a vegetarian and are planning a trip to Vietnam, god forbid you’re going to have a hard time. Especially if you’re planning to do the Ha Giang loop or thinking of going deeper into the mainlands. In the cities, you’d still find a variety of vegetarian, vegan even, options. That being said, I think I personally enjoyed the Vietnamese cuisine the most out of this trip.

And of course, I’ve already mentioned the coffee. I didn’t have a single bad coffee during the whole time I was there and it was a sentiment most of friends shared too.

Observations

I always note down interesting observations during travel, I feel some of them are worth sharing:

• Vietnam is a matriarchal society and the difference is stark. Coming from India, I was pleasantly greeted with every other business being run by ladies. It was a fresh departure from a traditionally patriarchal Indian society. Not just that, being a matriarchal society, Vietnam naturally lends itself to a save environment for women. This is ofcourse, my observations from slightly more than 2 weeks there but from what I’ve read so far, it is indeed true.
• How the locals enjoy their downtime. Scores of families and friends would pour out onto the streets at night, especially pronounced during the weekends, to eat good food and watch some live music. It was a sight to behold especially since we rarely see something like this in India. In fact, I noticed a similar trend when I visited Nepal earlier this year, locals tend to enjoy their downtime to the fullest. This is possibly entirely anecdotal but unlike India, where you mostly only see people from upper-middle class seated in a Starbucks or a Third Wave in order to have a good time, I was pleasantly surprised to see people from all walks of life to a nearby Highlands Coffee. I don’t know the exact reason but I guess it has at least something to do with its communist and socialist roots.
• People are extremely friendly. We talked to so many locals in our entire trip and all of them were quite willing to engage in a dialogue. Every single one of them was extremely interested in knowing about India and our lives.
• Restarurants closing quite early, as early as 7:30 in the evening. I noticed the whole staff having dinner together shortly after pulling the shutters down. I saw similar groupings of the restaurant staff at the end of the day in other eateries too. I’ve never seen something like this back home.
• The amount of western tourists. It naturally got me thinking, if you were born in a western country, you automatically have a lot more avenues for travel open up for you. First off, it’s dirt cheap and secondly, you don’t have to deal with Visas most of the times. Put another way, what I’m trying to say is that there’s always going to be a cohort of people who’d never be able to see and experience what’s out there in the world just because of where they were born, and that’s depressing.

I can’t say if these observations are a helpful addition to the post or not, but this is what I seek when I travel, a new worldview, understanding of how people live their lives and how societies are different from what you’re used to back home and I thought they are worth sharing.

Conclusion

After a failed attempt to travel to Vietnam right before COVID in early 2020, I was really looking forward to explore this land of hardworking people, as the saying goes. I wasn’t disappointed. Additionaly, I paired this trip with a reading of Vietnam: Rising Dragon, it was the perfect companion.

I remember I was having this discussion with a friend at work about how travel is not what it used to be with the advent of internet. One can even “experience” a new land with all the innovations in AR. But writing this post, I realize there are things I’d have never even read about lest I had travelled to the place by myself and that makes all the difference. It expands your worldview, for the better.

Sometime last year while browsing Spotify Wrapped, Strava Year in Review and other yearly reports from different applications, I thought about doing something similar with my command line usage. I spend a lot of time on the terminal and was naturally drawn to bringing this idea to fruiton. I started recording my shell history in a MySQL instance as a starting point shortly thereafter. Back then I did not know what kind of insights I’d be able to draw out of the data that I’ll have eventually or whether they’ll be interesting or not. But the idea of taking a step back to realize patterns from such data to see how I’ve “evolved” in a year’s span sounded a little too interesting to simply discard.

Part of the motivation was also my Year in Review 2021 which I thorougly enjoyed writing and I thought I might as well start doing it for other aspects of my life too.

The way I’ve structured this post is to go through some high-level insights first and then digging deeper into some of the most used commands and gain insights out of those. I’ll skip explaining the setup and how I visualized the data for this post and park it aside for another post.

Data

I started capturing my shell history in a MySQL instance in January 2021. Post that, all the commands that I’ve hit on my machine, along with metadata such as the current working directory and timestamp have been dumped into that same database. As of writing this post, in December 2022, I’ve gathered:

• Total commands: 63,264
• Distinct commands: 13,477
• Total commands in 2022: 21,475
• Total distinct commands in 2022: 5,060

Also, since the database contains data from 2021 and this post is aptly titled 2022, I’ll add a date filter on all the queries to only fetch data for 2022.

Let’s now dig deeper into the data and see what kind of insights we can draw. Again, there is no structured index to this post, I’ve kept it quite loose. We’ll talk first about the usage patterns for the year and then we’ll talk in-depth about the individual commands. Finally, we’ll talk about optimizations.

Usage

Before diving into the actual commands, let’s take a look at the kind of usage pattern that emerges from this dataset to see what kind of command line user am I.

Weekly Usage

The graph below shows my command line usage on a weekly basis for the past 12 months:

That one huge spike in March-April was when I was working on a feature I had to push which involved a fair bit of programming and infrastructure work.

The two dips in July and October correspond with my vacations where I did some work but certainly didn’t hit the terminal. On average, I think the graph pretty nicely shows a weekly wave-like pattern having crests and troughs depending upon the kind of work I’m doing that week.

Day of the week

Is there a day on which I’ve been programming or hitting the terminal the most, due to a reason?

Turns out that’s not the case. Saturday and Sunday lines up just fine, I do some open source work on weekends. And then weekdays were expected to have taken up the majority share. One surprising observation here is how Tuesday is on par with Thursday considering that the latter is the day in my week with the most meetings.

Hourly Usage

Finally, let’s take a look at how the usage is distributed when it comes to the time of day:

Aligns quite nicely with my natural circadian rhythm. I’m most productive in the mornings/first half. I tend to block time for focused work (read time spent on the terminal or thinking) in the first half and then move as many meetings/ad-hoc items to the second half, post lunch.

An interesting pattern to note is that since my early mornings(6-9) are blocked for reading and working out, I’m more likely to stretch in the later half of the day(6-9) as evident from the graph above.

Lastly, I can count the number of instances where I had been on the terminal post 10:30 at night, those were neither deliberate nor fun but the need of the hour, but it’s interesting to see that I was able to keep the routine in check willingly over the course of an entire year.

Top Commands

Let’s now move to the actual commands. The graph below lists the top 13 (odd I know, metabase resize ftw) commands in the past 1 year, ordered by frequency. These are just the base commands, we’ll dig deeper into some of these commands in separate sections where we’ll talk about the subcommands and arguments.

I had an inkling that the top 3 would be a combination of git, kubectl and docker but not in that order. We’ll dig deeper into each of these top 3 commands in later sections to understand how my usage varied over time.

But before doing that, I have a few observations on the other commands that made the cut in the list above.

• cd: I was half expecting cd to pop in at least in the top 10 but then I realized I don’t use cd as much as I do z for my directory switching needs. It is a great tool to jump to a directory if you know the directory name and have used it at least once before.

• vi: is again, unsurprising, as it’s the choice of editor for all things editing for me. I’ve actually aliased vi to nvim and almost never explicitly type out nvim on the prompt.

• make: since this data is from my work machine, you can see make also being used quite extensively. I’ve been using Makefiles for Go projects at work and it fits right in. Since then, I’ve used Makefiles in a lot of other places where I need some repeatable commands to be run as routine manually, I usually opt for Make over bash scripts.

• To debug microservices locally or on a development cluster, I use curl but It’s surprising that it’s up there in the top 10 with ~600 hits.

• kx: finally, I work with Kubernetes a lot and have multiple clusters/projects at work between which I jump around time to time. I earlier used to use kubectl config use-context <> but realized it wastes a lot of time, both writing the command and then writing the context name. I switched to kx, which coupled with fzf provides a quick and nifty way to switch kubernetes contexts.

I think that gave a really nice overview of my command line habits. I have a couple of staple commands which I use day in, day out. I have made some optimizations (read alias) because I had an inkling that I’m using a particular command a lot but I didn’t have the numbers back then to corroborate my hunch. Thankfully, I can now make informed decisions on how I can(or should I) better optimize my command line usage. We’ll talk more about optimization further ahead.

Let us now pick each of the top 3 commands and see how the usage is and how it differs from the command.

Git

Below are the top git subcommands ordered by frequency in descending order. It shows the most commonly used commands on top and groups a few at the end of the top 20 spectrum:

As is evident from the graph above, I’m obsessed with checking the repository status. Everytime I’m away from the editor for a while and get back to it, I instinctively hit git status to see how I had left things.

I then check for what all have I changed so far and then further going into the tunnel, I add the modified/added/deleted files to the stage and then commit and push. Looking at it this way, it’s more of a funnel than a tunnel as I mentioned earlier. I’m more likely to run the status or the diff sub command than I’d add or commit or push those changes, and this makes sense logically too.

A note I’d like to make here is that I don’t type these sub-commands every time I need to use them. I’ve used git aliases for almost all of them, for e.g. instead of git status, I’d simply do git s. The expansion of these aliases has been done here for presentation purposes. You can find the complete configuration here. We’ll talk more about how this translates into time saved later in the post.

Kubectl

Let’s look at how I’ve been using kubectl for the past one year so far. The following graph shows the top sub-commands I’ve used with kubectl ordered by frequency:

Naturally, I mostly use kubectl client to get resources deployed in the cluster. Digging further into get, we can see how skewed the distribution is. Majority of the time, I’m dealing with Pods, HPAs or Deployments.

Another fun insight is the use of config sub-command. I had used it extensively to switch across multiple contexts earlier. But I soon realized this and I swapped it with kx as mentioned previously. Now, I use config only for renaming the automatically generated context names to something more human-readable using the following:

$ kubectl config rename-context $(kubectl config current-context) <new-context-name>

Also, the -n is again, specifying namespace for the command to follow. This definitely has skewed the data generated by the query but the problem here is the use of the default namespace for some of the clusters. Going forward, I’ll add a trailing -n flag instead of a leading.

Let’s now explore the usage for get and edit.

kubectl get

I mostly work across a bunch of different clusters under different projects and cloud providers, namely GCP and AWS. Different projects call for debugging/operations/monitoring on different resources but the common theme across all of them are, unsurprisingly, Pods, HPAs and Deployments.

I’ve seen other engineers at work using graphical interface tools such as Lens to view the cluster state at a glance, but for some reason, I still feel comfortable running these commands on the terminal and seeing if things are fine or not.

kubectl edit

I’ve spent a lot more time editing HPAs than any other resource. At work, I’ve to quite frequently scale workloads manually(unfortunately) due to AWS control plane constraints and hence so many edit invocations. What’s more interesting is that I had realized that I’m spending so much time doing gruntwork that I worked on a custom k8s controller to do this job for me.

Docker

Last of the top commands, let’s take a look at the sub commands which I’ve been using the most with docker:

I knew ps would be on top. This is akin to my usage of git status. I’m a sucker for checking status and keeping things clean.

The next is quite interesting, I use docker so much so that I have to regularly run docker system prune to clear dangling containers, images, volumes, etc. I’ve seen that If I go without running this command for a couple of days, I end up accumulating GBs of dangling data. I don’t know how effective this is in terms of the actual cleanup that docker does.

Nothing out of the ordinary about the rest of the subcommands if you’ve been using docker.

Optimizations

Now that we’ve seen quite a few insights from my command line usage for the past one year, is there a scope for optimization? i.e. can I save my time and make my time on the command line more efficient? I’m personally quite wary of yak shaving and premature optimization and am not going to mindlessly plunge into this quest but it makes for a good discussion especially as we end this post.

Using aliases

Unfortunately, the disclaimer I used about using git aliases is not true for kubectl, which means, If I have to get pods in a specific namespace matching app=frontend label, I’ll have to write this every time:

$ kubectl -n namespace get pods -lapp=frontend

Let’s do some napkin math get a quick idea on the kind of savings we can do If I were to be using an alias for kubectl. My typing speed is ~82wpm. Assuming the average size of the word is 4, the character per minute comes out to be 328. We can simplify it to roughly ~5 characters per second or 1 character every 1/5s. Now, let’s say, If I had aliased kubectl to just k as many people do. That would mean I’m writing an additional 6 characters every time. Based on the Top Commands graph, I’ve used kubectl roughly 2500 times in the past one year. This gives me a total time savings of (⁶⁄₅ * 2500)/60 => ~1 hour of time saved if I had used that alias. I think, going into the new year, I might as well add this alias and save that one crucial hour.

I can apply this to the git command too and save me 30 minutes which I honestly don’t think is worth it.

Subcommands aliases

Unlike git, not a lot of commands provide a way to alias subcommands. kubectl still provides shorthand for some of the commands. For docker, the commands are short in and of itself so it might not make a lot of difference to actually have a shorthand.

But I really like how git allows you to configure this and much more to make your life using git easier. I wished other commands would support something similar.

Switching directories

Another interesting observation was when I tried to map out the arguments for cd. Out of a total of 1347 matches, the distribution for the top 3 arguments to cd are as follows(refer to the legend):

Again, an obvious change here could be to just alias .. and - to do their jobs without using cd. I know some shells support this out of the box, I just need to configure this and I believe it’ll help consderably.

Also, please note that these three are topping the chart above because I don’t use cd for actually moving to a directory. As previously mentioned, I use z for that purpose.

Drawback

One drawback that I’ve noticed when using aliases or custom functions extensively is how I’m essentially paralysed while using a different machine or working briefly on someone else’s machine or even pair programming.

But, the flipside is that you spend a stark majority of your time one your own machine, so any potential downside arising out of using some other machine for a brief period of time will be miniscule against the time saved by heavily optimizing your own setup.

Conclusion

No inference to be drawn in this post. I just enjoyed the whole activity. Learnt a thing or two about when and how I use the command line. Did some overthinking about optimization that might save me some time potentially. I personally feel data is extremely important to make informed decisions. Insights from the data are fun and they can tell you about patterns that are otherwise extremely difficult to be found.

The kind of post this is, I’d love to hear from you if you have something interesting in mind in terms of what I just talked about in this post. It could be an optimization, a new tool, virtually anything.

Obligatory request, if you find any issues/errors in this post, feel free to reach out or raise an issue here.

:wq

This post doesn’t follow a weighted order, it’s just some of the practices I’ve personally used and (some) would prefer to use whenever I write Dockerfiles. This post also assumes you have a basic understanding of Docker and Dockerfiles.

Order your layers

The lowest hanging fruit out of them all. The basic rule of thumb here is to add statements dealing with changes to your code at the very end of the file. And conversely, keep things such as installing dependencies or other prerequisites, essentially things that are not supposed to change once you’ve written them down, near the top.

FROM go

COPY . .

RUN apt-get update \
    && apt-get install -y psutil vim curl make gcc

RUN go build -o app

Based on our example, the COPY directive is going to be executed every time you change your source. And the way Docker works, all the layers below line 3 are going to be reevaluated again. Fix this by moving the COPY directive to a more appropriate place.

FROM go

RUN apt-get update \
    && apt-get install -y psutil vim curl make gcc

COPY . .
RUN go build -o app

Format your Dockerfiles

Was not originally planning to add it here, but I’ve seen countless examples of unformatted Dockerfiles to actually include this as its own section. Unfortunately, there’s no standardised formatting tool such as gofmt available for Dockerfiles and so it’s difficult to attain a quorum during discussions or reviews but one thing I’ve seen that helps is to refer to official Dockerfiles or files from popular open source projects.

Some things I’ve gathered so far: 1. While installing packages, specify each per line for better diffs & readability. 2. Use empty lines. Group sets of instructions based on functionality. 3. Break chained commands and start the next line with an &&. 3. Make use of comments only if necessary.

...

RUN apt-get update \
    && apt-get install --no-install-recommends -y \
        curl \
        make \
        gcc \
        psutil \
    && rm -rf /var/lib/apt/lists/*

...

These are by no means definitive rules that one must follow but if you don’t follow any, these can help as a good starting point. You can then take it from there.

Discard the fluff

Pretty self-explanatory but I hardly see people following this. It could either just be an awareness issue or the fact that the ROI is not huge when it comes to cleaning up after a dependency install for example. But either way, maintaining hygiene of your codebase in general is a good practice. So, make sure you’re only really installing what you need. It also reduces the surface area for potential security issues.

1. Don’t install unnecessary packages that you feel could come in handy. As much as I like having vim in an environment, it’s just not a requirement for my Dockerfile.
2. Use --no-install-recommends with apt while installing packages, this tells apt to not install recommended packages along with whatever you’re trying to install.
3. Clean up package manager cache. For apt for e.g. use rm -rf /var/lib/apt/lists/*

FROM go

RUN apt-get update \
    && apt-get install --no-install-recommends -y \
        curl \
        make \
        gcc \
        psutil \
    && rm -rf /var/lib/apt/lists/*

COPY . .
RUN make install

Once done installing the prerequisites, we can go ahead and build our application. Here, we’re using a Makefile target which wraps the go install command internally.

Multi-stage builds!

If there’s to be only one takeaway from this article, let it be this one. Docker allows you to base your docker image off of other docker images as part of a single build.

Let’s say you built your application from source by installing all the prerequisites and other requirements and finally you have your binary ready for use. But you don’t really need all the other dependencies you installed in your final environment. Build stages can help us define multiple stages, for instance, in this example, we can have a build stage where we build our application. We can then have a second stage which can be based off a lean alpine image and since we have access to the previous stage, we can simply copy our binary from the build stage to our final stage. Multi-stage builds allows us to properly define separation of concerns to significantly reduce the final image size.

# Stage 2; builder stage
FROM go:1.17 as builder

RUN apt-get update \
    && apt-get install --no-install-recommends -y \
        curl \
        make \
        gcc \
        psutil \
    && rm -rf /var/lib/apt/lists/*

COPY . .
RUN make install

# Stage 2
FROM alpine

COPY --from=builder /go/bin/myapp /usr/bin/myapp

This will give you a docker image with a small footprint and without all the extra unnecessary fluff that Go would’ve generated in the previous stage (think cache, artifacts etc.)

Using multi-stage builds can drastically improve both the size and hygiene of your Dockerfiles.

Multi-stage builds also serve other important purposes, for e.g. allowing you to have a dev environment build within your primary Dockerfile or allowing you to build your dependencies in parallel (concurrency pattern) resulting in faster builds.

Use ARGs effectively

Using ARGs effectively can help you achieve DRY when you write Dockerfiles. Let’s say you’re trying to install a 3rd party application in your Dockerfile. And to avoid any unforeseen issues in the future, you prudently make it a point of downloading a specific version of that application. Let’s say we have the following hitherto:

FROM debian

RUN wget https://download.com/version=1.2.4 \
	&& tar -xzf application_1.2.4.tar \
	&& cd application_1.2.4

...

You can make use of an ARG directive here and avoid specifying the version every single time:

ARG APPLICATION_VERSION=1.2.4

FROM debian

RUN wget https://download.com/version=${APPLICATION_VERSION} \
	&& tar -xzf application_${APPLICATION_VERSION}.tar \
	&& cd application_${APPLICATION_VERSION}

...

This makes the Dockerfile much more readable and extensible should you need to install another version of the same package in future or any other modifications.

Now, an obvious question arises as to why can’t we use ENV instead of ARG and what even is the difference between the two. One way to think about it is that ARGs are the environment variables for your build step whereas ENVs are environment variables for your container environment when you eventually run your container.

Minimize layers

It’s always better to keep as few layers as possible. Not every Dockerfile directive creates a new layer, only RUN, COPY and ADD create new layers. But I’ve come across Dockerfiles containing every single bash command in its own separate RUN directive. Fewer layers results in a lower size of the overall image because more number of images adds more overhead, think compressing, metadata, cache, etc.

It also allows you to build logical cacheable units, for instance, in the snippet below, that one RUN layer is installing CRIU which can be reused in other builds safely.

FROM debian

ARG CRIU_VERSION=3.17
RUN wget https://github.com/checkpoint-restore/criu/archive/refs/tags/v${CRIU_VERSION}.tar.gz && \
    tar xvf v${CRIU_VERSION}.tar.gz && \
    rm v${CRIU_VERSION}.tar.gz && \
    cd criu-${CRIU_VERSION} && \
    make && \
    cp ./criu/criu /usr/bin/

Understand ENTRYPOINT and CMD

This is probably the most commonly misinterpreted of them all. Part of the confusion arises due to the different forms available for both ENTRYPOINT and CMD. But to keep it simple, I’m going to only consider the JSON form here while discussing a preferred way to use these two directives.

There are two common ways one would use a Docker image. They are either used as an interactive sandbox environment wherein you could exec and do some tasks. Or they are used as a binary. For instance, in our example above, we really just want to run this docker image and expect it to start our Go binary. This is the format that’s commonly used when it comes to usage within container orchestration systems such as Kubernetes.

That being said, if you’re using your Docker image as an executable, your Dockerfile should have either of ENTRYPOINT or CMD or preferably both. ENTRYPOINT defines the command that is supposed to run in your container, CMD specifies default arguments that are passed to the command specified by ENTRYPOINT. Let’s understand this better with an example:

FROM debian

...

ENTRYPOINT ["ls"]
CMD ["-a", "-l"]

Once built, when you do docker run <image>, it should execute the ls command with the two arguments specified by CMD.

This is not to say that this covers the complete difference between the two directives, but it should help you make informed decisions for the most common use-cases wrt to specifying commands and arguments for your Docker image. I can maybe talk about the in-depth differences (signals, pids, etc) that arises with the usage between these two in a separate post of its own.

Dockerignore

Having an up to date dockerignore file in your project’s repository ensures no fluff is added to your Docker images and additionally, it can prevent you from accidentaly adding credentials or secret files to your Docker images. I could’ve added this to the discard fluff section but if implemented, it really helps maintain the hygiene of your repository, like how .gitignore helps keeps your upstream all neat and clean.

Conclusion

This is the kind of post which doesn’t really call for an inferential conclusion so instead, I’m going to quickly summarize everything we talked about in this post:

1. Order your layers with the least frequently changes on top and so on.
2. Spend time sensibly formatting your Dockerfiles.
3. Keep the image as minimal as possible, discard all fluff.
4. Use multi-stage builds as much as you can.
5. Make use of ARGs wherever deemed necessary.
6. Minimize the number of layers in your Docker image.
7. Understand the difference between ENTRYPOINT and CMD.
8. Use .dockerignore in your projects.

As mentioned previously, I’ve curated this list of practices from my own experience with writing Dockerfiles for the past 3 years. There is a good chance some of this runs counter against what you follow or what should be followed. And if that’s the case, if you came across anything in this article that might not be right(highly likely) or if you have suggestions for improvements, feel free to report an issue or reach out to me directly via email.

Early March 2022, I and a friend of mine, planned a workcation to South Goa. We didn’t plan to go to the north side of the town just because we didn’t plan to party or visit any tourist spots. Given this was a workcation, we wanted to spend as much free time we could get on our hands relaxing.

Short layover at Mumbai while flying from Delhi to Goa. Travelled to Linking Road for some ad-hoc work. It was scorching and not pleasant at all, we rushed back to the airport as soon as we were done with the work.

Reached Goa. Dabolim airport is about 2 hours from South Goa (Canacona). We took a cab and on the way, the gentleman who was driving us made us try this local Goan delicacy called “Ros omellete” on a roadside food stall which he said he frequented often.

It had gotten pretty late by the time we reached the Airbnb. But the next morning, we were able to appreciate the location. It’s a beachside property with quite a few huts built within and hence accessing the beach was as tiresome as walking to the other room, can’t complain. It wasn’t exactly cheap but wasn’t exorbitantly expensive either. It was a quaint little place in between a lot of other shacks on the Canacona beach, very quiet with a small cafe which served mostly english breakfast.

The cafe was super quiet most of the time except for when the French family next doors would come there to have a jolly good time with their children. I’m glad I don’t understand French much and for me it acted as a rather cacophonous yet soothing background score for me to read with. I was able to read much more than usual on the trip, perhaps due to the peaceful surroundings. Mostly, I read Walden and it quite fit the setting.

With the beach so accessible, It ended up being a routine for us to go for a run early in the morning and then exploring the surroundings as a cooldown.

I had planned to work from here for a week and although I thought I’d be able to pull it off, getting a good connection was a struggle. Finally, I settled on attending meetings from the room (phone hotspot) and then working from the cafe on the beach with their provided WiFi so eventually it kinda worked out well. But going forward, I think I’ll have proper internet provisions sorted out before heading for a workcation next time.

Once the workday was complete, it was usually time for a swim. We realized it was the routine you had to follow there when we saw other tenants tossing away their headphones and hopping into the ocean once they were done with their meetings.

The next day was a Saturday and we had planned this to be the full-tourist day for us so we headed out for a breakfast early morning to this nice little Israeli place, Nireas which served the most authentic hummus I’ve ever had.

We then headed towards Cotigao Wildlife Sanctuary. It was quite secluded and rather empty. Safe to say, we didn’t encounter any wildlife in the hot forest and came back after a short hike. We also met a group of college graduates who inundated us with questions about computer science jobs (they had recently secured jobs and were out to celebrate!).

Post the hike, we went to another famous beach. It was very hot as usual and we just quickly did the customary photography and then a session of Kayaking in the backwaters. Kayaking, here, was more about showing off driving skills in a crowded bazaar, there were just too many folks there.

The day ended with a Goan thali from a nearby family-run place called Rama’s Homecooked Fish Thali. More about Mr. Rama at the end, he was an interesting and light-hearted fellow.

The week ahead mostly was me working throughout the day and finding time to head out for lunch and dinner at mostly the same places I’ve already mentioned before. Being a small and a closely knit community, the Canacona area didn’t had much to offer, but we were content with the offerings and making acquaintance with the few restaurant owners there.

Evenings were much more relaxed. Post the swim, for snacks, we frequented the only tea shop in the area which turned out to be a treasure-trove of a wide variety of people from different backgrounds and ofcourse some good snacks, like banana bread, coffee and pav.

Thursday was going just as usual, morning run, work and work until…

…our adventurous Airbnb host invited us on a fishing trip with her family. We tried a bunch of different locations around the rocky knoll and finally had to be perched up against the almost upright rockside like those mountain goats you see on YouTube. This was probably one of the most memorable experiences I’ve had in all my travels.

Friday was our last full day here, we had an early morning flight the next day and given how far the airport was, we had to leave in the middle of the night. Finishing work early, we decided to visit another beach, the Turtle Beach (Galgibaga) and as luck would have it, it wasn’t hatching season and we just saw the eggs carefully fortified with wooden fences along the beach.

It was a great trip. Worked, read, came to know new people and tried all the delectable food. Would definitely do this same trip again sometime. I think the true highlight of this trip were the people I came to know about. There was this one guy from New York who was sweeping tables at the chai shop, there was a group of guys working remotely from the Airbnb for over a month and finally, a young lady who ran an NGO and was teaching students over video call from the shack. Then there was our Airbnb host, running the Airbnb, managing her children and in the midst of it all, not forgetting to enjoy life, the next door grocery shop owner who was gracious enough to suggest us local dishes and helping us with other queries, then Uncle Rama from Rama’s homecooked fish thali, that guy was just happy running this 2-table family-run restaurant in his house’s garden. It’s interesting to know what all people are doing out there, it expands your world-view from the narrow confines of the usual 9-5. ¹⁰⁄₁₀ would do this again.

:wq

If you clicked on this post looking for insights into dietary experiments, I’m sorry. This is going to be primarily about media consumption in the digital format in our modern society. I’ll probably think about writing a post on my diet experiments some other day.

Problem

I’ve been through that unhealthy pattern of opening up a social media app and then just scrolling through mindlessly to find something, just something. That something doesn’t show up, of course, and you end up devouring unwanted information. I’ve come to call this kind of consumption Undirected consumption. Pretty self-explanatory, but this is the type of consumption when you don’t know what you’re looking for to consume but you’re consuming anyways. The flip side of this is Directed Consumption, which is a rather conscious decision about what you’re consuming. Let’s understand each of them.

Undirected Consumption

Undirected consumption is almost always is by design. Social media companies like Twitter, Reddit and Instagram have setup their homepages(feeds) to optimize for Undirected consumption. If you open up Twitter.com, you’re bound to see tweets not in chronological order but based on a certain heuristic designed to show you engaging content. Even more important is to show the content right on your face first thing you land on the page. And the cherry on top is the infinite scroll, you can comfortably spend countless hours on these things. This is extremely common amongst all such websites. However, I’m compelled to mention Google.com here for they have not done something like this with their search homepage(yet) and I wonder why but that’s a good thing (cue: iGoogle).

For example, let’s say you’re bored waiting for someone, and so you unlock your phone, open the first app in your recent applications list and start looking for a so-called serendipituous encounter with every new byte of information that takes up screen space on your device as you scroll.

I haven’t mentioned other apps like Snap, Tiktok etc here because I haven’t had much experience with them but I’m assuming they also optimize for Undirected Consumption given how successful and popular they are.

Directed Consumption

Directed consumption, on the other hand, is about a more conscious usage of social media. This means you’ll not open an application because you’re “bored”. You’ll not seek solace with irrelevant pieces of information in the form of text, images or videos. Unlike Undirected Consumption, which is optimized by the teams behind the product, Directed Consumption has to be optimized or rather put in place by the user. This means you’ll make it a point of not landing on the home page of such apps, or any app for that matter, to decide what to do. Instead, you’ll have a reason to be opening a particular app. This is very different from quitting cold turkey, which involves deleting or disabling one’s account completely.

For example, your friend shared a tweet with you over message. You’ll click on the tweet, read the content, have a discourse with your friend and then that’s about it. You don’t click on the twitter logo to open your, now infamous, feed because that will unknowingly trigger Undirected Consumption. And there’s a very good chance something like that could happen. As Tristan Harris mentioned once, “Some of the brightest minds of our times are working to take control of your attention using the latest technology.” So unless or until you’re a proven Stoic, don’t try to take that one cursory glance at your feed, chances are that you’ll have been played for by the time you realize.

Breaking out

How do you transition from UC to DC? There are various options. First and foremost is a willingness to commit because whatever approach you end up choosing can be undone just as easily. Once you’ve sorted things out in this area, you can leverage technology to structure or limit your consumption. These are some of the ways I’ve tried which have been working well for me for the past 6 months or so:

1. Site blockers: You’ve heard about these, these are apps that block a certain list of apps during a configured period of time. For e.g. LeechBlock, Freedom. Different apps provide you with different levels of blocking and the flexibility to disable those provisions to cater to people with different levels of discipline. Choose what suits you best.
2. Habit tracker: You can use gamification to build out this habit. I’ve tried maintaining a streak of “not using social media” which seem to work really well. Habit tracking in general is a really good tool to build strong habits which you stick to. I started seriously habit tracking after reading about it from James Clear.
3. Timelines: Social media is not all that bad either. I’ve made some friends online and have come across interesting information relevant to my career. But I’m not willing to sacrifice a majority of my time dilligently looking for such info. I’m okay with a perfunctory visit once in a while to check up on the hullabaloo. I felt it much better than totally giving up. So for e.g. I use whatever social media I want to use just on Sundays and not on other days.

Even if you try all of the above, there’s no guarantee that you’ll be transmogrified into a Directed Consumer out of the blue. But it’ll definitely make you think about it. To paraphrase Joshua Fields Millburn, we tend to think about the impace of things we were accustomed to but have now given up. This is to say, once we remove Undirected Consumption from our lives, we realize how much time of ours it used to take up.

Conclusion

The intent of this post was to put out this “theory” of our digital consumption model and how I’ve dealt with it. This is in hopes of helping or encouraging those who’ve had issues with our modern-day information overload and who want to do something about it. But as the old adage goes, being aware is the first step. And so I’ve seen a lot of folks who are not even remotely aware of what’s going on and it often becomes difficult to nudge them in the right direction. So If you’re reading this post, just think about how you consume media and whether you need to do something about it or not. Altruism is best avoided when it comes to helping someone with their digital consumption.

I’m tempted to end this post with this witty excerpt from Walden:

We are eager to tunnel under the Atlantic and bring the
old world some weeks nearer to the new; but perchance the
first news that will leak through into the broad, flapping
American ear will be that the Princess Adelaide has the
whooping cough.

:wq

There are certain deployments which, upon deploying for the first time, pull resources externally and may take a considerable amount of time before they are up and running or before they start responding successfully to probes(If you have them setup, you should if you haven’t). For instance, in one of the workloads that I was working with, if any changes were made to a configMap that was fed into that pod, the pod(s) would take anywhere from 15 minutes to 1 hour to pull the relevant artifacts before showing up as Running.

Now ideally, that shouldn’t be a problem but as part of our CD pipeline, we also check the deployment status of all our deployments to ensure all of them were rolled out without any issues before we trigger a subsequent pipeline. So, we essentially list all the deployments and then run the kubectl rollout status ... command on all of them in a sequential manner. It looks something like this:

DEPLOYMENTS=$(${KUBECTL} get deploy -o custom-columns=NAME:.metadata.name --no-headers=true)
for DEPLOYMENT in ${DEPLOYMENTS[@]}; do
    ${KUBECTL} rollout status deployment $DEPLOYMENT
    if [ "$?" -ne "0" ]; then
        echo "Rollout failed for deployment: ${DEPLOYMENT}, quitting."
        exit 1
    fi
done

If you’re wondering, yes, we can possibly also run all of them in the background using Unix job management.

Problem

Coming back to that special configMap change which leads to certain deployments taking a lot of time than usual. By default, Kubernetes waits for a deployment to come up and the pods to be in Running state for 10minutes. After 10 minutes, Kubernetes considers this deployment a failure. This is know as progressDeadlineSeconds. In other words, kubectl rollout status <deployment command returns a non-zero exit code. It’s default value, as mentioned, is 600s (==10m).

So in our case, whenever we make changes to this configMap, we basically end up having a successful deployment albeit a slow one to come up and a failed Jenkins job with failed follow-up triggers.

Solution

You can use the progressDeadlineSeconds in the pod spec to a value which aligns more with the kind of workloads you are running. In our case, it can be set to 3600s. This will ensure that Kubernetes will wait for an hour for the pods for your deployment to come up before deeming it as a failure. A sample deployment.yaml manifest would then look like:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      progressDeadlineSeconds: 3600
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

But that’s not all, if you have a setup which is anything like our i.e. if you are using the kubectl rollout status ... command, you additionally need to specify a --timeout=1h flag so that the kubectl client doesn’t pre-emptively return a timed out request and fail your pipeline yet again. The bash snippet shared earlier should now look like:

DEPLOYMENTS=$(${KUBECTL} get deploy -o custom-columns=NAME:.metadata.name --no-headers=true)
for DEPLOYMENT in ${DEPLOYMENTS[@]}; do
    ${KUBECTL} rollout status deployment $DEPLOYMENT --timeout=1h
    if [ "$?" -ne "0" ]; then
        echo "Rollout failed for deployment: ${DEPLOYMENT}, quitting."
        exit 1
    fi
done

Conclusion

This was supposed to be a short post and I’d reiterate again the use case depicted above is very specific but it’s interesting how Kubernetes caters to a wide array of use-cases and how there are, at least now, a myriad of Kubernetes features that haven’t gained mainstream adoption yet which I think is understandable.

If you found an error or an improvement in the post above, please feel free to report it.

The problem

A few requests on one of our workloads were timing out one fine day. Out of the blue, nothing out of the ordinary. Now, these timeouts would span over a few minutes and during that time, we’d also see quite a few of the pods failing their readiness and liveness probes. This happened twice in a period of ~1 month. The kind of workloads we were running, timeouts on certain requests at random wasn’t acceptable at all. So we had to figure out why this was the case, even though it was just for 2 minutes that one time.

It quickly became a priority and I went about trying to figure out what really went wrong. Before going into the post, a quick disclaimer that I’m going to deliberately avoid talking about Resource Requests in this post since they are in no real way involved in the aforementioned issue. You can read more about Resources in Kubernetes here.

Resources limits

A quick short primer. Kubernetes allows you to specify Memory and CPU limits for your workloads. There’s a lot of configuration options available when it comes to managing Resources for your workloads on Kuberenetes, for e.g. QoS, LimitRanges etc. We’re going to specifically focus on spec.resources attribute that you set in your pod manifest. The spec.resources.request property allows you to specify how much memory and CPU your pod would require. This information is used by the scheduler to schedule pods. But your application can certainly exceed their allocated/requested quota of resources and you don’t want other workloads running on the same node to be affected by a badly behaving workload. To avoid running into this situation, you can specify spec.resources.limits which “ensures” your workloads never goes haywire when it comes to resource utilization.

...

resources:
  limits:
    cpu: 100m
    memory: 256Mi
  limits:
    cpu: 200m
    memory: 512Mi

...

In the above snippet, the scheduler will find nodes which have 256Mi of free memory and 100m of CPU available. Once scheduled, the pods can use more than 256Mi of memory and 100m of CPU but not beyond 512Mi and 200m.

Memory limits

Memory being an incompressible resource, if your workloads are using more than the allocated memory limit, it will get killed. This specific event of killing a process because it is consuming more memory than it’s allowed is known as OOMKilled(Out of memory killed). If and when this happens, you should be able to figure out whether you’re actually experiencing an OOMKill event by going through the logs, the kernel or the container runtime dumps such events. With that, it becomes easy for you to gauge memory usage for your workloads and to decide whether you need to increase the allocated memory. Do note that in Kubernetes, an OOMKill will not restart your container if it’s not the init container.

CPU limits

CPU Limits work in an entirely different way. Being a compressible resource, the system based on certain heuristics, will allow or disallow CPU cycles to your process if and when your process starts to consume more than its fair share of allocated CPU. This disallowing of CPU to a process is known as throttling.

That’s all well and good but when it comes to visibility, unlike memory where you can clearly find OOMKill events, figuring out whether there’s any CPU throttling that your apps are experiencing is not trivial by any means. It’s not impossible either, the Kernel exposes throttling metrics but it’s not commonly supported by monitoring solutions and you’ve to set those up manually.

How CPU Limits are enforced

The Kernel uses CFS(Completely Fair Scheduler) to facilitate CPU allocation/disallocation to processes in a system. The CFS in turn uses two configuration options:

1. cpu_period_us: The CPU cycle interval which the schedueler uses to reset the used quota for a process. It’s default value is 100ms.
2. cpu_quota_us: The total runtime for which the process can use the CPU in a given period before being reset.

Now, let’s understand how this works. First of all, when you specify 1CPU as a limit for your workload, it means that the processes can use 1000 milicore/milicpu of CPU. In CFSSpeak, that means the cpu_quota_us would be set to 1000m. cpu_period_us on the other hand, is unchanged and is set to a default value of 100ms. Putting this all together, you can calculate when a process is eligible for throttling by:

cpu_quota_us/cpu_period_us * 1000 => x mili CPU

That is, once it uses x milicpu in the given cpu_period_us, the process will be throttled and will have its quota reset at the start of the next period.

Let’s try to understand this better with the following scenarios:

• You want your process to be able to use 1 CPU completely. If it goes beyond 1CPU, it’ll be, well, throttled.

cpu_quota_us=250ms
cpu_period_us=250ms

250/250 * 1000 => 1000m => 1CPU

• You want your process to be able to use 20% of a CPU:

cpu_quota_us=10ms
cpu_period_us=50ms

10/50 * 1000 => 200m => 0.2CPU

Okay, that sounds exactly like how it should behave but there’s a catch, if you’re running your applications in a multi-threaded environment or your pod or container (cgroup) is running multiple threads in and of itself, the cpu_quota_us is calculated across all the threads. Which is to say, if you’re running 10 threads and you’ve the following config..

running 10 threads

cpu_quota_us=10ms
cpu_period_us=50ms

(10)/50 * 1000 => 200m CPU => 200

quota per threads = 10/10 => 1ms

..you’ll exceed your quota in 2ms and your processes will be throttled for the remaining 48ms of that cycle. As mentioned, this is especially pronounced in multi-threaded environment and Kubernetes is a prime example. In fact, Kubernetes is the reason this issue came to light in the first place.

To state more bluntly, In the last example, when all the threads use up the allocated quota of 10ms within a period, the scheduler will not allow the process to run until the next period. This is throttling in a nutshell.

To add insult to injury, there happen[s|ed] to be a bug in various Linux kernel versions which aggravates this issue by throttling CPU for processes which are not close to their quota. This, again, is pronounced in multi-threaded environments like Kubernetes. Refer to this gist for a breakdown of what’s going on.

Solutions?

A lot of folks have experienced this issue especially when working at scale. But there was no single one-shot fix for this since there were multiple things going on. There was the obvious using up of quota within a period by multiple threads. Then there’s the Kernel bug which made matters worse. That being said, you’ve the following options:

1. Disable CPU Limits. If there are no CPU limits on your workloads, the CFS will not come into action and the whole chaos will be averted. This has its own issues in that it’s not feasible for folks to run their applications without a CPU limit. This comes down to the kind of workloads you’re running i.e. optimized applications that you’ve written yourselves or running 3rd party binaries which you don’t have any control over whatsoever when it comes to their resource usage.
2. Disabling CFS. This essentially is disabling CPU limits but you’ve to go through a different route to do this. And it might be possible that your cloud provided doesn’t have support for this yet.
3. Reducing the cpu_period_us to a lower value. The default value is 100ms which a little too high when it comes to high performance web services or other such applications. Because when the cycle is longer, the processes are throttled for longer before their quota is reset. This again comes down to the kind of workloads you’re deploying and it may or may not be suitable for all. For e.g. Monzo added support for this in their own fork of Kubernetes and have seen good results.
4. Update Linux kernel. This is reserved for last for a good reason. If you’re reading this, there’s a high probability that you’re using Kubernetes as part of a cloud offering for e.g. GKE, AKS or EKS, etc. If that’s the case, it becomes extremely tedious to update the Linux kernel version on your worker node images. Even if you manage to do this, it fixes part of the problem, CPU throttling would still be there as long as you have limits.

I personally tried removing the CPU limits and it worked as expected. For others, maybe disabling the CFS is easier or just waiting out for GKE et.al to come up with node images with the updated Kernel version.

Kubelet

But wait, there’s more. Remember I mentioned in passing that the pods also faced a lot of readiness and liveness probes failing? Well, I reached out to GCP for info on that.

My hypothesis was that during such a crisis i.e. the processes getting throttled and adding to that, the CFS bug in the kernel lead to the Kubelet also being in a resource crunch and hence unable to complete probes. The kubelet being unable to performa a network request to check the container health lead to a somewhat domino effect in the cluster. Pods failing the readiness and liveness probes would trigger new pods to come up unnecessarily. At the same time, some workloads already running at max would just run out of running replicas affecting ongoing traffic.

One suggested fix for this is to move to exec based healthchecks instead of httpGet allowing the Kubelet to not having to setup the context to send and then subsequently receive response over HTTP. But rather just executing a command in the container namespace.

Conclusion

It was quite a ride from seeing certain requests getting timed-out to understanding how CPU limits are enforced by the Kernel. Kubernetes, as developer-friendly as it is, also comes with a lot of rough edges and it’s very easy to get hit if you’re not careful.

References

• https://medium.com/omio-engineering/cpu-limits-and-aggressive-throttling-in-kubernetes-c5b20bd8a718
• https://github.com/kubernetes/kubernetes/issues/51135
• https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

If you found an error or an improvement in the post above, please feel free to report it.

Been planning to do this for a while but never got around to doing it. Ever since I started doing yearly planning and reviews (2020), It made all the more sense to do these reviews publicly. Not just because everybody else does it, that’s part of it. But more importantly, committing publicly naturally pushes you to accomplish the goals you’ve set for yourselves.

Work

Beginning of the year, I worked quite a lot on purely infrastructure-related problems which were really interesting but H2 2021 was more scattered in terms of the kind of things I was picking up. I wasn’t very impressed with the work I did in late 2021 especially. Would’ve loved for it to be more streamlined but that’s how it was. In terms of the stack, I continued working with Go, Kubernetes and the general cloud-native infrastructure domain for two main reasons. First, I like working on infrastructure and developer tooling in general, the stack I mentioned above lends naturally to these domains. Second and rather fortunately, I still work on infrastructure a little bit as part of my day to day work at HackerRank.

Projects

End of 2021, I finally got around to working more on open source which was super fun and refreshing to get back to. I also wrote a (fork) socket.io supplement emitter which I brought up to speed with the current main socket.io API along with other improvements, this was part of a problem I was solving at work. Secondly, as part of one of the yearly goals I had set for myself, I wrote a tool that uses AST(Abstract Syntax Trees) to format your Golang code, specifically the structs to make it more readable and easy on the diffs. I also wrote an accompanying vim plugin for it.

Looking back now, I realized I didn’t “hack” much in 2021 and I’ve considered that while planning for 2022.

Reading & Writing

I did quite a lot of reading this year, very diverse. You can find all the reviews of the books here. I also finally managed to automate syncing reviews with Goodreads, thanks to Arpit for the API. After a year-long gap with audiobooks, I was back into it, partially because Audible generously provided a 3-month membership for free twice! If I had to pick one non-fiction and one fiction I enjoyed the most in 2021, I’d say Perfect Health Diet by Paul and Shu-Ching Jaminet and Kim Jiyoung, Born 1982 by Choo Nam-Joo.

Writing, on the other hand, was a disappointing affair. Just go back one page and you’ll see that I’ve written just 2 articles in the past 1 year. I’ll save you the effort:

$ find _posts/ -type f -name "2021-*" | wc -l
2

Again, this is something I’m going to actively focus on in 2022 (this post is part of the initiative:)) I’ve also started Journaling again after doing it in 2019. I don’t have a clear goal in mind with it but I’m hoping I’ll somehow write more this year because of it and simultaneously reap the mindfulness benefits.

Health

Gyms were closed for a good portion of the year and only when the second wave of COVID in Delhi passed(more on that later), they were allowed to open again. I started with PPL (Push, Pull, Legs) and then later switched to a slightly modified variant of 531 by Jim Wendler. I was doing 531 for the second time and this time around added pull-ups and chin-ups as assistance exercises. Thanks to the overload, I suffered from bicep tendonitis and a rotator cuff injury. I haven’t gone back since.

I finally caved in and bought a cycle, did a lot of it when there were very little COVID cases and when the pollution in Delhi made it possible, I found it to be a surprisingly cathartic experience. I also finally learned swimming. I’m able to front crawl, do backstroke and was still perfecting breaststroke when the pools closed for the winter season. I’m happy with the results though. Finally, badminton has been more or less consistent over the past year, it’s the only anchor I have most of the days when work becomes too much.

Other

On travel front, I was able to make two domestic trips this year, Andaman and Mussoorie. Both were fun trips but not as relaxing as I thought they’d be.

I’ve been consistently learning Japanese for a good chunk of the year(actually every day as per Anki). Started learning just the vocabulary using Anki and later on took a 3-month course with two great instructors. I’m comfortable with Hiragana and Katakana. Kanji, on the other hand, is a totally different beast altogether and so far, I’ve only learnt about ~300 words. I’m planning to appear for JLPT sometime this year. We’ll see how that goes.

Remote work has its ups and downs. During the honeymoon phase of it, I was very excited about all the time I’d save and the things I’d do with that time. But it turns out that when you’re working from home, it’s very difficult to draw a line between work and life. Even if you do draw one, it’s a blurry one. On top of that, you’re missing out on a lot of necessary social interactions due to remote work. A friend at work recently mentioned that he barely knows all the new folks that have joined the company and that rings true with a lot of us. I’m happy that HackerRank is now a remote-first company officially and employees can work from wherever they feel the most productive. I’m thinking of making optimal use of this policy but I don’t have a clear picture of what or how.

Lastly, COVID has been extremely tough for everybody and I’m grateful to be where I am right now. The 2nd wave was extremely distressing. I jotted down some observations on the societal impacts that came with it. As I write this in January 2022, things are becoming normal again after the 3rd wave but it really feels like it’s never going to disappear completely.

2022

I have some specific areas I’ve identified which I’m going to be focusing more on in 2022, most notably:

• Education: Learning new things, reading about more diverse topics, focusing on retaining more information that I’ve read (needs a post of its own), etc.
• Health: Gaining more strength, continuing with lifting, cycle more, swim again.
• Career: Taking more ownership, focusing on good engineering, learning more about how to interview candidates.

That’s about it for this review, I had a ton of things to say but that’ll be me deviating from the topic so I’ll try to branch out and try to post more. See you again next year for a similar retro.

Thanks to Arpit for help with a bunch of stuff, Robert for some very valuable feedback on the article and Sarthak for pointing out errors.

:wq

On the contrary, folks in the IT industry, like me, have had it easy. Much easier I must say. We all started working from home at the earliest signs of the pandemic and we’ve been getting paid more or less the same. All of my friends in the IT field are living a more comfortable life, they can work from the comfort of their home and not bother about commuting to office. Some of us have also been given stipends to buy equipment to set a home offic so that we’re more productive, we’re also occassionaly invited to virtual events organized by our orgs to play online games with our colleagues. Apart from mental burnout which is often the case in IT especially in India, things have been comfortably good. The pandemic for folks in IT just meant a whole new way of working. Sure, there are challenges with working remote but nothing that cannot be solved. Now If you compare it with some of the other folks in other industries, you’ll realize how hard it has been for them. I remember during the peak Covid cases surge here in Delhi, India, a known vegetable seller in our vicinity asked asked me to help him with applying for an e-pass so he can go door to door delivering vegetables in person. Imagine, the whole world being shit scared to step out of their homes and this guy is desperately seeking a way to do the exact opposite. Why? Because he can’t have customers come to his shop due to the lockdown and since it’s his only source of income, there’s no other option available for him. You’ll find plenty of such examples out there if you try to look for them.

Privilege is the word that comes to mind when I think of such things. I realized Covid has just brought this disparity out on the surface, it’s always been there, this divide, carefully maintaining the intangible equilibrium that a lot of us conveniently and subconconciously (often conciously) choose to ignore. There’s an obvious segregation in our society, of those who are privileged and those who are unfortunately not and it’s evident as soon as you step out of your house. Just the other day, I was cycling to the gym early in the morning and halfway to the gym, I pass through a shiny blue Bentley Mulsanne (costs around ~6 crore INR). Right next to the car, on the footpath, I see a person using water collected in a puddle from the rains overnight to take a bath (ref: Matthew Effect). Imagine having to come across a sight like that early in the morning. Another time, during one of my cycling runs, I came across an indigent individual on the road on a signal in one of the more affluent areas of Delhi, having a meal (rice and some curry) directly off the road. It will give you the chills, these images, if you were to experience them first hand. These are stories from the national capital, I don’t even know what’s going on in other parts of the nation let alone the whole world, I just hope it’s not this bad. Now you might think that’s a sure-shot case of socioeconomic inequality and it is, but it’s not just that, it reeks of our failure, the society as a whole, to at least try to ensure that nobody has to go through something like that. It’s distressing.

By now, you must be asking, what did I do to help those folks I just talked about, and truth be told, I did nothing. I didn’t know what to do, I didn’t know what to say to a guy who’s eating off a road. I didn’t know either what to say to a guy who’s using puddle water to bath. I’m still not sure. Unfortunately, there are no right answers here at this point, not from me at the very least. But until there is one, I try to support local NGOs who are focusing on education and food shortage amongst the needy. I honestly don’t know if it makes a difference and If you’ve suggestions around this, I’d love to talk about it.

What’s the point of this post? there isn’t one, to be honest. I was reading this post by Armin Ronacher of the same title and I found myself thinking about these things I’ve experienced in the last year and a half. I thought I should put it out there for two reasons I guess 1) So that I have a clearer idea of what I believe in when I think of a fulfilling life and 2) Just like how I was inspired to write this piece, maybe it’ll help someone else pen their thoughts down and share it with the world so that more of us are aware of our privileged background and upbringing and to not take it for granted.

Closing this weird(for me) post with this one line I heard from one of Vir Das’s sets, not sure which one. Here’s how it goes:

"..there will always be someone who has it better than you,
and there will always be someone who has it worse than you."

Be humble.

core_pattern

A core_pattern by the way is a pattern which defines the name and path for the core dump file. A core dump file is generated when a program receives certain signals. It contains an image of the process’s memory at the time of termination. You’ll almost never use it directly but will pass it to a debugger (GDB) to inspect the program and to potentially successfully figure out why the program died. Most commonly it’s a segmentation fault hinting that your program tried to access an “unreachable” memory location.

core_pattern and k8s

If you’re running your application on containers deployed using Kubernetes, /proc of the underlying machine is mounted onto the containers with certain masks and the fact that it’s read only. This means our core_pattern would be propogated from the host’s /proc/sys/kernel/core_pattern to our containers for our applications to refer to. Now this might not be a problem if, by default, the core_pattern doesn’t route the core dump to certain debugging or monitoring programs like how Ubuntu does with apport.

But if whatever machine you’re containers are deployed on does route the core dump to another application or has a pattern which includes specifiers like timestamp or TID, PID or hostname or a combination of all those, it becomes tedious to parse. Besides, if you’re running a container, you’re pretty much isolating the application’s access to some extent and for that same reason, you’d want a core dump to be generated in the current working directory and be done with.

I faced this issue recently with our application deployed on GCP when the GKE cluster upgraded to a newer available version. This update broke our application which relied on a core dump to be generated in the current working directory. It was the dreaded core_pattern being changed.

Multiple solutions sprung, first and foremost being somehow being able to configure the node pool for the nodes to come up with the updated core_pattern, a init script of sorts but turns out that isn’t possible with GKE right now. While GCP supports configuring certain kernel options, it doesn’t support core_pattern attribute as of now.

DaemonSets or InitContainers

Upon searching a little more for a possible workaround of the same, I came across a few Github issues which suggested using a DaemonSet or an InitContainer on your application Deployment resource to invoke the sysctl command before your application runs.

With a DaemonSet, you make sure as soon a new node comes up, you’re going to run the sysctl command to update the core_pattern like so:

apiVersion: "apps/v1"
kind: "DaemonSet"
metadata:
  name: "sysctl-corepattern"
  namespace: "default"
spec:
  selector:
    matchLabels:
      app: sysctl-corepattern
  template:
    metadata:
      labels:
        app: "sysctl-corepattern"
    spec:
      containers:
        - name: "sysctl"
          image: "busybox:latest"
          resources:
            limits:
              cpu: 10m
              memory: 10Mi
          securityContext:
            privileged: true
          command:
            - "/bin/sh"
            - "-c"
            - sysctl -w kernel.core_pattern=core.%e.%p.%t && sleep 365d

You’ve to make sure the container is run in privileged mode(caution). We’re also using the busybox image and we have a nominal amount of resources being allocated to the container. The command simpy updates the core pattern and sleeps for long long time. Why? DaemonSets are meant to run daemons or long-running programs which are essential to your application’s functioning. If your daemon exits, it is restarted which in our case would be pointless. So we update the config and let it sleep.

If you don’t quite like the idea of the hack above, you can run your sysctl command in an InitContainer as part of the DaemonSet like so:

apiVersion: "apps/v1"
kind: "DaemonSet"
metadata:
  name: "sysctl-corepattern"
  namespace: "default"
spec:
  selector:
    matchLabels:
      app: sysctl-corepattern
  template:
    metadata:
      labels:
        app: "sysctl-corepattern"
    spec:
      initContainers:
        - name: "sysctl"
          image: "busybox:latest"
          securityContext:
            privileged: true
          command:
            - "/bin/sh"
            - "-c"
            - sysctl -w kernel.core_pattern=core.%e.%p.%t
      containers:
        - name: pause
          image: gcr.io/google_containers/pause
          resources:
            limits:
              cpu: 10m
              memory: 50Mi
            requests:
              cpu: 10m
              memory: 50Mi

Here, we’re doing the same thing but in an initContainer. Note that we’re using the pause image as an equivalent of sleep 365d from the previous example. Now the pause container is meant to be used as a container which keeps the pod retain it’s network setup. Normally when all the containers in a pod die, the network namespace dies and is setup from scratch. Pause keeps the network setup in place while the main/appliation container restarts. Pause, in fact, does nothing but sleeps until it receives a particular signal. Refer to the man page and the source.

I don’t prefer this version since it’s just another version of sleep hack albeit a little more sophisticated.

The last approach that I came across was to run an initContainer as part of the Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      initContainers:
        - name: "sysctl"
          image: "busybox:latest"
          securityContext:
            privileged: true
          command:
            - "/bin/sh"
            - "-c"
            - sysctl -w kernel.core_pattern=core.%e.%p.%t
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

Ok, that’s exactly the same as the one with the DaemonSet but you get the idea.

There are a few problems with this approach though:

1. You wouldn’t want to slow down your application start up time by making it run the initContainer first which would in turn involve pulling the image if it’s not already present on the node.
2. You don’t want the sysctl command to run for every pod that ends up on that node, just running it once would’ve sufficed but this would make it run for every single pod coming up on every node. No thanks.
3. Finally, if you’re running an application which handles data at scale. You’re most certainly are or have optimized for auto scaling and with that, you’re not going to willingly add additional overhead to your infrastructure (i.e. the previous two points).

Conclusion

I ended up using the first approach i.e. DaemonSet with sysctl and a sleep since If you’re anyways going to use a hack, try to keep it simple for the sake of readability and maintainability. I would’ve loved to have linuxConfig support changing this configuration on the node level so as to prevent that extra set of resources being hogged on a sleep.

Introduction

runc is a lightweight portable container runtime parts of which were internally used by Docker and were packaged as a single binary and released as an open source project under the the Open Containers Initiative (OCI) as a way of giving back to the community[1]. Explained simply, runc is a lightweight tool written in Go which helps manage a container’s lifecycle i.e creating, running, killing and deleting a container. You’ll go through each of these steps in this post and see how runc differs from Docker when it comes to running containers.

You’ll work with the official golang docker image as a running example for the purpose of this article. You can find more details about this image on Docker Hub.

Requirements

Unlike docker, runc doesn’t abstract tedious tasks under the hood and rather expects you to arrange whatever is it that is required for it to work. Let’s run through the requirements:

runc Binary

runc is a portable piece of software, for GNU/Linux users, it’s as simple as fetching the binary and giving execute permissions to it in order to work with runc:

$ curl -L -o /usr/bin/runc https://github.com/opencontainers/runc/releases/download/v1.0.0-rc10/runc.amd64 && chmod +x /usr/bin/runc

You can head over to the project’s release page to download other versions. It’s also worth noting that runc doesn’t work on macOS so if you are on a Mac, your best bet is to run runc inside a docker container.

runtime-spec

Specifying configuration options such as volume mounts, memory limits or uid:gid mapping while running a container with docker is as simple as specifying command line options to the docker command. While using runc, these configurations are passed to runc as a file. This configuration file, the runtime-spec, is a configuration standard put in place by the Open Container Initiative (OCI) to specify options for a container and is used by runc[2]. In simpler words, the runtime-spec is a JSON file named config.json consisting of configurations pertaining to a specific container.

You’ll go through a few attributes in the runtime-spec in this post to get enough understanding so as to be able to run containers. Later sections will focus on attributes which require further explanation. Consider this sample runtime-spec snippet:

{
	"ociVersion": "1.0.1-dev",
	"process": {
		"terminal": true,
		"user": {
			"uid": 0,
			"gid": 0
		},
		"args": [ "sh" ],
		"env": [
			"PATH=/usr/local/bin/
			"TERM=xterm"
		],
		"cwd": "/",
		"capabilities": { ... },
		"rlimits": [{ ... }],
		"noNewPrivileges": true
	},
    ...
}

runc expects the configuration file in the exact same format with a certain required fields and a few other optional ones. You might have figured out a few options from the snippet above alone for instance, the environment variables, the current working directory and the uid:gid mapping. There are other configuration options which give you more control over how the container should perform that you’ll go through in the sections to follow.

Root filesystem

This is exactly what it sounds like, root filesystem for the container. There are multiple ways you can get hold of a root filesystem from a Docker image but the easiest and the one advised by the OCI is using docker export:

$ docker export $(docker create golang) | tar -C rootfs -xvf -

This will generate the unarchived root filesystem for our golang docker image which you’ll now use to create containers using runc.

Now that you’ve gathered up the required prerequisites, you can now go ahead and run containers using runc.

Running a container

Before running the container, you’ve to make sure you have all the prerequisites in the required manner. The runc binary, by default, looks for the root filesystem and the runtime-spec in a directory which is referred to as a “bundle”. The bundle directory must have the runtime-spec(config.json) and the root filesystem(rootfs) in order for runc to work.

~/golang
.
├── config.json
└── rootfs/
      ├── home/
      ├── tmp/
      ├── ..
      └── ..

As shown above, having such a directory, you’ll be able to run runc commands in the ~/golang directory for it to work appropriately.

With that out of the way, you can go ahead and create the container:

$ runc create golang

# list active containers
$ runc list
ID              PID         STATUS      BUNDLE          CREATED                        OWNER
golang          39          created     ~/golang        2020-04-12T14:23:46.7358607Z   root

# starts running the user-defined command inside the container
$ runc start golang

$ runc list
ID              PID         STATUS      BUNDLE          CREATED                        OWNER
golang          39          running     ~/golang        2020-04-12T14:23:46.7358607Z   root

# creates and starts the container
$ runc run -d golang
$ runc list
ID              PID         STATUS      BUNDLE          CREATED                        OWNER
golang          39          running     ~/golang        2020-04-12T14:26:46.7358607Z   root

Contrary to the title of this section, there are multiple commands to “spin up” a container using runc. In the snippet above, we used the following commands:

• create: creates an instance of the container from the bundle but it doesn’t run the command specified as an entrypoint in the runtime-spec(config.json).
• start: runs the init command(specified in the runtime-spec) for a created container. This essentially means your container is now “running”. Your application or whatever command you’re running inside the container can now respond to requests or do the job you intend it to do.
• run: it creates and starts the container. Passing a -d will run the container in detached mode. You’ll be using this command more frequently than individually creating and then starting the container.
• list: lists containers you’ve created, they could be running, paused, stopped or in created state. (not an exhaustive list of states). This is most useful to inquire about the current status of the containers that you’ve started or stopped with runc.

You can now “spin up” containers with runc, killing or deleting containers with runc is also fairly straightforward.

Stopping a container

Once you’ve started the container and played around with it, you need to discard it in such a manner that it does not run in the background holding onto resources or populating your pid tables for virtually no reason.

Although there’s no stop command provided by runc, the runc kill command, when run without any options, sends a SIGTERM signal to the init process of the container thereby stopping the container. This moves the container to a stopped state. Your container might still be holding on to certain resources used by processes contained by your container though. As a side not, while you’re at the topic of killing of processes inside containers, it’s worthwhile to go through issues associated with handling of zombie processes in docker.

The runc delete command frees up any resources that a container is holding on to thereby completely removing the container. You won’t be able to inspect this container in any way (checking filesystem or otherwise), and it will not show up when you do runc list.

# creates and starts the container
$ runc run -d golang
$ runc list
ID              PID         STATUS      BUNDLE          CREATED                        OWNER
golang          39          running     ~/golang        2020-04-12T14:26:46.7358607Z   root

$ runc kill golang
$ runc list
ID              PID         STATUS      BUNDLE          CREATED                        OWNER
golang          39          stopped     ~/golang        2020-04-12T14:26:46.7358607Z   root

$ runc delete golang
$ runc list

You’ve sent a signal to a running container and then released any resources held by that container in stopped state to completely “stop it”. Read more about additional options you can use with these commands by running runc <command> --help.

You can now manage a container using runc, you’ll take it one step further and see how you can checkpoint/restore a container with runc.

Checkpoint/Restore a container

Similar to docker, runc provides a handy functionality to checkpoint/restore running containers. You can think of checkpointing a container as serializing the state of a running process and then storing the contents on disk. This serialized blob has all the metadata required to restore the process to the same point from when it was checkpointed at a later point in time. That’s an extremely simplified explanation of how criu works. It’s a lot more complex than it sounds but it’s adequate for the purpose of this article.

To accomplish this, runc uses criu, an open source piece of software which does the actual checkpoint and restore. In order to use runc commands such as checkpoint and restore, you ought to have the criu binary installed on your host system. You can find instruction for installing criu here.

# run the container in detached mode
$ runc run -d golang
$ runc list
ID              PID         STATUS      BUNDLE          CREATED                        OWNER
golang          39          running     ~/golang        2020-04-12T14:26:46.7358607Z   root

$ runc checkpoint golang

$ runc restore golang
$ runc list
ID              PID         STATUS      BUNDLE          CREATED                        OWNER
golang          39          running     ~/golang        2020-04-12T14:26:46.7358607Z   root

In the above snippet, you checkpoint the golang container while it was running. When you checkpoint a container, criu stores the checkpoint metadata in the current directory. You can restore the golang container to the same state it was in at the time of checkpoint at any later point in time. You’ve to make sure that the files created by criu are not modified in any way whatsoever.

This might not be readily clear as to why or how this would be useful but there are cases when such a functionality can help you improve your workflow, some of the usecases I can think of are:

• If container startup time is something that’s holding you back i.e. If you have a container in which the init process takes time to startup. You can start the container, let the server come up and running, checkpoint the container and then just keep on restoring the container. You are essentially discarding the inital overhead of waiting for the init process to be ready. A common candidate for this could be your database servers.
• If you are running your application in an environment in which CPU/Memory usage is a critical metric, it would help to keep the checkpointed container on the filesystem and restore it right before serving a request, might be useful for jobs which need to run at intervals while maintaining internal state both in memory and disk.

There might be other more sophisticated use cases but I think checkpoint/restore is a great way to save up on time and resources depending upon the requirements.

You can, apart from managing a container lifecycle, also do things like checkpoint/restore. It’s imperative to focus on the runtime-spec now as it is what controls how the container is run, what resources it gets and other configuration options.

runtime-spec

The runtime spec, as discussed above, is a set of configuration options stored in a file read by runc to apply configuration before running the container. In docker, you can explicitly set some of the options that you can provide in the runtime-spec. Going through the most common configuration options, you can:

• Limit the amount of memory that can be allocated to the processes running under the new cgroup inside the container. For instance, the snippet below will instruct runc to limit the amount of memory available to the container to 2068MB. A value of -1 means unlimited memory.

  "memory": {
  "limit": 2168455168,
  "swap": 2168455168,
  "kernel": -1
  }
  

• Apply limits on a per-process basis, for instance, you can limit the number of open file descriptors that a process can open. You can specify all the rlimits here, refer to the official manpage for an exhaustive list.

  "rlimits": [
  {
      "type": "RLIMIT_NOFILE",
      "hard": 1024,
      "soft": 1024
  }
  ]
  

• Control the number of processes that can run inside your container. If applied, the following will limit the number of processes to just 1024. This can be crucial if you’re running code from untrusted vendors because they can very well turn into a fork-bomb and you might end up losing your container. On the flipside, don’t go overboard with this limit, almost all the programs uses fork/exec to start subprocesses and limiting those could end up hampering your container’s functionality.

  "pids": {
  "limit": 1024
  }
  

• Allow the rootfs to be readonly or not. With the following configuration, the rootfs of the container will be readonly i.e. no writes would be allowed:

  "root": {
  "path": "rootfs",
  "readonly": true
  }
  

• Run processes in the container as a different user:group. The following snippet runs the container as user 101 and group 201. If you’re doing this, also make sure to check if you need to give your container setuid and setgid capabilities.

  "user": {
  "uid": 101,
  "gid": 201
  }
  

• You can, like Docker, mount volumes from the host to the container. The following mounts /home/Documents/images from the host to the container at /mnt/images with read and write permissions:

  "mounts": [
  {
      "destination": "/mnt/images",
      "type": "bind",
      "source": "/home/Documents/images",
      "options": [
          "rbind",
          "rw"
      ]
  }
  ]
  

There are lots of other configuration options in the runtime-spec that you can use to configure your container as per your requirements. It would not be wise to discuss them all here for the sake of brevity and keeping this post relevant. You can head over to runtime-spec’s Github repo for all the options available.

Conclusion

You learnt how to manage a container’s lifecycle using runc, a lightweight, portable container runtime. It is highly unlikely you’ll use runc to manage containers on a daily basis, docker is much more user friendly and convenient for that especially when you factor in the added time and effort to gather the pre-requisites for runc to do it’s job. On the other hand, runc can prove to fill in gaps Docker can’t, these can vary from performance to the lack of additional, often unnecessary features built into docker that runc doesn’t come bundled with. But those are decisions for you to take a call on. Anyhow, it’s always a good idea to learn something new just for the fun of it.

If you found something in this article that’s incorrectly stated or can be improved, feel free to raise a PR or contact me.

References

1. [https://www.docker.com/blog/runc]()
2. [https://github.com/opencontainers/runtime-spec/blob/master/spec.md]()

I journal daily(not anymore) but I’m not a fan of using pen and paper although they are supposed to be better at this task. There are two reasons for this, first, digital note taking is easier to organize and get back to and second, at some point in time, I’d like to do a textual analysis on my journalling data (r/dataisbeautiful).

In order to make the process as painless as possible, I’ve created a setup for journalling using Vim which has been working really good for me lately. I’d walk you through my setup in this post.

Basics

I use markdown formatting for writing most of my documents including my journals and other notes since they are easy to format, are used universally and have good support by most applications be it web or desktop. You can easily export them to a plethora of document types, LaTeX flavored PDF being one of them.

Secondly, the choice of editor despite the title, is (neo)vim, if you’re reading this article, you probably know the difference between the two, even if you don’t, it wouldn’t really affect anything you might get out of this article, so read on :D

With that out of the way, let’s move on to some more interesting pieces of this setup.

Vim templates

A template in Vim is a simple .skeleton file which can be used to populate buffers based on certain rules. For instance, if you find yourself adding the shebang header to every shell script you write, you can “automate” it using a skeleton file so that everytime you open a .sh file, your buffer will be preloaded with the text you’ve specified inside the .skeleton file, which could very well be just the following:

#!/usr/bin/env bash

You can create your own templates and place them in a directory, I prefer the ~/.config/nvim/ directory itself just so that all the configuration files remain together. I have kept my journal template very simple with the following contents:

## What went wrong today?


## What went right today?

We’re going to add one more piece to it later on in the post but this is mostly it. As I mentioned above, you can have it your way, this is just how I prefer to write my journal everyday.

Autocommands

Autocommands are a great way to automate functionality in vim. Here, we’ll use autocmd to populate our buffer from our skeleton file when the buffer metadata matches a certain pattern. My journal directory looks something like this:

journal/
├── 2019
│   └── 01
│   └── 02
│   └── 12
└── 2020
    └── 01
    └── 02

And for the same, I require that every .md file that I open inside ~/journal/* should be populated with my journal template. We can do that by using the following autocmd:

autocmd VimEnter */journal/**   0r ~/.config/nvim/templates/journal.skeleton

The above autocommand is triggered on the VimEnter event i.e every time you enter vim from the command line. It is triggered on the pattern */journal/** which recursively matches it’s subdirectory. Finally, it reads and populates the active buffer with the journal template using the r(read) command.

We’ll also wrap our autocommand(s) in an augroup which helps removing and executing a group of autocommands together, it also makes it easier to organize your autocommands and eventually your vimrc. We’ll add more autocommands to the journal augroup as shown below:

augroup journal
    autocmd!

    " populate journal template
    autocmd VimEnter */journal/**   0r ~/.config/nvim/templates/journal.skeleton
augroup end

That looks much more readable, we will add subsequent autocmds to this augroup.

Completion

Another important piece here is the completion source. For instance, a month into your journaling journey, you will be repeating or referencing a lot of words whether that’s names of people, places or some technical jargon related to work. In order to minimize the effort and keystrokes required to repeat such words, we can make use of vim’s completion feature which allows us to select custom sources. We can set a custom completion source by setting an appropriate value for the complete option:

autocmd VimEnter */journal/**   setlocal complete=k/Users/danish/journal/**/*

The interesting thing to note in the above command is the setlocal complete=... command being triggered upon opening a new/existing file which matches the glob pattern specified. The k char before the file path tells vim to scan and source words for completion from the path that is specified local to the buffer matching the pattern and event. I urge you to read more about this on the help document, there are a bunch of potentially other useful flags for this option.

Note: In order to get Vim to scan the files recursively, this answer pointed me to the right globbing pattern to use.

Header

I write the present day’s journal the next morning, so in the morning when I open Vim to write the previous day’s entry, I like to have the date populated there along with the templated content, something like this:

# 22-02-2020


## What went wrong today?


## What went right today?

Since this is dynamic, we can’t make use of Vim templates here. Instead, let’s use a combination of the shell and some Vimscript here. First off, I have created the following alias in my shell:

journal='nvim $(date -v-1d "+%d-%m-%Y").md'

Using simple command substitution and the date utility, this will open (n)vim with the filename of the buffer as the previous day’s date. For instance, If I’m writing the previous day’s entry today (23/02/2020), the filename would aptly be named 22-02-2020.md.

Now, as soon as this command is executed, our templating autocommand, which we had setup previously, will spring into action and will populate the buffer with our skeleton file. To add the header, which is now the filename itself, we will use some good’ol Vimscript:

" set header title for journal & enter writing mode
function! JournalMode()
    execute 'normal gg'
    let filename = '#' . ' ' . expand('%:r')
    call setline(1, filename)
    execute 'normal o'
    execute 'Goyo'
endfunction

We get the filename minus the extension, prepend a hash to that so that it formats as a markdown header and set that to the topmost line of our buffer. We then move on to the next line and then enter focus/zen mode provided by the junegunn/goyo plugin. Let’s also add this to our journal augroup so that it does it’s job at the right time:

" workflow for daily journal
augroup journal
    autocmd!

    " populate journal template
    autocmd VimEnter */journal/**   0r ~/.config/nvim/templates/journal.skeleton

    " set header for the particular journal
    autocmd VimEnter */journal/**   :call JournalMode()

    " https://stackoverflow.com/questions/12094708/include-a-directory-recursively-for-vim-autocompletion
    autocmd VimEnter */journal/**   set complete=k/Users/danish/programming/mine/journal/**/*
augroup END

Our journal augroup now looks complete with all the configuration we talked about in the previous sections.

Appearance

This is purely subjective, you may or may not decide to read this but I decided to add this section purely for posterity.

I don’t use syntax highlighting, I’ve come to realize that It’s much easier to focus for me without the colors and text formatting involved. I made a monochrome theme just to help me with the transition from a lot of colors to none and shortly after, I disabled syntax highlighting altogether. Aside from that, here’s how my setup looks once I enter the journal command in the respective directory:

Conclusion

This is just a way to remove a little resistance from the process of daily journaling by automating some of the parts in a way that all that is required from you is just the writing part which is how it should be. But again, this varies widely from person to person but I’m sure there’s something or the other you can pickup from this post. There are bound to be errors/mistakes in the article or a way to better the process, if that’s the case, reach out!

Tokyo

I flew from New Delhi to Narita with a short layover in Hong Kong. It was a nice comfy flight on a Boeing 777-300ER, one massive and majestic piece of engineering. I landed in Tokyo shortly after noon and then took an express train to the city center from where I took the Ginza line to Akihabara. I took in the bigger than life skyscrapers with the latest and greatest anime characters plastered on each and every one of them. It was shortly before I was supposed to leave for Japan when a friend of mine told me that Akihabara is apparently the anime city.

I checked into my hostel, Hiromas, and then proceed to head out to have some quick bite at the local ⁷⁄₁₁. It was soon dark and it was then that Akihabara truly shines, quite literally. All the billboards come alive with character moving around with grace on the brightly lit mini LED bulbs. It was nothing short of a sensory overload.

The next day was the conference, VimConf 2019, I was busy the whole day. I’ve written more about the conference in a [separate post](). Got to meet lots of Vim enthusiasts with an energy I’ve personally not seen before for an editor, it was infectious.

During the conference, the locals from as far as Kansai gave me hints on how and where to visit for the next few days I was in Japan. Although I had already planned most of my itinerary, it was good to have suggestions, especially from the locals. I roamed around Tokyo the next day, doing the touristy stuff and eating around.

First stop was the famous Shinjuku Gyoen, it’s a vast expanse of greenery bang in the middle of the city. It wasn’t long after I saw something similar in Osaka that I realized how great of a job the Japanese have done preserving natural and historical lands while building a metropolitan city all around. Throngs of people on a weekday were there with their families having a good time. Old folks clicking photographs on their shiny cameras and people finding a shade to read under. I had my first real matcha with daifuku in the old tea house within the garden premises.

I then walked around the residential areas and clicked as many photographs of the clean and aesthetic alleys as I could. Eventually I reached the famous Meiji Jingu Temple. It had an eery silence to it which is quite difficult to shake off, I thoroughly enjoyed it. After this, in quick succession, I visited piss alley, Shibuya and then back to Akihabara to call it a night.

Osaka

Early int the morning, I took the Shinkansen from Tokyo to Osaka where I’d spend the next 3 nights. It is customary to buy bento from the station and have it onboard the Shinkansen, and I did the same with a shot of Suntory coffee. The train had great wifi and I spent some time working on the way. The Salaryman(サラリーマン) next to me pointed out Fuji San when it passed by, It was majestic.

I checked in to my hostel, CharinCo and then headed out to walk around the area. It was close to a residential area and was relatively quieter than usual. Japan is an extermely walkable place, almost every city there is so friendly to both pedestrians and bikers, it was honestly surprising. Sequestered away in the middle of a shopping street is the Hozenji Temple. As I mentioned, Japan does an commendable job blending the past and present and even future in some cases (Akihabara). The temple entrance was so unassuming, I passed it thrice without realizing while having Google Maps open the whole time until an elderly hunched Japanese grandpa smiled and directed me towards the place, as he must’ve had done for countless such visitors.

To end the day, I travelled far out of the city center to the famous collosus Tempozan Giant Ferris Wheel and the famous acquarium nearby. I had no intention of riding it or even going inside the acquarium but I ended up doing the latter just because I had some time and it was freezing cold outside.

Next morning, I visited the Osaka Castle, Shitennoji Temple, Tennoji, Osaka zoo. A lot of walking, I wonder how much steps I clocked that day. But it was a pleasant sunny day, perfect for exploring the city. Shitennoji also had a stillness in the atmosphere similar to Senso-Ji which I had come to really like. In Tennoji shopping district, I also saw some Geisha or Kabuki performers roaming around while tourists hogged them for photos.

Kyoto

Kyoto is close enough to Osaka for it to be a day long trip. So I took the express train from Osaka to Kyoto and reached within 1.5 hours I guess. I first went to the Fushimi Inari shrine. It’s a long long way up to the top with lots of stairs and Torii gates all the way up to the top. The hike itself is calming, albeit a little tiring because of the 12000 steps in total. Everybody was walking up to the Shrine in good spirits and it was a fun, cheery atmosphere.

Coming back from the shrine, I roamed around the residential area surrounding it and appreciate the quaint little homes with properly trimmed bonsais out in front or in the yard behind the gates. I then took the local bus to Ryōan-ji Zen temple, which, honestly was one of the most peaceful places I’ve ever been to anywhere so far. There were lots of tourists around but apparently there was no sign of any noise apart from the one that that light breeze that evening was producing. It was surreal, to sit around the rock garden and just not think about anything else, nothing short of magical.

It was almost sundown when I headed out of the temple, I took the crowded bus to the city center and by the time I reached, It was dark but the streets were glowing courtesy of the countless stores out there. It was quite nice to walk around. I also realized what a grave mistake I did by not staying in Kyoto for longer, definitely another reason for me to go back soon.

Early next day, I took the Shinkansen back to Tokyo and stayed in the Akasaka area for my last night. The next morning, I left for Haneda airport, all the while wishing to come back as soon as possible.

Observations

There were lots in this one, let me jot down the most notable ones:

• Japan is extremely civilized. Before I went to Japan, my then manager told me to enjoy the Structural Chaos in Japan and I knew this as soon as I landed at Narita. There was a huge line at the airport at the exit point. Some people running ahead while others just added onto the line making it even longer. I also did the same, trying not to commit a faux pas in a foreign land. After a while, I noticed people are deliberately leaving one side of the elevator for those who are in hurry even though it crowded the airport exit. Pareto principle, in reverse.
• It’s extremely clean. Obssessively clean. I saw people picking random trash on the streets to put in the bin, talk about civility. I’m sure this happens in other western nations too, but its not common here in India.
• Pokemon Go was all the hype. Whole alleys in Akihabara were filled with people trying to “catch’em all”. It was honestly a bit depressing and seemed like an episode of Black Mirror being shot discreetly.
• Accessible public transportation. People in wheelchairs were assisted by the subway staff and the bus staff if they were onboard, making the city more accessible to them. This is something I had never ever seen in India, I hope this is replicated everywhere else in the world.
• It’s true that there’s a sense of individualism in Japan. Whether it’s the he hole in the wall ramen joint, or a bar or a dessert shop, you’d almost always find single seating and most of the times, they’d be full. Even in the subway, there’s an odd silence with everybody gazing at their smartphones incessantly. Another episode.

Obligatory disclaimer that I was just there for around 10 days and almost everything I observed could be anecdotal so please bear that in mind.

Conclusion

I personally am someone who believes in a society where people follow rules for the sake of others. I’m not referring to being a conformist but rather being a responsible citizen and thus making the society better for the rest. After visiting Japan, I tell everyone how a society should function. Now of course, there are concepts such as the pervasive drinking culture or death by overworking (Karoshi) and many more. But ammortized benefits of the structure in Japanese society, imo, still makes it one of the best places in the world.

:wq

Talks

The theme for VimConf this year was essentially how can one be more productive with Vim. There were a lot of great talks this year:

First, mopp, MC for VimConf 2019, started with the opening giving out instructions about the conference, the layout, the history of the conference and a few words about the sponsors. Then there was a keynote session by Prabir Shreshtha, the author of [vim-lsp](). His talk, titled the Vim-Renaissance, talked about how he ended up developing the vim-lsp plugin and now, what’s the current situation with vim plugin development. There was a brief session by atWare, one of the sponsors for the conference. Justin Keyes, maintainer of Neovim was also the keynote speaker, he talked about how drastically differently Neovim does things in his talk titled We can have nice things. He pointed out problems with the heirarchy of Vim’s development “team” and how Neovim has solved it in the last few years. There were a lot of topics that he had to skip because of the lack of time but it was an interesting talk with a lot of references from Alan Kay.

After a long break for lunch, round of talks started again. In his talk titled Your Vim is Only for You, mopp talked about his vimrc, his various policies and configs he has acquired ever since he started using Vim, it was a great talk. Next, IK walked us through his rather hilarious journey from being oblivious to Vim to contributing and developing various plugins for the editor. His talk was titled Grown up from Vim User to Vim plugin developer side After this, there was an interesting talk about the tag stack in Vim by daisuzu titled Usage and manipulation of the tag stack, it started off with the basics of tags, like how to jump and go back to the previous tag in the list and so on. It then talked about the tag stack, the data structure, it’s elements and what each of them mean. This talk was rather technical and interesting.

After a short coffee break, m-nishi, talked about Vim’s tests and how he came about writing own on his own, this was something you don’t hear about pretty often, interesting. Post this, gorilla0513 talked about his Vim life in his talk, how he started using Vim at work the previous year and how, he is developing plugins and also fostering a community of it’s own and a lot more, great talk. Next up was my talk, it was titled Using Vim at Work. Keeping in mind this year’s theme, I decided to talk about how I started using Vim at work and what changes I made over time which helped me get through the obstacles in doing so. It had quite a lot of psychology behind it and how it related to someone using/learning something.

After another short break, it was the last round of talks. It started off with a really good talk by Hezby Muhammad titled Let’s Play with Vanilla Vim. Hezby talked about how he decided to use vanilla vim with a bare minimum of a vimrc and shared his pleasant experience with us all. Next, Tatsuhiro Ujihisa talked about 13 Plugins he uses every day. The talk was accompanied by a fun live-coding session. The final talk of the day was by Shougo wherein he talked about his journey of developing his dark plugins over the course of 10 years. He talked about his policies when developing a plugin among other stuff in his talk titled My dark plugins development history ~ over 10 years ~.

Lightning Talks

After this, we had lightning speakers who presented some really weird yet interesting talks which covered topics such as Customer support with Vim by note103, Can we use Slack on Vim by higashi136_2, Strong Static Typing with Vim script by aiya000 and Lose Weight with Vim and Go by micchiebear. And with this, all the talks were over and mopp did away with the closing.

Interpreters

A word out to the interpreters who did a tremendous job interpreting English to Japanese and vice versa. It sure is a difficult job especially when you don’t know prior context to what is being spoken about, kudos.

Afterparty

As is tradition, if you will, there was an afterparty with all the attendees, speakers and organizers from the conference. It was really fun to get to know some really interesting people. It had some really good food and there were some lightning talks going on in one corner of the room. A perfect way to wrap up the conference.

Thoughts

It was my second talk to be precise and although I was nervous, it went pretty well. I had earlier decided to talk about Text Objects in Vim but then after some discussion from mopp, it made more sense to talk about something which aligned more with the theme of the conference. It’s really cool to meet such enthusiastic people working towards building such a community. I learned quite a lot of new things at the conference along with meeting new people, all in all it was a great experience.

It’s worth mentioning the fact how VimConf Preparatory Committee went above and beyond to resolve every single issue that crept up with the logistics, they elegantly and promptly handled every single issue, thank you very much. Looking forward to VimConf 2020.

Links

• VimConf 2019 Videos
• VimConf 2019 Slides
• VimConf 2019 Photos

We’ve been using make as a build tool for one of our projects at HackerRank which is written in Go and it has been working out fairly well. In this post, I’ll point out a few features and intricacies of GNU Make we’ve used which eventually improved the overall productivity of members of our team.

Introduction

make is a simple utility which detects which part of a large project needs to be recompiled and executes user-defined commands to carry out compilation or other required actions. It’s also widely used as a build tool wherein you specify a set of commands to be run which you inherently used to write on the command-line, often times repeatedly. The latter is what the rest of this post is about.

For the purposes of this post, we’ll assume we’re working on a Go project, “stringifier” and will be writing a Makefile for the same which is also named Makefile.

Build and Run

These are two actions that Go programmers use quite frequently, so let’s add these targets to our Makefile:

build:
	go build -o stringifier main.go

run:
	go run -race main.go

I added the -race flag to the run command because it detects race conditions in your Go code when you run it which is an otherwise unpleasent exercise.

Cleaning and DRYing

After building the binary and running the application just fine, let’s make sure we are cleaning the binaries before proceeding with anything else. Our updated Makefile should look something like this:

build:
	go build -o stringifier main.go

run:
	go run -race main.go

clean:
	go clean

There are two things we can improve upon here, first, we are explicitly reusing our application name, it’s natural that our application name will be used in a myriad of places throughout our Makefile, we should reuse that. Second, we need to run the clean rule before we go ahead and build our application every time, let’s fix these:

APP=stringifier


build: clean
	go build -o ${APP} main.go

run:
	go run -race main.go

clean:
	go clean

This example previously used to rm -r ${APP} but thanks to suggestions from readers, it now uses go clean now.

Looks much cleaner, doesn’t it? You can define Makefile variables at the top and make will automatically expand them when you invoke the make command.

PHONY targets

By design, make executes the rule if one of the prerequisites or the target file has been changed. But since we are are not relying on the ability of make to detect file changes, we are putting ourselves in a potential pit.

Imagine that there’s a file in our project directory named build, again this is a hypothetical situation. In this case, when you run make build, make will check for changes to the file build and its prerequisites which there are none and hence won’t execute the recipe which is not what we want. We might end up using the existing binary for our use, which is misleading and a road to a lot of confusion down the road.

To avoid this problem, you can specify the target in question to be “phony” by specifying it as a prerequisite to the special target .PHONY:

APP=stringifier


.PHONY: build
build: clean
	go build -o ${APP} main.go

.PHONY: run
run:
	go run -race main.go

.PHONY: clean
clean:
	go clean

Now that you’ve specified all the above targets as phony, make will run the recipes inside of the rules every time you invoke any of the phony targets. You can also specify all the targets you want to specify as phony at once like so:

.PHONY: build clean run

But for Makefiles which grow really big, this is not suggested as it could lead to ambiguity and unreadability, hence the preferred way is to explicitly set phony target right before the rule definition.

Recursive Make targets

Let us now assume that we have another module tokenizer in our root directory that we use in our project. Our directory structure is now something like this:

~/programming/stringifier
.
├── main.go
├── Makefile
└── tokenizer/
      ├── main.go
      └── Makefile

Quite naturally, at some point, we would like to build and test our tokenizer module as well. Since it’s a separate module and a potentially separate project at some point, it makes sense for it to have a Makefile in it’s directory (cue for the post title) with the following content:

# ~/programming/stringifier/tokenizer/Makefile

APP=tokenizer

build:
	go build -o ${APP} main.go

Now, anytime you are in the root directory of your stringifier project and want to build the tokenizer application, you wouldn’t want to give in to hacky command-line tricks such as cd tokenizer && make build && cd - to invoke rules in Makefiles written in sub-directories. Thankfully, make can help you with that, you can invoke make targets in other directories using the -C flag and the special ${MAKE} variable. This is the original Makefile from the stringifier project:

# ~/programming/stringifier/Makefile

APP=stringifier


.PHONY: build
build: clean
	go build -o ${APP} main.go

.PHONY: run
run:
	go run -race main.go

.PHONY: clean
clean:
	go clean

.PHONY: build-tokenizer
build-tokenizer:
	${MAKE} -C tokenizer build

Now, anytime you run make build-tokenizer, make will handle the directory switching for you and will invoke the right target in the right directory for you in a much more readable and robust manner.

Targets for Docker commands

Now you wish to containerize your application and susequently write make targets for the same for convenience which is completely understandable.

Now, you have the following rules defined for the docker commands:

.PHONY: docker-build
docker-build: build
	docker build -t stringifier .
	docker tag stringifier stringifier:tag

.PHONY: docker-push
docker-push: docker-build
	docker push gcr.io/stringifier/stringifier-staging/stringifier:tag

Ok but now there’s room for improvement yet again, for starters, you can reuse your ${APP} variable again. Next, you need to be rather flexible and make sure you can easily control where you push your image, whether that’s your private registry or some place else. Then, you would like to be able to push your image to two separate registries pertaining to staging and production environments respectively based on some input on the command-line from the user. Finally, like a sane developer, you would like to tag your images, with the current git commit sha, in your case. Let’s fix things up:

APP?=application
REGISTRY?=gcr.io/images
COMMIT_SHA=$(shell git rev-parse --short HEAD)

.PHONY: docker-build
docker-build: build
	docker build -t ${APP} .
	docker tag ${APP} ${APP}:${COMMIT_SHA}

.PHONY: docker-push
docker-push: check-environment docker-build
	docker push ${REGISTRY}/${ENV}/${APP}:${COMMIT_SHA}

check-environment:
ifndef ENV
    $(error ENV not set, allowed values - `staging` or `production`)
endif

Okay now, let’s go over the changes above:

• You started using variables for the application name, the image registry and for the commit sha.
• You generated the commit sha using the special shell function. In this case, you ran the git command which returned the short commit sha and assigned it to the variable ${COMMIT_SHA} to be used later on in your Makefile.
• You added a new rule check-environment which uses the make conditionals to check whether the ENV variable is specified or not while invoking make. This helps removing the ambiguity to which repo, out of staging and environment, to push the the docker image of your application.

Expanding on the check-environment rule here:

check-environment:
ifndef ENV
    $(error ENV not set, allowed values - `staging` or `production`)
endif

You are using the ifndef directive which checks whether the variable ENV has an empty value or not, and if it does, then you use another built-in function that make provides, error which, as it sounds, throws an error with the error message following the keyword.

$ make docker-push
Makefile:33: *** ENV not set, allowed values - `staging` or `production`.  Stop.

$ ENV=staging make docker-push
Success

Essentially, you are making sure that the docker-push target has a safety net which checks that the user who invoked the target has specified a value for the ENV variable.

Help target

A new member has joined the project and is wondering what all the rules do in the Makefile, to help them out, you can add a new target which will print all the target names along with a short description of what they do:

.PHONY: build
## build: build the application
build: clean
    @echo "Building..."
    @go build -o ${APP} main.go

.PHONY: run
## run: runs go run main.go
run:
	go run -race main.go

.PHONY: clean
## clean: cleans the binary
clean:
    @echo "Cleaning"
    @go clean

.PHONY: setup
## setup: setup go modules
setup:
	@go mod init \
		&& go mod tidy \
		&& go mod vendor
	
.PHONY: help
## help: prints this help message
help:
	@echo "Usage: \n"
	@sed -n 's/^##//p' ${MAKEFILE_LIST} | column -t -s ':' |  sed -e 's/^/ /'

Focus on the last rule, help. Here, you are simply using some sed magic to parse and print on the command line. But to do that, you already wrote the target name and a short description before every rule as comments. Notice another special variable, ${MAKEFILE_LIST} which is a list of all the Makefiles you have referred to, only Makefile in our case.

You are passing the file Makefile as input to the sed command which is parsing all the help comments and printing them to the stdout in a tabular format so that’s it’s easier to read. Output for the help target for the previous snippet would look like the following:

$ make help
Usage:
	build             Build the application
	clean             cleans the binary
	run               runs go run main.go
	docker-build      builds docker image
	docker-push       pushes the docker image
	setup             set up modules
	help              prints this help message

Well, that looks quite helpful. It will most certainly come in handy for a lot of people and even for you at times.

Conclusion

Make is a simple yet a highly configurable tool. In this post, you ran through a host of configurations and features offerred by make to write an effective and productive Makefile for your Go application.

Here’s the complete Makefile after adding a few trivial rules and variables for completeness’s sake:

GO111MODULES=on
APP?=stringifier
REGISTRY?=gcr.io/images
COMMIT_SHA=$(shell git rev-parse --short HEAD)



.PHONY: build
## build: build the application
build: clean
    @echo "Building..."
    @go build -o ${APP} main.go

.PHONY: run
## run: runs go run main.go
run:
	go run -race main.go

.PHONY: clean
## clean: cleans the binary
clean:
    @echo "Cleaning"
    @go clean

.PHONY: test
## test: runs go test with default values
test:
	go test -v -count=1 -race ./...


.PHONY: build-tokenizer
## build-tokenizer: build the tokenizer application
build-tokenizer:
	${MAKE} -c tokenizer build

.PHONY: setup
## setup: setup go modules
setup:
	@go mod init \
		&& go mod tidy \
		&& go mod vendor
	
# helper rule for deployment
check-environment:
ifndef ENV
    $(error ENV not set, allowed values - `staging` or `production`)
endif

.PHONY: docker-build
## docker-build: builds the stringifier docker image to registry
docker-build: build
	docker build -t ${APP}:${COMMIT_SHA} .

.PHONY: docker-push
## docker-push: pushes the stringifier docker image to registry
docker-push: check-environment docker-build
	docker push ${REGISTRY}/${ENV}/${APP}:${COMMIT_SHA}

.PHONY: help
## help: Prints this help message
help:
	@echo "Usage: \n"
	@sed -n 's/^##//p' ${MAKEFILE_LIST} | column -t -s ':' |  sed -e 's/^/ /'

If you found any issues/mistakes or have any suggestions or additions related to this post, please feel free to reach out to me.

:wq

Vim macros

Macros in Vim are defined as recordings of commands which you can save to a particular register and replay later on. You can also think of them as functions in a way because they offer you to avoid doing repeatable tasks. Coming back to the list of states, it looked something like this:

Andhra Pradesh
	

Hyderabad, Amaravati

Arunachal Pradesh
	

Itangar

Assam
	

Dispur

Bihar
	

Patna

Chhattisgarh
	

Raipur

I’ve snipped the whole output of this for brevity but you get the idea. Essentially, what we have here is the name of the city, followed by two newlines and then the name of it’s capital, repeated for all of the 27 states. Nevermind the mess, let’s get to it.

You alread know that a macro is a sequence of commands which you can execute again and again by specifying the register to which the command is spaced. So, all I need to do is come up with a sequence and then repeat it n times.

Start recording the macro by hitting the q. For the first state, we can build a key value pair using what we have using the following sequence of commands, assuming you are on the first line of the file:

• qq: start recording the macro into register q
• 0: move cursor to first char
• 4J: join the 3 lines below the current one
• hi:Esc: move cursor one position left and insert colon
• I"Esc: insert quote before the first char on line
• t:a"Esc: insert quote before the colon char
• wi"Esc: insert quote on the beg of the next word
• A",Esc: insert quote and a comma to the end of line
• jdd: delete the next empty line and put the cursor on the next non-empty line

That might seem like a lot but it really is just a sequence of easy to understand Vim commands. You can hit q again to stop recording the macro. When you are done recording, you would have ended up on a new line, a new state. You can hit @q to replay the commands stored in the q register. It will instantly repeat the commands we just tried out. @@ is a shortcut to run previously run macro.

Recursive Vim macros

Now, you don’t want to go about hitting @q or @@ 26 times. You can make use of recursive Vim macros to get away with repetitive tasks which involve macros. We will make a small addition to the list of commands that we ran previously. All we need to do is to repeat the macro when we are on a new line and as already discussed previously, we can use @<register> to replay the macro stored in . So, our new set of commands would be

• qq: start recording the macro into register q
• 0: move cursor to first char
• 4J: join the 3 lines below the current one
• hi:Esc: move cursor one position left and insert colon
• I"Esc: insert quote before the first char on line
• t:a"Esc: insert quote before the colon char
• wi"Esc: insert quote on the beg of the next word
• A",Esc: insert quote and a comma to the end of line
• jdd: delete the next empty line and put the cursor on the next non-empty line
• @q: To replay this macro at the beginning of a new line assuming the fact that we have used q to save our macro.

Once you enter this somewhat arcane list of normal commands while being on the first non-empty line of the buffer, you will see that Vim “magically” arranges everything for you and now you have this almost JSON-like file which you can read using Python and whatnot. I said almost JSON-like because it is missing the opening and closing parenthesis at the end and beginning of the file which you can just go ahead and put in manually. Your buffer should now look something like this:

{
    "Andra Pradesh": "Hyderabad, Amaravati",
    "Arunachal Pradesh": "Itangar",
    "Assam": "Dispur",
    "Bihar": "Patna",
    "Chhattisgarh": "Raipur",
    "Goa": "Panaji",
    "Gujarat": "Gandhinagar",
    "Haryana": "Chandigarh",
    "Himachal Pradesh": "Shimla",
    "Jammu and Kashmir": "Srinagar and Jammu",
    "Jharkhand": "Ranchi",
    "Karnataka": "Bangalore",
    "Kerala": "Thiruvananthapuram",
    "Madya Pradesh": "Bhopal",
    "Maharashtra": "Mumbai",
    "Manipur": "Imphal",
    "Meghalaya": "Shillong",
    "Mizoram": "Aizawi",
    "Nagaland": "Kohima",
    "Orissa": "Bhubaneshwar",
    "Punjab": "Chandigarh",
    "Rajasthan": "Jaipur",
    "Sikkim": "Gangtok",
    "Tamil Nadu": "Chennai",
    "Telagana": "Hyderabad",
    "Tripura": "Agartala",
    "Uttaranchal": "Dehradun",
    "Uttar Pradesh": "Lucknow",
    "West Bengal": "Kolkata",
}

Other uses

This might have been trivial for some of you reading but if you think about the underlying idea this article proposes, you can put Vim macros to a variety of uses. I’ve used Vim macros often times, in fact whenever I see myself repeating normal mode commands, I try to do the same using recursive macros. Some of my recent uses involving Vim macros are:

• Transform all the keys in a Go map from uppercase to lowercase
• Remove multiple values from a Python dict and adding a new one
• Add a newline to lines in a file which matches a certain pattern

It’s fun to solve problems in a unique way, albeit somewhat time consuming initially. They tend to pay off at the end in a sense that you get to learn a skill which comes in handy for times to come. Vim macros are a powerful way to level up your editing game. Learn more about macros in Vim using :h macro.

:wq

We’ll write a simple shell that that will support almost all the basic commands. We’ll also implement piping for our shell which will allow us to pipe the output of a command as input to another command, more on that later.

Program flow

We’ll start off with our main function where we will handle the program flow. First, let’s get the user input which is quite trivial, we’ll just run an infinite loop and prompt the user for input.

def main():
    while True:
        command = input("$ ")
        if command == "exit":
            break
        elif command == "help":
            print("psh: a simple shell written in Python")
        else:
            execute_commands(command)

We’re handling the exit command simply by breaking out of the loop. The next most important task is to execute the command entered by the user which we’ll manage in a separate function, let’s call that execute_commands(command). We’ve also added a simple help function to let the user know what’s actually happening.

Executing commands

Let’s execute the commands entered by the user. We’re using the subprocess builtin module here, so import subprocess and we’re good to go. The run function in particular is used here to execute commands in a subshell. For those coming from C, this saves us from going about forking and creating a child process and then waiting for the child to finish execution, let Python take care of that this one time.

def execute_commands(command):
    try:
        subprocess.run(command.split())
    except Exception:
        print("psh: command not found: {}".format(command))

We’re making sure that our shell doesn’t come crashing down if the user enters cs instead of cd by mistake, hence the try/except mechanism. Just a heads up, there are many other ways to execute system commands from within Python including os.system and commands etc but using subprocess is the preferred way of doing it.

Changing the directory

While all our commands would work using the subprocess module, the cd command would not work this way. This is because subprocess runs the command in a subshell and when you try to change the directory, it actually changes the directory but does so in the subshell instead of in the original process and hence we get the impression that the command didn’t work. We’ll handle this separately in a different function and add a conditional in our main function.

def psh_cd(path):
    """convert to absolute path and change directory"""
    try:
        os.chdir(os.path.abspath(path))
    except Exception:
        print("cd: no such file or directory: {}".format(path))

Here, we’re using os.chdir to change the directory and we also make sure to convert the path entered by the user to an absolute path before passing it to os.chdir. Note that we’ll have to edit our main function to add this condition.

Pipes!

Let’s get to the fun part. Pipes allow us to transfer output of one process as input to another and so on, in a chained manner. Consider this image[1] which shows how pipes are used and what they do.

You can think of a pipe | as a pair of file descriptors. For instance, If we create a pipe, we’ll get two file descriptors reserved for our usage, for e.g. f1 and f2 wherein we can write data to f2 and read the same from f1. Python allows us to create pipes using os.pipe which returns a tuple containing the integer value which refer to the file descriptors. Consider the implementation of our execute_commands function with piping below:

def execute_command(command):
    """execute commands and handle piping"""
    try:
        if "|" in command:
            # save for restoring later on
            s_in, s_out = (0, 0)
            s_in = os.dup(0)
            s_out = os.dup(1)

            # first command takes commandut from stdin
            fdin = os.dup(s_in)

            # iterate over all the commands that are piped
            for cmd in command.split("|"):
                # fdin will be stdin if it's the first iteration
                # and the readable end of the pipe if not.
                os.dup2(fdin, 0)
                os.close(fdin)

                # restore stdout if this is the last command
                if cmd == command.split("|")[-1]:
                    fdout = os.dup(s_out)
                else:
                    fdin, fdout = os.pipe()

                # redirect stdout to pipe
                os.dup2(fdout, 1)
                os.close(fdout)

                try:
                    subprocess.run(cmd.strip().split())
                except Exception:
                    print("psh: command not found: {}".format(cmd.strip()))

            # restore stdout and stdin
            os.dup2(s_in, 0)
            os.dup2(s_out, 1)
            os.close(s_in)
            os.close(s_out)
        else:
            subprocess.run(command.split(" "))
    except Exception:
        print("psh: command not found: {}".format(command))

Here, we have created a pipe with the values fdin and fdout. We’re manipulating our file descriptors using the pipe we’ve created, so the input and output of every sub-command(each of the piped commands) executed during the loop will depend on the value of fdin and fdout.

Don’t fret if that seemed confusing, let’s see what’s happening in the above snippet of code line-by-line.

• Lines 6-7: We’re creating temporary variables, s_in, s_out to hold the original values of stdout and stdin to restore them later on.
• Line 11: Create a duplicate of stdin and set fdin to it so that the first sub-command recieves input from stdin when we later redirect the standard input to fdin.
• Line 14: The loop iterates over the sub-commands.
• Line 17-18: Redirecting stdin to fdin. There can be two cases here. First, if the sub-command is the first in series, then fdin would be pointing to stdin . Second, if the sub-command is not the first, in which case, the value of fdin would be storing the value of the readable end of the pipe we have created in the previous iteration of the loop.
• Lines 21-28: Redirecting stdout to fdout. There are two possibilites here as well. First, if the sub-command is the last in series, in that case, fdout would be redirected to s_out, which is the original stdout we stored. Second, if the sub-command is not the last in series, in which case, we would redirect fdout to the writable end of of the pipe we have created in this very iteration of the loop. Note that in line 24, we are creating the pipe.
• Lines 30-33: We’re executing the sub-command here. It will read input from fdin and will write it’s output to fdout theoretically since we redirected both stdin and stdout accordingly.
• Lines 36-39: We’re restoring the values of stdin and stdout to their original values that we had stored earlier.
• Lines 40: Execute the command normally if no pipe operators are present.

Putting it together

We have all the pieces figured out now. Let’s put them together to get this shell working. I’ve made some additions here and there which are too trivial to explain before putting it out here. You can also see the sample output below the code.

#!/usr/bin/env python3

"""psh: a simple shell written in Python"""

import os
import subprocess


def execute_command(command):
    """execute commands and handle piping"""
    try:
        if "|" in command:
            # save for restoring later on
            s_in, s_out = (0, 0)
            s_in = os.dup(0)
            s_out = os.dup(1)

            # first command takes commandut from stdin
            fdin = os.dup(s_in)

            # iterate over all the commands that are piped
            for cmd in command.split("|"):
                # fdin will be stdin if it's the first iteration
                # and the readable end of the pipe if not.
                os.dup2(fdin, 0)
                os.close(fdin)

                # restore stdout if this is the last command
                if cmd == command.split("|")[-1]:
                    fdout = os.dup(s_out)
                else:
                    fdin, fdout = os.pipe()

                # redirect stdout to pipe
                os.dup2(fdout, 1)
                os.close(fdout)

                try:
                    subprocess.run(cmd.strip().split())
                except Exception:
                    print("psh: command not found: {}".format(cmd.strip()))

            # restore stdout and stdin
            os.dup2(s_in, 0)
            os.dup2(s_out, 1)
            os.close(s_in)
            os.close(s_out)
        else:
            subprocess.run(command.split(" "))
    except Exception:
        print("psh: command not found: {}".format(command))


def psh_cd(path):
    """convert to absolute path and change directory"""
    try:
        os.chdir(os.path.abspath(path))
    except Exception:
        print("cd: no such file or directory: {}".format(path))


def psh_help():
    print("""psh: shell implementation in Python.
          Supports all basic shell commands.""")


def main():
    while True:
        inp = input("$ ")
        if inp == "exit":
            break
        elif inp[:3] == "cd ":
            psh_cd(inp[3:])
        elif inp == "help":
            psh_help()
        else:
            execute_command(inp)


if '__main__' == __name__:
    main()

# running the shell
$ python3 psh.py

# prompt from our shell
$ pwd
/home/psh

# pipe multiple commands 
$ cat psh | wc -l
      81

Conclusion

So we’ve written a functional shell in Python in about ~80 lines of code, that’s not too bad considering we’ve got piping sorted out here. There are a lot of features missing from this shell but the intent behind this was not turning this into a daily driver but to rather see the implementation of a shell in Python. In the process, we learned how to execute system commands from within Python the right way, we also learned how to manipulate file descriptors in Python to redirect input/output of a command as per our needs.

For further practice, you can try implementing the following features in this shell

• Implement the history command, see this for reference.
• Add globbing support.
• Add comment support at the prompt.Comments are disregarded by the shell so this should be fairly trivial.

If you find any issues/mistakes in this post, contact me or open an issue on this project’s repository.

:wq

I’ve been writing Python for a while now and I was working on this big open source project recently, trying to fix a trivial issue which ended up failing the builds. When I inquired about it, it turns out that the builds failed because I forgot to make the strings in my changes translatable. So, I set about looking for answers and there it was, Localization. In this post, we’ll walk our way towards providing translations to a simple python program.

GNU gettext

There are other ways with which you can provide localization for your applications. But we’ll use the gettext module for the purpose of this post. It provides internationalization and localization services for your applications and comes bundled with the standard python installation. It exposes two different API’s for you to work with, a more standard gettext API which affects your entire application’s translations and a class-based API which is more suitable for Python modules and applications. We’ll make use of the latter.

Hello World!

We’ll use this legendary hello world program in this post with a simple addition.

def main():
    print("Hello World!")
    print("Localization is fun!")

if '__main__' == __name__:
    main()

Translations

In order to provide the translations which could be read by the gettext module, we need to create a separate directory named locales, this can be named anything as long as it’s intuitive. For the sake of keeping this post brief, we’ll be providing a German(Deutsch) translation for our Hello World program. For the same, our locales directory would look something like this.

locales/
├── de
│   └── LC_MESSAGES
└── en
    └── LC_MESSAGES

Where de and en are language codes for german and english respectively. You can find a list of all the languages and their respective language codes here.

Now we need to provide translations for each of the strings in our program in German. We can do this by marking all the strings in our program that we need translated and for which we have provided proper translations. The standard accepted way to do this is to surround your strings with _(). Don’t worry if this seems arcane, refer to the example below.

import gettext

_ = gettext.gettext

def main():
    print(_("Hello World!"))
    print(_("Localization is fun!"))

if '__main__' == __name__:
    main()

Since functions in Python are first-class objects, we can move them or pass them around or even assign them to variables. This is what we’re doing here. We’re assigning the gettext function of the gettext module to _ which we call later in our programs while passing our strings as arguments. You can think of this as writing print(gettext.gettext("Hello World")). Nifty, right?

Now that we’ve marked the strings we will provide translations for, let’s provide actual translations for them.

pygettext

We’ll create a template with all of the strings which we’ve marked in our program so that it’s easier for us to write the translations for the strings. This template is a POT with a .pot extension which stands for portable object template. To generate the template for our program, we can use the pygettext.py module which also comes bundled with the standard installation of python.

$ $(locate pygettext) -o locales/template.pot hello_world.py

This will generate our template file template.pot in the locales directory. If you view the contents of this file, you’ll see metadata about the file and at the end, we can see our two strings which we had wrapped using the gettext function.

# SOME DESCRIPTIVE TITLE.
# Copyright (C) YEAR ORGANIZATION
# FIRST AUTHOR <EMAIL@ADDRESS>, YEAR.
#
msgid ""
msgstr ""
"Project-Id-Version: PACKAGE VERSION\n"
"POT-Creation-Date: 2018-09-09 09:23+0530\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Generated-By: pygettext.py 1.5\n"


#: hello_locales:13
msgid "Hello World"
msgstr ""

#: hello_locales:14
msgid "Localization if fun!"
msgstr ""

Here, msgid denotes the original string and msgstr contains the translated string. Note that it also provides some info about the string, namely the filename and the line number. Next, we can just copy this file to locales/de/LC_MESSAGES and provide the appropriate translations. It should look like this minus the metadata.

#: hello_locales:13
msgid "Hello World"
msgstr "Halo Welt"

#: hello_locales:14
msgid "Localization if fun!"
msgstr "Lokalisierung macht Spaß"

For the english translation (en/), we can make do by simply copying template.pot to locales/en/LC_MESSAGES. We can think of this template as a global template we can use for every locale or language we wish to provide a translation for, all that’s required is to copy this template to the proper directory as shown in the directory convention above.

msgfmt

There’s one more step to this before we fire up das program. gettext module cannot directly use the .po files and hence we are required to convert these files to their equivalent .mo files. These .mo files are binary machine-object files that are parsed by gettext. We can use the msgfmt tool to generate these, which also comes with the standard python installation.

$ $(locate msgfmt) locales/de/LC_MESSAGES/template.po
$ $(locate msgfmt) locales/en/LC_MESSAGES/template.po

This would create an equivalent .mo file to be used by the gettext module. At this point of time, our locales directory should look like this.

locales
└── en
    └── LC_MESSAGES
        ├── template.mo
        └── template.po
└── de
    └── LC_MESSAGES
        ├── template.mo
        └── template.po

Putting it together

Let’s modify our program to use the appropriate translations or more aptly, to use the .mo files we have generated.

import gettext
import os

LOCALE = os.getenv('LANG', 'en')

_ = gettext.translation(
        'template', localedir='locales', languages=[LOCALE]).gettext


def main():
    print(_("Hello World"))
    print(_("Localization if fun!"))


if '__main__' == __name__:
    main()

Here we invoke the gettext.translation function which returns a Translation instance on which we call the gettext function and assign it to _. We pass the name of our template file as a string, also called the domain. Then we specify the directory which has all our translations for different locales, locales. Next, we provide a list of lanugages(language codes) that we wish to be parsed by gettext. Finally, we have used an environment variable to easily switch locales from the command line.

We can now test our program.

$ python hello_world.py
Hello World
Localization is fun!

$ LANG=de python hello_world.py
Halo Welt
Lokalisierung macht Spaß

It works as expected for both english and german locales, look at how we’ve used the environment variable to switch to German(de) locale.

Conclusion

In this post, we’ve seen how to localize our application. We learned how to use the gettext module and a bit about portable objects and portable object templates. We also used environment variables to switch locales of our application.

If there’s something you’d like to improve in this article or if you’ve found something that’s not correctly stated, feel free to contact me.

This happened to a friend of mine recently while he was working on a big project, after we started looking for solutions, there were times when it seemed like the changes would have to be done again manually for every file. But git had us covered in a sense that git never loses anything, ofcourse unless you do git gc, and you can almost always recover your lost stuff.

Detached HEAD

When you checkout a previous commit on a branch, say master, you are not on any branch, instead you are in a state called detached HEAD. This is somewhat confusing but it actually makes sense. Think of it this way, going back into history and changing something there may result in a conflict.

Dangling commits

The commits you’ve made while in a detached HEAD state are called dangling commits because they don’t belong to a branch. After you’ve checked out a branch and are no longer in a detached HEAD state, you may think you’ve lost the dangling commit but it is not quite right. Dangling commits can be located in many ways, two of which are:

1. git fsck --lost-found: It identifies all the dangling commits and writes them to the directory .git/lost-found/commit and to stdout. It may not be the best way to identify dangling commits since it just shows the commit hash and not the commit message.

2. git reflog: It keeps track of all the refs, which are references to commits, in your local directory. Hence the name reflog. It is highly unlikely that you won’t find your lost commit here. You can sift through the output to find your lost commit and then copy it’s hash.

Applying commit

After copying the commit hash obtained from git reflog, we can simply do git merge <hash> to apply the changes to our current branch. But make sure you are on the master branch before doing this otherwise we’re back at square one.

Conclusion

We can easily recover a lost commit using these steps. Although there might be other cases where this may not be applicable.

• Use git reflog to find the lost commit.
• Copy hash of the lost commit.
• Do git merge <commit_hash>

$ git reflog
# copy lost commit's hash

$ git merge <hash>

You can repeat this process for a number of commits that you’ve “lost” but make sure you are on the correct branch before actually going ahead and merging the commits.

I came up with a simple trick which has worked good enough for me for quite some time now, it’s intuitive and configurable.

Function

This is a simple bash function with find and fzf. Load this up in your .zshrc. Follow along to get a grip on how this actually works.

function quick_find () {
    dir=$(find ~/programming -type d -not -path '*/\.*' -maxdepth 1 | fzf)
    cd $dir
    zle reset-prompt
}

Let’s go over each and every element in the function defined above.

1 find: find helps you find files and directories on your system. Here, we’re doing a search for all the directories (-type d) in the programming directory (this is just me, you can add more directories in which you want to lookup). Next, we’re making sure that hidden directories don’t show up (-not-path '*/\.*). And finally we make it a rule that the depth to which find should look for directories is just 1 level deep, that means it won’t go into a subdirectory, say subdir which resides ~/programming/dir/subdir, which is actually 2 levels deep.

2 fzf: You probably already know about fzf but if you don’t, it’s a fuzzy file finder and it is really fast. We pass the results from find which is nothing but a list of directories to fzf which opens up a nice little interface where we can get suggestions as soon as we start typing, refer to the gif below.

3 cd $dir: Finally, cd into the selected directory. A note here, you need to hit the control return (enter) key for the cd to actually take effect, I’ve been looking into the why of this but couldn’t find a plausible explanation. Hit me up if you have something of value in this context.

Update: Thanks to /u/maji_yabakune for helping with a solution to cding into the directory without hitting the return key, explained below.

4 zle reset-prompt: This redraws the prompt to take into immediate effect the new working directory so you don’t need to hit the enter key anymore.

Creating a shortcut

Now, we would like to have a handy key combination to open our switcher, how about Ctrl-p? We would want our function to fire up as soon as we type Ctrl-p, so a simple bindkey should do, right? Well, no, zsh doesn’t allow you to bind keys to a function, instead we would create a widget which maps to the function and finally bind that widget to the key combination. We can do that by creating a widget using the zsh line editor, zle and then we can specify our key combination mapped to this widget we just created.

zle -N quick_find_widget quick_find
bindkey "^p" quick_find_widget

That’s about it, simply put the function and these two lines in your .zshrc and you’re good to go.

–

Conclusion

If all goes well, you should have something like the gif below making it easier for you to switch between directories easily.

Also, if you can think of improvements to this, please open an issue and/or a PR on this repository, there is more stuff there. Or if you have other such handy tricks, do let me know. Feel free to contact me via email.

There are essentialy two steps in this process:

• Writing a formula for your application.
• Creating a tap to host your formula file.

Don’t worry about the terminology just yet, we’ll get to know them as and when required.

Your application’s tar archive

Homebrew accepts a tar archive of your application in order to process it. We can generate one by simple creating a new release on your project’s github repository. Once you’ve successfully created a release, click on the releases tab and find the link to the tar archive below the release label. My application’s release(s) for example.

Write your formula

Homebrew formulae are essentially ruby scripts which stores information about your application. It tells homebrew important information regarding the installation of your application like where and how your app is going to be installed. This is what a simple formula for an application written in bash would look like.

class Goodreadsh < Formula
	desc "Command-line interface for Goodreads"
	homepage "https://github.com/danishprakash/goodreadsh"
	url "https://github.com/danishprakash/goodreadsh/archive/1.0.1.tar.gz"
	sha256 "63d1de17449611f1490b1930e63cc8890f9f10c7e317f02c901e6a79236c10e2"
	head "https://github.com/danishprakash/goodreadsh.git"

	def install
		bin.install "goodreads"
	end

	test do
		system "#{bin}/goodreads"
	end
end

Now, almost all of the fields here are pretty much self-explanatory but we’ll go through them quickly.

• desc: small description of your application, make sure you don’t end it with a period (.)
• homepage: a webpage for your project, if you don’t have one, the project’s github repo will do.
• url: this is the url of your application’s tar archive. Use the project’s github repo if you don’t have one.
• sha256: this is the sha value of your application’s tar for authentication. You can calculate the sha value of your application using the command openssl dgst -sha256 <(curl -s -L <URL>) where URL is the link to the tar archive to your application.
• head: a link to your github repo since Homebrew can understand several vcs, this option is sometimes used to install your application using the --HEAD option for development purposes.

You can use this template or you can generate a formula of your own. Homebrew provides a simple command for this. brew create <URL>

$ brew create https://github.com/danishprakash/goodreadsh/archive/1.0.1.tar.gz

The URL in question is the link to the tar archive of your application. I’m using the one created automatically by Github when you create a new release and/or tag. As soon as you execute this command, an editor will open up with the formula.

You can check whether your formula is working or not by installing your application from specifying the formula.

$ brew install --build-from-source <formula.rb>

If it works, good enough. Now make sure your formula abides by the guidelines defined by homebrew using this command. Rectify any errors or warnings if there.

$ brew audit --strict <formula.rb>

Install & Test

Since this is a rather trivial application that we’re writing a formula for, the formula for this would also be a simple one. You can have multiple things defined in your formula depending upon what you expect it to do.

Moving ahead, we can see a function definition install in which there’s a single expression bin.install "goodreads". This will move the file in double quotes into the Formula’s bin directory and make it executable (chmod 0555 goodreads).

Similarly in the second function definition test, we execute the installed executable with the system command in ruby which is used to execute commands from within ruby scripts. The system function takes a single quoted argument which is the absolute path to the executable.

Putting it together

Before moving one, there’s one thing I’d like to make clear. There are more than one way to allow users to allow installing your application via homebrew. First, having your application formula in the homebrew-core repository and the second is creating a tap. Having your application formula accepted in homebrew-core isn’t very straightforward since there are various rules that determine whether a package should be in the homebrew-core repo, for instance, your application’s Github repo should have more than 30 watchers, 75 stars and 30 watchers and should be more than 30 days old.

Taps on the other hand are your own personal formula repository where formula(e) for your applications can be stored. Homebrew can look up formula(e) from this repo and allow you to install the said application. In this tutorial, we’ll be creating a tap. Now, let’s create a repository where we’ll store our homebrew formulae, let’s call it homebrew-formulae and add our formula file here.

We can now install the application via homebrew. But before doing that, we need to tell homebrew where to find our application since it is not there in the homebrew-core repository. We can do this by the tap command. And installing our application afterwards.

$ brew tap danishprakash/homebrew-formulae
$ brew install goodreadsh

Another upside to having a designated formula repo (tap) of your own is the simple fact that you can simply add another formula in the future and have it installed in the same way. So this acts like a collection (repo?) of all your application’s formulae making it easier to maintain and update.

–

Conclusion

Hopefully you can now write a homebrew formula for a trivial application and distribute it via homebrew by creating a tap. There are lots of options for you to consider when you write a formula for your application in the Homebrew Formula Cookbook. I’d suggest taking a good look at the formula cookbook before attempting to write your own formula.

When I say vanilla zsh, I meant to say we are not going to use any of the fancy frameworks available for zsh namely oh-my-zsh or prezto. I personally avoid using such frameworks that add unnecessary overhead to your tool wherein the functionality you’re seeking can easily be added via simple configurations. One thing to note here is that we are just talking about having our current working branch displayed in our zsh prompt, if you are looking for a more sophisticated git integration, take a look at prezto, oh-my-zsh or read about the vcs_info package offered by zsh here.

Note: If you are just looking for the solution, jump to Conclusion directly and read on if you don’t mind bits of interesting info thrown around.

Git branch

Before I started looking for solutions to this particular problem, I only knew one way to echo the current branch in a directory which is the good old git branch command. But turns out there’s one more and possibly others.

$ git symbolic-ref HEAD

From the official documentation, git symbolic-ref <name> reads which branch head the given symbolic reference refers to and returns the path to it relative to the .git directory. In other words, when you give HEAD as an argument to git symbolic-ref, we could figure out the name of our current branch.

$ git symbolic-ref HEAD
refs/heads/master

I’m using the latter version since it will give us a rather brief and static output than git branch which would list all the branches there are in your current working tree.

Suppressing errors

In directories which are not git repositories, our command will throw an error. To supress such errors, we could simply redirect the error. Our updated command:

$ git symbolic-ref HEAD
fatal: Not a git repository (or any of the parent directories): .git

$ git symbolic-ref HEAD 2> /dev/null
$ 

The 2> /dev/null redirects the stderr or standard error to a special file /dev/null which takes input but doesn’t really do anything with it. So we’re sorted here. Read more about standard streams here.

Comes in awk

You certainly don’t need ref/heads/master in your prompt, let’s filter the unwanted stuff out. We’ll use a simple awk script (command really) to do the job.

$ git symbolic-ref HEAD 2> /dev/null | awk 'BEGIN{FS="/"} {print $NF}'
master

In the awk command above, first of all we specify the field separator (FS="/") since our git command returns a path which is separated by a forward slash and finally we print the last column in the separated list. $NF here means the number of fields and subsequently points to the last column. Now that we have our output sanitized, let’s get this onto our prompt.

Updating prompt

Now that we have all the inputs ready, let’s put these together to get it working. Add this function in your .zshrc, preferably on the top so that it becomes easier for you to re-use it.

function git_branch() {
    branch=$(git symbolic-ref HEAD 2> /dev/null | awk 'BEGIN{FS="/"} {print $NF}')
    if [[ $branch == "" ]]; then
        :
    else
        echo ' (' $branch ') '
    fi
}

Here, we initialize a branch variable with the value of the output of the command we put together in the steps above. If you are wondering why we have the command enclosed inside $(), this is called command substitution wherein a command is replaced by it’s output. So we’ll be left with the actual branch name in the variable branch. The conditional logic after that is used to handle cases where the current directory isn’t really a git repository and in which case we want the prompt untouched.

setopt prompt_subst
PROMPT='%~ $(git_branch) >'

Here, we set the prompt_subst option in our zsh, this allows command substitution to be able to be performed in prompts. In other words, when we defined the function git_branch above, we actually defined a custom command named git_branch which is used in our shell prompt since functions in shell scripts are essentially commands that can be executed from the shell. Now the shell performs command substitution everytime the prompt appears on the terminal. After this we set the prompt using the PROMPT shell variable. The %~ will be expanded to the current working directory. Read more about prompt expansions in zsh here.

Finally, source your .zshrc file to see the changes take effect. Your prompt should look like this assuming you are inside a directory which is a git repository and is on the master branch. This is how it looks on my machine.

–

Conclusion

So, having the current branch displayed on your zsh prompt was all in all a simple git command with a pint of awk in it and an even simpler shell script. For those restless souls who landed here in search of the solution, here it is.

# function to return current branch name while suppressing errors.
function git_branch() {
    branch=$(git symbolic-ref HEAD 2> /dev/null | awk 'BEGIN{FS="/"} {print $NF}')
    if [[ $branch == "" ]]; then
        :
    else
        echo ' (' $branch ') '
    fi
}


setopt prompt_subst             # allow command substitution inside the prompt
PROMPT='%~ $(git_branch) >'     # set the prompt value

I’m an advocate of using vanilla vim or neovim. But there are times wherein you feel the need to install plugins, after working on vim for the past couple of months religiously, I’ve come across certain plugins that I feel help me in my dev workflow. Although there are some plugins that I’ve mentioned in this article whose functionality can easily be added by configuring my .vimrc but I think if a plugin helps you save time that’d be rather spent configuring the setup, you’re better off using the plugin.

This post should not be seen as one of those “Must have plugins for vim” but rather as a post describing the plugins I use and why I use them, and how they help me with my developer workflow. Using or not using these plugins is something for you to decide.

Note: I’m using vim-zen as my plugin manager, It’s a lightweight barebones vim plugin manager with multithreading (python). You can learn more about it here.

1. junegunn/goyo.vim
   It helps transform vim into a minmalistic looking note taking application. I mostly use it while write markdown for my blog. In essence, this plugins disables numbers, the statusline and other elements from the editing window leaving you with just the text.

2. airblade/vim-gitgutter
   I use it for just one simple reason, It shows me the lines where unstaged changes are present and the type of change that has been done (+, -, ~). This comes in very handy when I’m about to push some changes and I can quickly jump through parts of code where changes have been made.

3. tpope/vim-surround
   This is useful where you need to wrap a word or a text block inside quotes or tags. For instance, wrapping a long string of text inside the parens of a method call wherein you forgot to put quotes around the actual string. The command ysi(' would do the job.

4. townk/vim-autoclose
   Simply completes characters which work in pairs, for instance, as soon as I type in a ( vim will automatically add ) and put the cursor in between.
   ( --> (|)

5. tpope/vim-commentary
   Works in a similar way as vim-surround but instead, as the name suggest, it comments out the area specified by the motion.It makes commenting out your code a breeze.
   
   For instance, gcG -> comments out contents of the file from current cursor position to end of file. You can use it with a lot of other motions.

6. morhetz/gruvbox
   This is the colorscheme that I’ve been using for quite a while now. I’m using the same scheme with iTerm2 as well. This is how it looks with a vimscript file open.

7. junegunn/fzf
   A fuzzy finder which helps finding files really QUICKLY. Just enter FZF as an ex-command and It’ll open up a small window with all the files there in your working directory. Start typing out the file name and see it bring up the matching file names in a jiffy. I have it mapped to Ctrl-t to quickly open it up while editing stuff.

8. scrooloose/nerdtree
   It’s a file system explorer that sits inside your vim editor. Think of this as the the file explorer in editors like VSCode and SublimeText. Although I don’t use Nerdtree pretty often to open files but I like to have the directory structure visible to me while I’m working.

9. ervandew/supertab
   This allows you to use Tab to autocomplete keywords. Note that this uses vim’s native completion feature and not any of it’s own. This is not similar to YCM in any way.

10. danishprakash/vimport
    This plugin helps you quickly import and drop python packages and/or modules that you’ve imported in your project. I use it when I’m working on python projects wherein I’m adding and removing modules on a rolling basis.

Updates

I’ll keep this list updated with new plugins that I install or remove plugins from the list above if I find a good enough alternate for it within vim itself or another plugin which might offer better features.

Introduction

First, let’s understand some of the terms used in this context.

1. Vim directory - For GNU/Linux and macOS, it’s generally ~/.vim. This is where vim plugins and autoload files are stored just to make everything more organized and easy to use. Your colorschemes, plugins, and even plugin managers are kept in this directory.
2. runtimepath - If you are familiar with what $PYTHONPATH in python dev environment does or what $PATH in Unix does, then this is quite similar to these two. The runtimepath or rtp is where vim looks for commands which are not defined natively such as plugin commands or auto commands.

What really goes behind the curtains when you install a vim plugin is essentially just this.

• You write or download a vim plugin.
• place it in a folder, preferably ~/.vim/plugins/
• Now add this path to the vim runtimepath
• Finally, source the plugin file

Installation

If you’ve been following along, you probably have an idea now how to manually install a vim plugin. If not, it’s alright, keep reading. We’ll be using one of my plugin for this example.

1 Clone the plugin to the appropriate vim directory.

$ git clone https://github.com/danishprakash/vimport.git ~/vim/plugins/

2 Update the vim runtime path. The easiest way to do this is by appending the plugin dir path to the runtimepath. And you can verify this by checking the value of the rtp variable.

:set rtp+=~/.vim/plugins/vimport
:echo &rtp

3 Finally, let’s source the plugin file.

:source ~/vim/plugins/vimport/plugins/vimport.vim

Now you should be able to run the plugin, try this command.

:Vimport requests

–

Conclusion

As it turns out, it’s pretty simple to actually install a vim plugin. But if you are one of those who use quite a number of plugins, the management of these could turn into a nightmare pretty quick. And for the same reason, I’d rather use a plugin manager and so should you.

Shell

What is a shell? In a very simple way, it could be defined as a tool or a program through which you can(should) interact with the operating system. This definition is very vague but it gives the idea.

Requirements

• C programming knowledge
• gcc
• text editor

Starting of the shell

This is the driving function of the loop. Let us see how things are implemented and how they work when the shell starts. First things first, a prompt lets the user know that the terminal is ready to accept commands from the user. A prompt can be heavily customized but for the sake of simplicity, we’ll set our prompt as a very basic yet popular > symbol. Consider the code below.

void loop() {
   char * line;
   char * * args;
   int status = 1;

   do {
      printf("> ");
      line = read_line();
      flag = 0;
      args = split_lines(line);
      status = dash_launch(args);
      free(line);
      free(args);
   } while (status);
}

Now, let’s actually focus on the more important parts. We declare a char pointer and 2d char pointer, line and args respectively. The line char pointer will hold the command(string) entered by the user using the read_line() function which is explained below. The status variable stores the return value of functions invoked during command execution. It will determine the termination of the loop i.e If the user enters the exit command, the exit function will return 0 which will force the control to break out of the loop and the shell would terminate. The last two lines of the do-while loop basically frees the memory used by the two pointer variables using the free() function, freeing up memory explicitly is required in C and is a good practice.

Reading user commands

char * read_line() {
  int buffsize = 1024;
  int position = 0;
  char * buffer = malloc(sizeof(char) * buffsize);
  int c;

  if (!buffer) {
    fprintf(stderr, "%sdash: Allocation error%s\n", RED, RESET);
    exit(EXIT_FAILURE);
  }

  while (1) {
    c = getchar();
    if (c == EOF || c == '\n') {
      //printf("\n");
      buffer[position] = '\0';
      return buffer;
    } else {
      buffer[position] = c;
    }
    position++;

    if (position >= buffsize) {
      buffsize += 1024;
      buffer = realloc(buffer, buffsize);

      if (!buffer) {
        fprintf(stderr, "dash: Allocation error\n");
        exit(EXIT_FAILURE);
      }
    }
  }
}

Let’s see what’s happening here, first of all, we declare an int variable, buffsize, and initialized it to 1024 bytes. Next, a char pointer, buffer variable is allocated memory through malloc the size of buffsize.

The reason for using dynamic alloacation here instead of static is because you cannot determine the lenght of the command entered by the user. So, the most logical way is to allocate more memory as and when required.

An infinite while loop is started off, first line of the loop gets the character entered by the user and stores it in c. If it is and EOF or \n, a null terminator is returned. If not, the character is stored in hte buffer char pointer.

In the final conditional statement, we check if the size of buffer is equal to or greater than that of bufsize i.e if the current size of the buffer char pointer is equal to or greater than the size we initialized it with, we need to allocate more memory buffer so as to continue reading user input. For this, we double the value of buffsize and then pass it onto the realloc() function along with buffer. The realloc returns a variable with the new size passed as argument and all the data copied to the new variable.

The if(!buffer) checks are for making sure memory was allocated to buffer successfully otherwise malloc and realloc return NULL in unsuccessfull memory allocation attempts. If that happens, our function returns with an error.

Tokenizing Input

Once we have the command entered by the user as a char pointer array. We’d tokenize (read split) it making it easier for us to execute them. We define the function split_line() with one a character pointer as an argument. In this function, we’ll do memory mangement in the same way we did it in the read_line() function. Other variables here include **tokens and *token. We will be using the strtok() function for the task. It takes two arguments, the string to be tokenized and the delimiters. We’ve specified 4 del

For instance, consider this:

str1 = strtok("this is it!", " ");
// str1 -> "this"

str1 = strtok(NULL, " ");
//str1 = "is"

str1 = strtok(NULL, " ");
//str1 = "it!"

First call to the strtok function returns the first token and every subsequent call expects the input as NULL and starts from where it left off in the previous iteration. Now, the code for split_line should be easily understood.

char * * split_line(char * line) {
  int buffsize = 1024, position = 0;
  char * * tokens = malloc(buffsize * sizeof(char * ));
  char * token;

  if (!tokens) {
    fprintf(stderr, "%sdash: Allocation error%s\n", RED, RESET);
    exit(EXIT_FAILURE);
  }
  token = strtok(line, TOK_DELIM);
  while (token != NULL) {
    tokens[position] = token;
    position++;

    if (position >= buffsize) {
      buffsize += TK_BUFF_SIZE;
      tokens = realloc(tokens, buffsize * sizeof(char * ));

      if (!tokens) {
        fprintf(stderr, "%sdash: Allocation error%s\n", RED, RESET);
        exit(EXIT_FAILURE);
      }
    }

    token = strtok(NULL, TOK_DELIM);
  }

  tokens[position] = NULL;

  return tokens;
}

After every iteration, we update the tokens variable by assigining the token in it’s respective position. And finally return the tokens variable.

Exiting the shell

Since it is a simple program, a simple return 0 statement would be enough for us to exit the program successfully. Let’s create a trivial function which returns 0.

int dash_exit(char **args)
{
	return 0;
}

We’ll later on check if the user has entered exit and invoke this function appropriately.

Executing commands

After all the hard work above, the last step is rather trivial, thanks to the syscalls execvp fork.

int dash_execute(char * * args) {
  pid_t cpid;
  int status;

  if (strcmp(args[0], "exit") == 0)
  {
  	return dash_exit();
  }

  cpid = fork();

  if (cpid == 0) {
    if (execvp(args[0], args) < 0)
      printf("dash: command not found: %s\n", args[0]);
    exit(EXIT_FAILURE);

  } else if (cpid < 0)
    printf(RED "Error forking"
      RESET "\n");
  else {
    waitpid(cpid, & status, WUNTRACED);
  }

  return 1;
}

fork allows us to create a new process by duplicating the current process, referring it to as the child process. The current process is thereby referred to as the parent process. The child process is a duplicate of the current(parent) process except for the process ID. When we invoke the fork system call, it returns the process ID of the child in the parent process. In the child process, the process ID is 0. So, after invoking the fork call, we check the value returned by it to make sure we are in the child process or the fork syscall executed successfully.

If the fork was successfull, we will use the execvp command to execute the command. This plays out well for us because the way execvp works is, it replaces the current process with a new process image which in this case is the command that needs to be executed. It returns -1 only if there is an error. Lastly, with the waitpid function, we are making sure the child process finishes successfully.

Note here that we are doing a check for the exit command. It doesn’t matter if the **args variable has more items than just the string exit. It’ll simply return the function dash_exit which in turn will return 0. We could’ve had returned 0 right inside the check but this makes it much more understandable and makes for a good practice.

Everything that we’ve done

Here’s the complete code for our basic shell. The code still has some on the things that aren’t explained in the article but they are crucial to the functioning of our shell for instance, color-coded errors and warnings.

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

#define TOK_DELIM " \t\r\n"
#define RED "\033[0;31m"
#define RESET "\e[0m"

char *read_line();
char **split_line(char *);
int dash_exit(char **);
int dash_execute(char **);

int dash_execute(char **args) {
  pid_t cpid;
  int status;

  if (strcmp(args[0], "exit") == 0) {
    return dash_exit(args);
  }

  cpid = fork();

  if (cpid == 0) {
    if (execvp(args[0], args) < 0)
      printf("dash: command not found: %s\n", args[0]);
    exit(EXIT_FAILURE);

  } else if (cpid < 0)
    printf(RED "Error forking"
      RESET "\n");
  else {
    waitpid(cpid, & status, WUNTRACED);
  }
  return 1;
}

int dash_exit(char **args) {
  return 0;
}

char **split_line(char * line) {
  int buffsize = TK_BUFF_SIZE, position = 0;
  char **tokens = malloc(buffsize * sizeof(char *));
  char *token;

  if (!tokens) {
    fprintf(stderr, "%sdash: Allocation error%s\n", RED, RESET);
    exit(EXIT_FAILURE);
  }
  token = strtok(line, TOK_DELIM);
  while (token != NULL) {
    tokens[position] = token;
    position++;

    if (position >= buffsize) {
      buffsize += TK_BUFF_SIZE;
      tokens = realloc(tokens, buffsize * sizeof(char * ));

      if (!tokens) {
        fprintf(stderr, "%sdash: Allocation error%s\n", RED, RESET);
        exit(EXIT_FAILURE);
      }
    }
    token = strtok(NULL, TOK_DELIM);
  }
  tokens[position] = NULL;

  return tokens;
}

char *read_line() {
  int buffsize = 1024;
  int position = 0;
  char *buffer = malloc(sizeof(char) * buffsize);
  int c;

  if (!buffer) {
    fprintf(stderr, "%sdash: Allocation error%s\n", RED, RESET);
    exit(EXIT_FAILURE);
  }

  while (1) {
    c = getchar();
    if (c == EOF || c == '\n') {
      buffer[position] = '\0';
      return buffer;
    } else {
      buffer[position] = c;
    }
    position++;

    if (position >= buffsize) {
      buffsize += 1024;
      buffer = realloc(buffer, buffsize);

      if (!buffer) {
        fprintf(stderr, "dash: Allocation error\n");
        exit(EXIT_FAILURE);
      }
    }
  }
}

void loop() {
  char *line;
  char **args;
  int status = 1;

  do {
    printf("> ");
    line = read_line();
    args = split_line(line);
    status = dash_execute(args);
    free(line);
    free(args);
  } while (status);
}

int main() {
  loop();
  return 0;
}

Conclusion

This post is more inclined as a learning exercise rather than a full-fledged product. You’d probably never want to use a shell this basic but you probably now know how your favorite shells work under the hood.

I’ve written a more advanced version of this shell including piping, history and other inbuilt commands here.

Contribute

If you find any errors or if you think this article can be improved in any way, ping me.