Amherst College, a premiere residential liberal arts college, lost network connectivity for almost a week. The college supported almost all of the technology services locally, which meant that pretty much everything was inaccessible for that period – Email, Learning Management System, Web site, administrative systems etc. And the faculty could not connect to the web from classrooms and students needed to use their cell phones to connect to the outside world. As one of them tweeted, students who could not afford to have unlimited data plans were limited from doing even this. You can read about the details here.

The IT staff did a remarkable job given the circumstances and had the community support all through, based on what I have heard. And I am so thankful for them coming out and sharing their experiences openly with their colleagues. This is so important for the rest of us to learn from, not just the technology piece, but how to best manage such a crisis.

What really happened? It is a complicated story on a lot of fronts, but the core issue that caused this outage is due to lack of investments in network hardware. Because they are still running on hardware that is pretty old, their network is configured as a “flat” network (Layer 2). Most modern networks are Layer 3 networks where we can segment networks based on a variety of criteria, such as a separate segments based on particular buildings, or connections from classrooms etc.

Amherst suffered what is called a Mac Flap Storm. Each network device has a unique address, called the MAC address and the networks operate under this uniqueness assumption to forward the network data to the appropriate device. Any compromise to that can “flood” the network and it is especially worst in Layer 2 networks. It will basically cripple the entire network. This can happen either because network wires create a short circuit or a misconfiguration either of which can advertise the same Mac address on two or more ports. This is most probably what happened in Amherst case. The worst thing about the MAC flap storm is there is no easy way to detect them!

(more…)

Congratulations to the Patriots on an incredible feat! It was one of the most boring and uninspiring games, but in the end, what matters is the score.

For me, this is Superbowl XL. The first superbowl I saw was in January of 1979, barely 5 months after I arrived in this country. As a young man thousands of miles away from home, deeply homesick, I needed distractions and sports provided that.  Just like the millions of young Indian men, I was a devoted fan of cricket at that time (and continue to be). Only issue was that cricket was not as abundant as it is today. And it was played all around the world in odd timezones. We used to be glued to short wave radios for commentary and read the newspapers for additional details and photographs. I come to the US where cricket was only known as an insect at that time and not as a sport!

I used to go every day to Indian Embassy, some 6 blocks away from Hunter College with the hope that they had “The Hindu” newspaper. This is a regional newspaper that covered Tamil Naddu, where I am from, and I enjoyed their Sports coverage. The Embassy typically had newspapers from approximately a week prior and The Hindu was not their priority because it is a regional newspaper. Imagine getting scores a week later and sometime missing a key day! It was depressing.

So, i gave in to learning about American sports. My apartment mate tried to teach me football by having me watch both College Football and Pro football on the tiny 13 inch black & white TV.  It is enough to say that he was much better at teaching me theoretical chemistry than football. I became friends with another Indian family in the neighborhood and that is where I learned the intricacies of the game.

(more…)