Everyone knows that I have been and continue to be an active software developer. After I wrote my last blog post, my ex-boss at Wesleyan (John Meerts) reminded me of the first homegrown registration system that I helped write at Wesleyan in the mid 1990’s. This was a game changer at that point and it created the trust of the community in our abilities. This allowed me to both write and lead several other major programming initiatives there which I have carried with me to other institutions.

But, as a software developer in small institutions with small staff to work with, you always encounter issues. There is a lot of excitement around software development, but there is also a lot of risk involved. The community comes to rely on them and wants constant enhancements which can never be predicted ahead of time. Also, with constant changes to the web browser and underlying technologies, you need to continually monitor and apply fixes. As a result, we end up struggling to support these in the long run. Also, when we write our own, our capacity to thoroughly test them is limited and it leads to mistakes. I just wanted to highlight some such mistakes that I have committed myself or have seen it done by others. For the sake of anonymity, I am not going to say who committed which one and leave it to your imagination…

When I began my career at Wesleyan in 1986, my primary responsibility was to support  high end scientific computing. However, I also wanted to learn the ropes of programming by my fellow IT folks. We were using Digital Equipment Corporation’s DEC-20 systems and just beginning to get into VAX/VMS system. We were connected to BITNET, a precursor to the internet, to which many other higher eds were also connected. For reasons beyond my comprehension, the IT administration decided not to use the native emailing system from VAX/VMS which would have worked with BITNET and the earlier versions of the internet. Frankly it was a terrific emailing system.

So, we developed a home grown system called MINT. It was written fully in VAX Macro which is an assembler language. Assembler language is like one step above the machine instructions of 0s and 1s. This was an amazing accomplishment in the end, but was a horrendous system to support and debug. We had several and constant errors to contend with.

Related to this, BITNET itself provided some nice utilities such as the ability to send receive files from other connected institutions. We needed something similar for internal users to send and receive files from others. So, we decided to write a home grown send/receive program in VAX/VMS Macro. Basically, when you want to send, you specified one or more files to send to a list of recipients and the program will make a copy of the files to a central protected space and send a notification to the recipients to run the receive command to copy them to their own directory of choice.

I will spare you the details, but this needs to be run in privileged mode because there is a protected central repository involved. In VAX/VMS Macro, you can grant additional privileges to a program, which can either be applied to the entire program or the programmer can activate these on and off within the program. Due to the laziness of the programmer, send/receive was privileged at the top level rather than at each step. What this meant was when you issued a send command it was privileged and the issuer had access to everyone’s files!

Any user can use the wildcard some_users_directory:*.*, which meant select all files in some other user’s directory and send it to myself. But that would not work because everyone had limited disk space. However, there were some common files in everyone’s directory that had the same name. And one was MINT.TXT file, containing all emails. So, any user can send anyone else’s email file to themselves or their friends! How about that? And a smart student found this out, demonstrated this to a few friends and it was a party! Of course, soon after we found out, we stopped the command from being able to run, looked at the code and fixed it. The privilege was needed only when one needed to copy the file to the central protected area and not at the time of fetching the file!

The other thing that I distinctly remember is that the online registration that we had developed did not its job properly during one cycle. Basically, we allowed the faculty to reserve seats for different classes and any leftover seats were redistributed according to an algorithm after each class finished registering. This was done by a script whose name I distinctly remember – redistribute_seats.pl (just because I remember this, I strongly suggest you don’t make a mistake by thinking I made the mistake I am about to describe 🙂 ).

One year this simply did not do its job right and the students were really upset. We looked at the issue and found the issue. In perl like languages, there is a distinct difference between one equal sign (=) and two equal signs (==). The former is an assignment type of operator, meaning if you said $a = 10; a variable name a is assigned a value of 10. The two equal signs is used for a logical comparison of equality. For example if you had a statement if ($a == 10) then you are asking if the variable a is equal to 10 and if so, do whatever you ask the system to do within a block. Similarly, you can test the equality of a variable to a text constant, such as if ($b == ‘Registered’).

And a programmer did not understand this fundamental difference (because that person was learning perl at that time, though that person was very knowledgeable in other languages) and had modified the logic to read if ($some_variable = $some_value). This simply equated the value to a variable and returned a true always which completely changed the actions of the script and landed us in deep trouble!

Most recently we had a permission bleed issue, something very similar to the send/receive type issue. A user was logged into one app and for some unknown reason, the programmer had granted an enhanced access to all data in that app. This did not matter for that specific app because by having this enhanced privilege, they did not gain any more information than if they didn’t. However, because of the way our apps are architected using single sign on, this enhanced privilege transferred to any other app that the user was accessing and one such app exposed more data to an end user than it should. Once it was reported, we fixed the issue, but this created a lot of problems as you can imagine. Here is a case, after the fact, one would say we should have thoroughly tested these scenarios. Absolutely, it is just that it never occurred in the first place. There is a drastic difference between the consequences of these errors now than in the 90’s and early part of this century. The new laws around data and privacy are so strict that any such errors with now create very expensive consequences to the institution.

Though we are becoming very vigilant about data access issues, bugs are a way of life in the software development world. As you know, even the large software companies don’t get this right all the time and we hear about bug fixes and breaches as a result of bad design all the time. Sometimes, it may be best for a small organization like us not to be in the software development business. I just don’t see how, at this point, but we need to get there!

As always, there are many many more that I can think of… I just have to stop here and not bore you to death and just got a couple of bug report emails that I need to attend to 🙂