As is often the case real IT operators in large organisations find themselves having to deal with "enterprise" software which has been imposed upon them. The decision to implement such software is usually determined by perceived business requirements (which is reasonable enough), but with little consideration of the operations and flexibility for new, or even assumed, needs.

For example, a certain large university which I have some familiarity with has decided to introduce an absolutely $awful_ticketing_system campus-wide. It serves as an excellent example because every step of the way there are just terrible things about this which illustrates how IT management decisions cause unnecessary problems for IT operators.

The beginning of this story comes with what initially seemed to be a simple problem. Emails that were sent to the ticketing system were being automatically assigned as an Incident, which had a tight Service Level Agreement period, as one would expect. The tickets that were being received, however, were not Incidents, but rather feature Requests. Changing this would be simple and easy (more on this in a moment), right?

Apparently not. I feel for the poor person who had to respond to me with this explanation.

Technically, an incoming mail to a mailbox can be converted into any type of ServiceNow ticket.

Well, this is good news.

But, out of all ticket types, this conversion is the most complicated for "request" type of tickets due to its three layered structure (REQ, RITM, CTASK). It has to have all three layers created for each request type of ticket. There is no out of the box or simple way of converting incoming mails to request tickets. Hence, it has not been implemented so far.

Wait, what? In well-known FOSS ticketing systems (e.g., OTRS, Request Tracker, Trac), the operator can set the queue or equivalent. Here, with this proprietary enterprise software, not even the administrators can do it in a simple manner. The solution being developed? Get rid of email tickets and use a web portal instead.

... the option of mailbox that creates tickets in Servicenow, was made available for use cases where users (such a externals, guests, students etc.) could not access forms and hence could not request relevant services. But the idea was always to use $awful_ticketing_system Portal as the access point to report any issues or request services, thereby avoiding the need to use mailboxes.

I'll leave it to others to think about how a web portal won't be associated with a mailbox or how this will allow operators to assign the ticket type. But don't spend too much time on it. Because this specific software and this specific example isn't the real issue at hand.

The first problem is that IT Managers don't listen carefully enough to IT Operators. An article by Jeff Ello, ten years old now, explains the many problems involved. The entire thing is worth reading, multiple times even, but to give a summary of a single paragraph from the article:

While everyone would like to work for a nice person who is always right, IT pros will prefer a jerk who is always right over a nice person who is always wrong. Wrong creates unnecessary work, impossible situations and major failures. Wrong is evil, and it must be defeated. Capacity for technical reasoning trumps all other professional factors, period.

There is one additional statement that needs to be added to this: This is not optional. In many other human-to-human roles, such as management, it is possible to reconstruct questions, find a compromise between competing agendas, and so forth. This is not possible with IT Operators (assuming they are honest and knowledgeable), not because they don't want to do it (whatever "it" is), but because it simply isn't possible. To repeat: technical reasoning trumps all other professional factors. When it comes to technical matters, IT Managers need to do more than just take advice from the Operators; they need to do what the Operators tell them. Otherwise, the wrong technical decisions will be made and that will cost time and money in the future.

The second problem, and yes, it cascades from the first, is that IT Managers have an erroneous propensity to choose enterprise software which is complex and easy rather than free and open source software which is simple and hard. The difference between the two has been well stated by Rich Hickey. Managers tend to choose software which is easy to use because they are typically not Operators themselves. They become especially enticed by software which is easy to use and feature-rich because they incorrectly perceive that will satisfy the requirements of (often dogmatic) business logic. When it is inevitably discovered that there are new business requirements the enterprise software needs to be somehow re-developed (at additional cost and time, as determined by the external body), or worked around. or try to find a new enterprise software product (and then experience the horror of vendor lock-in).

All of this is something that would make an experienced FOSS IT Operator's remaining grey hairs stand on end. In our world if something isn't working right, we fix it there and then, and if we can't we ask around because somebody else will know how. Collectively we have more knowledge than any of us individually can have. In the FOSS world, a product can be hard, in both senses of the word. It does have a steep learning curve, but it also is durable. It is however, also simple, that is, it isn't interleaved. Simplicity is a prerequisite for reliability said Edsger W. Dijkstra in 1975. Most FOSS (and yes, there are exceptions) is built on the UNIX philosophy: Design programs to do only a single thing, but to do it well, and to work together well with other programs.

The third problem, and yes, it cascades from the second, is that IT systems are often too heavily orientated towards singular solutions for the entire environment. Now initially I (erroneously) agreed with this approach. There was a time that I would have argued that a single ticketing system throughout an entire campus was a good idea as it lowered administrative and training costs and allowed for a single point from which a variety of metrics could be derived. My concern was that in selecting a single system that the wrong system might be chosen (and remember "wrong is evil and it must be defeated") or, one is terrified to say, the worst system might be selected (and usually it is, for the reasons previously stated).

The problem is with the conceptualisation. A singular system as a monolith will almost inevitably suffer the aforementioned issues of complexity, and if it's a closed-source enterprise product, there's nothing you can do about it. A single system, especially a cloud-based system, will suffer massive performance issues simply due to the physics of distance and physics is not optional. So rather than a single system, it really is worth using multiple systems according to what is most contextually appropriate and which can interact with other systems when needed. Again, getting back to these core computer science concepts of modularity, specialisation, and connectivity. The supposed gains in reducing administrative overhead are actually losses if the operator is unable to adapt to new systems, and that adaption is achieved through knowing how generic systems work together.

By way of conclusion, it is worth thinking in terms of stripped-down, minimalist, free and open-source programs which satisfy all functional requirements and can be modularly extended in a manner that is simple but durable. The philosophy of the team behind suckless.org is an excellent approach. As for a practical example of a ticketing system, Joe Amstrong's example of the Erlang Ticketing System as a "minimal viable program" satisfies the criteria.

"Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand."
-- Archibald Putt, "Putt's Law and the Successful Technocrat", 2006