David Slater August 18, 2024
In our quest to understand and predict the performance of complex systems, it’s easy to become carried away with the imagination of the engineering, the potential and sophistication of the AI algorithms, and the elegance of the models. Yet, in a recent piece I wrote reflecting on these topics[1], I realized a glaring omission: there was not a single mention of “safety.” This led me to ponder why I could overlook and not discuss such a now mandatory and critical concept, and what this oversight and omission implies about our relationship with the systems we create.
I reluctantly came to the conclusion, that it represented an implicit admission of ignorance, or the realisation of the limits of our ability to control completely our environment and systems. Because if we concentrate on understanding how these systems work in practice, being able to confidently predict how our designs will operate in the real world, then there is no need to consider it, it will always do what it says on the tin –(safely?)
But in the real world we recognise variability and uncertainty. So is a concern for “Safety” an admission of unavailability of total control and certainty (or possibility of incompetence)?
Safety, it seems, is an implicit acknowledgment of the limits of our control over the environment and the systems we design. It’s a recognition that, no matter how well we plan, or how precisely we execute the implementation, there is always variability in the delivery, i.e., unpredictable factors that can lead to “failure”, foreseen or unexpected (unprecedented?). In an ideal world, where systems operate flawlessly, then safety might seem unnecessary. But the reality is far different.
The notion of “belt and braces” comes to mind—a metaphor for the dual layers of protection we rely on to safeguard against the unforeseen. In this context, the belt represents the result of doing things “properly” from the start, ensuring that the system is designed and built with robustness in mind. The braces, on the other hand, symbolize the additional “safety” measures we incorporate—resilience, redundancy, and fail-safes—to mitigate the risks that our initial design cannot eliminate.
As our systems grow more complex, the need for these “braces” becomes more apparent. We may strive for perfection in design, but the inherent unpredictability of complex systems means that we can never guarantee absolute behaviour as expected (imagined?). This is not just an admission of our limitations, but also a pragmatic approach to dealing with the reality that total control and certainty are unattainable. It’s a recognition that, despite our best efforts, things can—and sometimes will—go wrong.
So how do we proceed in a world where we cannot guarantee that no harm will come from the systems we deploy? The answer we have adopted lies in a concept deeply ingrained in human nature: defence against threats. Just as our ancestors developed tools and strategies to protect themselves from predators, (man, beasts, or the gods), we build “safety” measures into our systems to defend against these threats of their potential consequences. These measures are not integral parts of the design, but can be thought of as (desirable, critical?) add-ons, ensuring that even when something does go wrong, the impact is minimized. We now need to focus as well, on the design of the braces (or suspenders?)
In practical terms, this means designing additional systems, addressing some critical questions: What’s the worst that can happen? Can we survive it? These are not just theoretical exercises but essential considerations that should guide the design of safety measures. Although when this becomes too demanding people tend to settle for addressing the concerns they can afford to (ALARP, BATNEEC, CATNIP if you’ll pardon a regulators inside joke!).
This approach is thus about ensuring that the “belt”—the core design can cope with real world variabilities; that it is strong and reliable. But now also it seems we recognise that the “braces”—the additional safety measures—will also be needed and justified to obtain permissions to operate in the real world, where unscheduled impacts can have serious consequences. But while this was practicable in simple and even complicated systems, in today’s complex sociotechnical operations we need to appreciate that these add-ons can add extra layers of complexity, with even more unforeseen and unexpected consequences.
But the primacy and the importance of the “belt” was pointed out by Trevor Kletz, a pioneer in the field of process safety. Kletz famously advocated for inherently safe design: the idea that “what you don’t have can’t leak”. In other words, by designing systems that eliminate hazards from the outset, we reduce the need for additional safety measures. But in cases where hazards cannot be entirely eliminated, the focus shifts to resilience and real-time recovery and adaptability.
The problem we face is that as we continue to develop ever more complex systems, the role of these “safety” add-ons (safety critical (sub)systems) becomes increasingly crucial. We realise that our “safety cases” need not only to “prove” their effectiveness against currently appreciated threats, but imply that we can guarantee that our systems that can survive and adapt when these complications emerge. As we know in the real world, this is a practical impossibility. This means that “safety” becomes more of a political negotiation, not an engineering design issue.
So, as we push the boundaries of what our systems can do, we must also push the boundaries of how we think about and include “safety”. If it’s about acceptability of uncertainty, then it’s not enough to design for performance; we must also design for resilience. It seems that currently we implicitly accept the possibility of failure, by embracing the “belt and braces” philosophy, which implies we are relying on safety nets. On reflecting on the article, I realised that it was arguing strongly that we re-embrace the Kletz philosophy, so that we create systems that are not only powerful and efficient, but also designed to contain the unexpected, systems that can weather the unpredictable and continue to function in the face of adversity. After all, in a world of increasing complexity, safety is the one thing we cannot afford to ignore, but perhaps is a word we need to design out and make redundant. (get your retaliation in first?).
David Slater
[1] https://www.researchgate.net/publication/383206382_Navigating_the_Complexity_of_Modern_Systems_How_FRAM_provides_a_new_and_more_pr