In a series of three posts, I have addressed some typical aspects of programme management for decision support. The first post defined programme and project management in Business Intelligence (BI) and its relationship with IT architecture. The second post dealt with some issues in governance, the tensions that arise between business and IT and how to deal with them. This post proposes a new way of designing systems for both transaction and decision support to improve the organisation’s effectiveness further.
This third post looks at a new way of designing systems for both transaction and decision support to improve the organisation’s effectiveness further. I will examine the concept of BI architecture further and give hints of how BI programme management can evolve towards an ideal architecture which merges transaction and decision support systems in a powerful ensemble, ready for the new economic challenges.
I propose an “Idealtyp” knowing that no existing organisation can achieve this in less than a decade for reasons like sunk cost fallacies, the dialectics of progress and simply resistance to change.
But new organisations and innovators who can make the change will notice that the rewards of this approach are immense. They will combine architectural rigidity with business agility and improve their competitive power with an order of magnitude.
I am a fan of Max Weber’s definition of “Idealtyp”[i], which has direct links with architecture in information technology. BI architecture is an abstraction of reality, and as such an instrument to better understand a complex organisation of hardware, network topologies, software, data objects, business processes, key people and organisational units. All these components interact in –what appears to outsiders- in a chaotic way. An architectural framework brings order to the chaos and provides meaning to all the contributors to the system.
Architecture is used as a benchmark, a to be situation by which the present state of nature can be measured. It is a more crisp and more manageable concept than CMM-like models which express maturity sometimes in rather esoteric terms. For a quick scan, this will do but for in-depth managing of the above mentioned BI assets, an architectural framework is better for BI environments.
Table 1. Example of the BI version of the Capability Maturity Model as described in Business Analysis for Business Intelligence on page 202. In the book, it is positioned as a tool to help the BA with identifying broad project management issues
Proposing an ideal BI architecture is one thing, achieving it, another. I will only mention three serious roadblocks on the path towards this ideal BI architecture that unifies transaction systems and decision support systems: the sunk cost fallacy, the dialectics of progress and resistance to change.
The sunk cost fallacy is a powerful driver in maintaining the status quo; organisations suffering from this irrational behaviour consider they have invested so much effort, money, hardware, training, user acceptance and other irretrievable costs that they should continue to throw good money at bad money. And sometimes the problem is compounded when the costs were spent on technology from market leaders.
No one ever got fired for buying… (fill in any market leader’s name)
No matter what industry you look at, market leaders fulfil their basic marketing promise: provide stability, predictable behaviour and a very high degree of CYA (google it) to the buyer. But that doesn’t mean the purchase decision is the best possible decision for future use. Market leaders in IT are also very keen on “providing” vendor lock-in, disallowing the client to adapt to changing requirements.
As a footnote: today, buyers are more looking at the market cap or the private equity of the Big Data technology providers than at their actual technical performance and their fit with the organisation’s requirements. Yes, people keep making the same mistakes over and over…
At the other end of the spectrum are the dialectics of progress: this law was discovered by the Dutch journalist Jan Romein who noticed that gas lights were still used in London when other European capitals already used electricity. This law suggests - and I quote an article on Wikipedia -that making progress in a particular area often creates circumstances in which stimuli are lacking to strive for further progress. This results in the individual or group that started out ahead eventually being overtaken by others. In the terminology of the law, the head start, initially an advantage, subsequently becomes a handicap.
An explanation for why the phenomenon occurs is that when a society dedicates itself to certain standards, and those standards change, it is harder for them to adapt. Conversely, a society that has not committed itself yet will not have this problem. Thus, a society that at one point has a head start over other societies, may, at a later time, be stuck with obsolete technology or ideas that get in the way of further progress. One consequence of this is that what is considered to be the state of the art in a certain field can be seen as "jumping" from place to place, as each leader soon becomes a victim of the handicap.
As always, resistance to change plays its role. New tools and new architectures require new skills to be trained, new ways of working to adopt and if one human species has trouble adapting to new technologies it is… the tech people. I can produce COBOL programmers who will explain to you that COBOL is good enough for object oriented programming or IMS specialists who see nothing new in the Big Data phenomenon…
Here’s architecture explained in an image. Imagine Christopher Wren would have disposed of modern building technologies. Then either the cathedral, based on the architecture “as is” would have looked completely different, with higher arches, bigger windows, etc… Or… the architecture could have evolved as modern technology would have influenced Wren’s vision on buildings.
Exactly this is what happens in BI architecture and BI programme management.
Figure 1. On the left: architecture, right: a realisation of architecture as illustrated by Wren’s Saint-Paul’s Cathedral
Architecture descriptions are formal descriptions of an information system, organized in a way:
It is also the interaction between structure, which is requirements based, and principles applicable to any component of the structure.
BI Architecture should reflect how the BI requirements are realized by services, processes, and software applications in the day-to-day operations. Therefore, the quality of the architecture is largely determined by the ability to capture and analyse the relevant goals and requirements, the extent to which they can be realized by the architecture, and the ease with which goal and requirements can be changed.
Figure 2. The Open Group Architecture Framework puts requirements management at the centre of the lifecycle management. The connection with business analysis for business intelligence is obvious.
Now we have established a common view on BI architecture and programme management, it is time to address the murky reality of everyday practice.
Although Frederick Taylor and Henri Fayol’s ideas of separation between doing and thinking have been proven inadequate for modern organisations, our information systems still reflect these early 20th Century paradigms. You have the transaction systems where the scope is simply: execute one step after another in one business process and make sure you comply with the requirements of the system. This is the world of doing and not thinking. Separated from the world of doing is the world of thinking and not doing: decision support systems. The business looks at reports, cubes and analytical results extracted from transaction and external data and then makes decisions which the doers can execute.
What if the new economy were changing all this in a rapid pace? What if doing and thinking came together in one flow? That’s exactly what the Internet is creating, and I am afraid the majority of organisations are simply not ready for this (r)evolution. Already in 1999, Bill Gates and Collins Hemingway [ii] wrote about empowering people in the digital age when they gave us the following business lessons:
I am afraid this advice still needs implementation in many organisations. The good news is that contemporary technologies can support the integration of doing and thinking. But it will require new architectures, new organisational and technological skills to reap maximum benefits from the technology.
The major and most relevant BI programme management decision criterion will be the answer to the question: “Which quality data yield the highest return in terms of competitive advantage?”
The schema below describes a generic architecture integrating transaction and decision support systems in one architectural vision. Let’s read it from left to right.
Any organisation has a number of business drivers, for example as described by Michael Porter’s generic strategies: be the cost leader, differentiate from the competition or focus on a niche. Parallel with the business drivers are decision making motives such as: “I want complete customer and product insight” and finally, the less concrete but very present knowledge discovery driver to make sure organisations are always in the lookout for unpredictable changes in the competitive environment. These three drivers define a number of business objects, both static and dynamic. And these entities can be endogenous to the organisation (like customer, channel, product, etc.) or they can be external like weather data, currency data, etc… These business objects need to be translated into data objects suitable for transaction and decision support.
Figure 3. This is the (condensed) target architecture of an integrated “Big Data Warehouse”: combining batch and stream processing using low latency for operational intelligence and aggregate data for tactical and strategic decision making. Built from the ground up using data in stead of business processes as the analytic cornerstone.
Conclusion: an integrated view on transactions and decision making will improve BI programme management supported by this architectural vision. The major and most relevant BI programme management decision criterion will be the answer to the question: “Which quality data yield the highest return in terms of competitive advantage?” And thus, which project (whether on the transaction or decision support systems need the highest priority in allocation of resources?
20 en 21 maart 2018 We worden al jaren geconfronteerd met een niet-aflatende stroom van nieuwe technologieën, ontwerptechnieken, architecturen en ideeën, zoals Hadoop, streaming, self-service data preparation, logical datawarehouse, d...
Data First. Dat is het voornemen van veel bedrijven in de transitie naar een datagedreven organisatie, maar ook gelijk het advies dat onze topsprekers op 20 en 21 maart aan de bezoekers van de Datawarehousing & BI Summit mee willen geven. We worden n...
12 april 2018 Praktische workshop Datavisualisatie en Data-driven Storytelling. Hoe gaat u van data naar inzicht? En hoe gaat u om met grote hoeveelheden data, de noodzaak van storytelling, data science en de data artist? Lex Pierik behandelt de...
17 en 18 april 2018 Het Logical Data Warehouse, een door Gartner geïntroduceerde architectuur, is gebaseerd op een ontkoppeling van rapportage en analyse enerzijds en gegevensbronnen anderzijds. Een flexibelere architectuur waarbij sneller ...
19 april 2018 De nieuwe Europese General Data Protection Regulation (GDPR of AVG) vervangt in mei 2018 onze Nederlandse privacywet, de Wet Bescherming Persoonsgegevens. Met hoge boetes, een toegenomen bedreiging van datalekken en hacks, de ...
14 en 15 mei 2018Een verhelderend seminar waarin kritisch en onafhankelijk de populaire technologieën voor het opslaan en bewerken big data onder de loep worden genomen door datawarehouse- en databasespecialist pur sang Rick van der Lans. A...
28 en 29 mei 2018Praktische tweedaagse workshop met internationaal gerenommeerde spreker Alec Sharp over het modelleren met Entity-Relationship vanuit business perspectief. De workshop wordt ondersteund met praktijkvoorbeelden en duidelijke...
29 en 30 mei 2018 Vergaar nieuwe databronnen en benut analytics en machine learning om de concurrentie een stap voor te blijven. Er wordt gekeken hoe machine learning en advanced analytics technieken zoals tekst-analyse, sentiment analyse, gra...