Data Modeling Is Changing: From Design to Discovery
Traditionally, we’ve used data modeling (conceptual, logical, physical) as a design method for technical data solutions. Today, however, we need to start considering the immense value of data modeling as a method for surfacing context that explains to humans and agents alike how our organization actually functions.
This requires a holistic approach where models are no longer solution-specific artefacts left to rot in project documentation, but become part of enterprise-wide information architecture, infinitely reusable. Data modeling is changing like so many other areas of data & analytics, but in a way that should encourage us to go back to basics.
What is data modeling really about?
There are many misconceptions about data modeling. Increasingly over the years, these misconceptions have been drifting into a highly technical direction, and data modeling has generally become understood as the specific activity of implementing a relational database schema. This is unfortunately an extremely limited view: data modeling can be so much more.
Let’s start unraveling this with a very simple statement: the point of data modeling is to figure out the structure of the information we have about some area of business.
Now, I’m being deliberately vague there. I don’t want to turn this into yet another all-encompassing definition of modeling – we have had plenty of those over the last few decades!
Rather, I want you to think about this from a practical perspective. Your goal might be to implement a relational database schema in the end. For that goal, figuring out the structure of information we have seems obviously helpful. But what if you were doing something else? Could you still benefit from data modeling? Could someone else, or something else, benefit from you documenting what you figured out?
To realize these benefits, we must be able to mentally disconnect data modeling from database implementation. Data modeling simply means we figure out the structure of information we have; we might then utilize what we learned for implementing databases, but that’s just one possible goal among many.
This misconception of data modeling as a technical implementation activity is a result of a technology-first approach and an excessive focus on physical data modeling. But in today’s world, the benefits of data modeling are increasingly shifting away from pure engineering and onto higher levels of abstraction.
The three levels
It is generally understood that data modeling has three different levels of abstraction. The exact definitions differ, but I have found the following to be a simple, practical, and quite helpful approach.
- Conceptual modelling is the highest level of abstraction, and the most business-focused. The conceptual model’s goal is to understand and document the business entities that, and map how they are connected to each other. It’s completely technology-agnostic; in fact, because a good conceptual model describes a slice of reality instead of a system, it should remain valid forever unless the business itself changes.
- Logical modeling is aimed at use case design. It involves finding and documenting a shape that best suits the needs of a particular use case. These shapes are what people usually consider different data modeling methods: dimensional, Data Vault, normalized, One Big Table, activity schema… It’s perfectly possible to derive multiple different logical models from the scope of a single conceptual model. After all, the business entities remain the same, only the use case-specific design changes.
- Physical modeling is where technology is considered. The physical model takes into account platform-specific optimization, such as constraints, indexes, partitions etc. It should also involve the mapping of data sources into our target model. In short, the physical model should be an exact description of the actual implementation.
The general wisdom has been for decades that in order to build sensible data solutions, we should follow a design process that includes these modelinglevels from conceptual to logical to physical. Of course, we all know of many projects where this is not done, and people have jumped directly into implementation (perhaps with some minimal physical-level design). Such a shortcut leads to solutions with at besta tenuous connection to what the business actually needs to know about. Often, teams skipping the conceptual and logical steps end up delivering what are effectively copies of source system data, leaving the actual data integration and transformation work to the end-users. Delivery can be fast without design, but there’s no such thing as a free lunch.
However, even when this design process is followed, the full potential of data models is not always utilized. The models can be immensely valuable far beyond a single solution’s scope!
Design vs. Discovery
The idea with the three levels of data models is to use conceptual modeling to capture an understanding of the actual business, and then to derive the logical and physical models from that in order to deliver a design for a solution or a data product. But the conceptual model, as described above, is a model of a slice of reality, and thus technology- and solution-agnostic. And who or what is hungry for information about how our business works?
In the era of AI, semantics and context are keywords filling everyone’s LinkedIn feeds and conference agendas. The idea is very simple: if we are able to tell AI ‘this is how things work around here’, then the AI system has much more relevant knowledge to work with when answering the questions we pose and performing the tasks we assign to it. The results will be more accurate, more specific to our business in particular, and more trustworthy – not only based on whatever generic information the AI’s training dataset contained but built on top of our own context.
Now, what is a conceptual model but documented context? When we model a domain or a problem area, we put in the model our shared understanding of what things exist, what they are called, and how they connect to each other.
The modern modeler uses conceptual modelingnot only as a Design step, but also as a Discovery one: discovery of organizational semantics. The model surfaces an agreed understanding of some area of the business that has likely only existed as tacit knowledge until now. What we have discovered must be fed into our organizational context and used and reused over and over again by our ever-increasing agent armies.
What that “organizational context” then looks like differs from one organization to another. Some data catalogs are evolving in a more semantics-oriented direction, elsewhere people are building knowledge graphs based on RDF or other methods. No matter what the shape, format, or technology of this context layer is, we should feed it with our conceptual modeling discoveries.
An added bonus to this Discovery path is that if we are also using the same conceptual model as a Design step for some data product, we’ll have no problem connecting our data with its context. The implementation design is derived from the same things and relationships that we just added into our context layer – a direct mapping between data and its meaning, the “semantic link”, should then be quite straightforward! And AI agents love that stuff, as it allows them to figure out what the data means for the business, and where information about various concepts can be found.
However, the ability to do both Design and Discovery at the same time isn’t something that just pops up organically in an organization: to make this happen, we need to consciously plan our ways of working accordingly.
Information Architecture is context management at two levels
The problem with connecting data modeling efforts into an enterprise-wide organizational context layer is that the traditional way of working – the Design path – is usually a project-level effort. The individuals and teams involved in the modelingwork focus on their own deliverables, and are often unaware of what other teams are doing. Moreover, it might be difficult for a project team to fully internalize the organizational value of the Discovery-related work, as it isn’t necessarily obvious how it helps them directly.
Just like we do Data Architecture work to manage the big picture of our data products and platforms, we need to start doing Information Architecture work to manage the big picture of our data models and context management.
The key to success lies in recognizing that two levels of work exist simultaneously, and that feedback loops between them make or break the Information Architecture effort:
- At the project or solution level, individual teams must be able to deliver valuable results as quickly and efficiently as possible.
- At the enterprise level, we must ensure that all teams across the organization are able to deliver valuable results as quickly and efficiently as possible.
The distinction is small but important. Actual delivery and value-creation happens in individual projects, but at the enterprise level our goal is to improve everyone’s success rate. Feedback loops are needed both ways: individual projects must feed the enterprise level with new discoveries, while the enterprise level feeds individual projects with reusable information that someone else has already figured out.
In terms of Information Architecture, this means two things:
- We must create processes that enable new data models to be systematically incorporated from projects at the enterprise level. Usually, this means that someone (an Information Architect, perhaps?) must be available to inspect the new project-level discoveries and to add them to the enterprise knowledge base. Crucially, this must be an asynchronous process: the project should be able to continue on its merry way while this goes on!
- We must enable enterprise-level information to be freely available and easily usable, so that new projects can kick-start themselves based on existing models and semantics. This means that the knowledge can’t just exist in some ivory tower EA tool that no one but the grand wizards will ever open. When a new project starts in, say, the invoicing domain, they should immediately get their hands on the current best understanding of what the “Invoicing domain model” looks like.
And of course, the same basic idea should apply not just to Information Architecture, but many other things as well! For too long we’ve built disconnected enterprise-level artefacts of all kinds. The feedback loop is what makes this work, and it requires an explicit design of roles and processes so that the positive spiral starts spiraling.
What this means for Data Modelers
The important takeaway here is that data modeling is changing in two ways:
- From design of technical database schemas towards business-focused design of semantics and conceptual structures
- From solution-level design workflows towards enterprise-level context discovery
As AI coding agents take more and more responsibility for low-level implementation work, the overall abstraction level of human work is changing everywhere. What I’ve described above is how I believe datamodelingis being affected. Interestingly, for us data modelers the changes seem to be equally driven by both AI’s “output” (i.e. AI-generated code) and AI’s “input” (i.e. the context AI needs to perform).
Adjusting to these changes means learning new skills and methodsat both the individual and organizational levels; but the good news is that we’ve always had these methods at our disposal. The AI era is simply reminding us to go back to the business – that’s what really matters!
