Generative AI is notoriously eager to help, to the point that if it can’t find something matching what you ask for, it’ll create it. So the problem with relying on guardrails is that all too often, a model will be wrong, showing a high confidence score for an incorrect answer because it’s relying on stale or non-canonical data.
Not only do you need to be able to track the lineage of data your model uses from source to token, something the EU AI Act requires, you also need to be able to take into account where the data came from, whether it’s out of date, if it changed in a way that affects the result, or if it was never really relevant or authoritative in the first place.
Gartner expects organizations will abandon 60% of AI projects because they don’t have the right metadata management, data quality, and data observability. IBM’s acquisition of Confluent also highlights the importance of real-time data with lineage, governance, and policy for AI agents, and one of IBM’s 2026 predictions was the importance of smarter data.
The usual approach is adding metadata and validation later in the data pipeline. That’s similar to the way the bronze, silver, and gold tiers of typical lakehouse architecture are supposed to represent how filtering, cleaning, and augmenting data improves structure and quality until it’s ready to use. That can mean an enormous amount of work since nearly three quarters of the CPU work in training a frontier model is data cleansing and validation.
But that can also remove a lot of the context crucial for gen AI. Rather than cleaning data and losing the original context, it’s often more effective to keep as much information about the original state of the data, says David Aronchick, open-source platform Kubeflow founder, and CEO of distributed data pipeline vendor Expanso. “You can’t pursue exactly purely clean data; that’s just not possible,” he says. “As you pull data into your ML model, every line should have some mechanism saying where it came from. Otherwise, you’re never really going to know because you can’t mix them together and tease them apart later. You can search your raw content, your raw logs, but it’s just not going to be there.”
IoT digital twin systems often tag data all the way back to the device capturing it so you can see whether a temperature spike is a critical failure, which you want to react to, or a routine calibration, which you don’t. But that information may well be relevant down the line when you want to use that data more broadly. So unless you capture at least some elements about the source of data before you move it, you’re not going to be able to easily reconstruct the context later, or at all sometimes.
Ulrik Hansen, co-CEO of Encord, a platform for managing and annotating data, calls this in-stream labelling and cautions it’s not an alternative to cleansing data. “Dirty conflates two things: actual corruption you should fix, and context dependence, where a reading only looks anomalous because you threw away the frame that explained it,” he says. “Cleansing kills both. The point isn’t to stop cleaning, it’s to stop normalizing away context you can never recover.”
Context can be cheap to capture at the source and nearly impossible to recover after, he adds. “The question isn’t whether to keep it,” he says, “it’s about curating what actually helps.”
Raw but not rancid
Aronchick characterizes the state of most bronze tiers as toxic waste because raw data doesn’t get validated before ingestion, or have a metadata wrapper on each data point. “You’ve taken raw data and stripped it of context,” he says.
Take a wind farm operator, for instance. When sensor data about the turbines is generated, it comes from a particular turbine at a particular position in a specific wind farm at a known location, running at a specific speed in specific weather conditions, at a particular time. “If you have other turbines also working in the field, the performance of your turbine will go down, but the field performance will go up,” says Aronchick. “The performance of your turbine going down isn’t a negative, but unless you have the context at the point of data collection, you’re going to make your life much harder later on, when someone asks about the efficiency.”
Metadata needs to be much richer, and it needs to be added as early in your data pipeline as possible when you have the most detail available to make sense of the structure and complexity of the data, Aronchick adds. “You want to capture as much about the data you’re collecting as possible, where it doesn’t require insane activity to do so.”
But not all the metadata you need will be generated with the data, he says. You almost certainly need to augment and annotate your data, and provide extra structure, especially for something like a point of sale system with very light metadata. “Data comes off these things in poor structure,” he says. “It’s not OpenLineage, it’s often a CSV or a text record, and you have to reconstruct them into a full structured log. So do smart things where you’re creating data. That might be compressing, sampling, converting, appending metadata to it, and enforcing schema and lineage all before you start moving anything.”
That doesn’t have to mean bloating your data, Hansen points out. He suggests capturing what’s free and unrecoverable. “The system of origin is the label,” he says. “You don’t tag HR policy, you capture that it came from the HR system. Anything a model can derive later, you can skip.”
Structure isn’t static
Routine changes to APIs, schemas, and how data is collected or stored happen in every organization, and need to be reflected in metadata that lives alongside the data or added as data is collected, not reconstructed later in a fragile process that depends on knowing about all those changes. Google’s research into these data cascades shows how easily context gets lost and how badly it affects data quality.
Shifting schema enforcement further left in your data pipeline so you deal with it as soon as possible allows you to make more effective downstream decisions. For a sensor recording temperature and humidity, you need to know the temperature scale it uses, readings, and how the timestamp is recorded. Checking that against the schema before ingesting the data lets you route it differently depending on whether it validates or triggers alerts about data quality.
“Maybe I’ll delete it, or send it off to some place where a human being or other tooling can reconstruct it into something valuable,” says Aronchick. “But what it doesn’t do is allow the polluted or bad data into my pipeline. Saying whether or not something passed your schema makes your downstream systems much more reliable.”
Sensing structure
Unstructured and semistructured data needs more augmentation. A PDF or Word document has an author and a creation date, but doesn’t necessarily include any context about the job title and department of the author, whether it’s up to date, only applies to a particular group of customers, or is based on accounting regulations that can change. If that information is available, it needs to travel with the document, not be left in a compliance spreadsheet.
Data platforms like DataHub and SurrealDB both capture and create context. The latter can analyze a photo, for instance, using vision AI to understand what’s in the image. “From completely unstructured data, we get as much structure as possible,” says the company’s CEO Tobie Morgan Hitchcock.
That’s paired with other data potentially useful for an AI agent down the line. “Understanding what happened around an event becomes a lot easier if you’re tracking the conversation, telemetry, tool and model usage, geospatial data, and the vector search and relationships,” he says. “You’re going to have a far better chance of getting an accurate understanding of that data, which started off completely unstructured, than if you weren’t capturing anything.”
Metadata about document authors, which might come from the company directory, can show how much authority a document has. He describes that as building an understanding of what trust and provenance is over time by the weight and authority of who’s updating the information. After all, he says, company-generated information has more trust or can have traced provenance compared to conversational inputs from a user.
Incentives for annotating
DataHub CTO Shirshanka Das saw how much of a mess data can be even with strong guidelines as former architect of LinkedIn’s GDPR strategy. “The data was a swamp, despite us having had pretty good data-first and schema-first practices,” he says. As well as cleaning up the data governance, they added in the first nuggets of the DevOps’ ‘shift left’ approach.
LinkedIn already required data checked in to its Kafka ecosystem to have a schema, and ran CI/CD pipelines to check backward compatibility. “I attached metadata attribution and collection around compliance metadata into that pipeline, where developers weren’t able to check in a schema until they had declared what every column meant.”
The extra work was unpopular until teams who didn’t participate saw the flood of tickets that came their way, which allowed him to extend that same proactive governance and annotation at source approach to pretty much every data set being produced.
“The starting point of data at most companies is a lot more swampy,” he says. “Many people are using Kafka, which is a very schema forward system, and yet they’re just shoving in JSON and unstructured stuff.”
That’s common, agrees Megha Kumar, research VP for analytics and AI at IDC, because while collecting more metadata provides better context and cleaner data lineage, it’s hard in practice. “Most organizations batch process data, so real-time context capture rarely happens,” she says. “Even the ones that process in real-time tend to have pre-defined schemas, so adding context requires changes to the data, which unfortunately happens later.”
People don’t know how to start, says Das, so DataHub Cloud tries to add back context by collecting operational metadata from multiple systems, including queries and BI tools to extrapolate a semantic model. “We confront the mess by giving them something they can react to,” he says. “They can quickly validate, and then it starts becoming a governance layer on top where humans annotate at source.”
Online whiteboard provider Miro, for example, dramatically improved AI agent query accuracy from about 50% to 90% using DataHub. Then they applied GitOps principles on top of what was inferred with a human in the loop for approvals.
So getting people to do the work happened the same way at LinkedIn, says Das. “When a data scientist gets 10 times more requests because they didn’t document their work well, resulting in the AI making lots of mistakes and stakeholders constantly pinging them for answers, they have the incentive to add the annotation when they produce an analysis, because then they get out of the critical path.”
DBOMs and data contracts
Provenance and lineage of data is critical, Aronchick says, so you can preserve details like who collected the data, when, from where, if the source was authoritative or canonical, what transformations were run, and exactly what the model saw.
“It’s not just about the version and the metadata,” he says. “Where things really start to change is when you can say along the way this data has gone through these steps, this is the root source, and these were the other elements.” You want to be able to find out if there were any experimental flags, like a new customer campaign running when it was collected, as well as what claims the data contributes to.
Aronchick advocates for a SLSA-style data bill of materials using a tool like Makoto, which can add signed provenance and attestation to simplify applying central concepts of governance and structure to upstream data.
The notion of a data contract or a data product spec is starting to become common in the financial sector says Das, defining it as a data set, or a group of data sets, bound together by a contract that defines expectations which aren’t just cosmetic but machine verifiable. They can also include operational SLOs for APIs as contracts describe not just the shape of the data but operational characteristics and guarantees.
Document graph markup language (DGML), a new open source specification from Docugami, promises provenance down to individual data points automatically extracted from documents.
“It’s critical to know the validity and provenance of the information your AI is relying on,” Docugami CEO and XML co-creator Jean Paoli says. “Establishing the validity of data right from the start, at scale, is vital and far more efficient than trying to clean up bad data later.” DGML combines semantic tags describing what content means in its business context with bounding boxes showing exactly where in the document the content comes from, with attestation to prove it.
AI demands provenance
All this context is the kind of metadata Anthropic’s context engineering guide recommends feeding to agents for accuracy. Developers are already used to giving coding agents more context, Das argues. “The same thing is happening with data, as when people realize when AI agents can’t make sense of what they’re doing, hallucinations happen,” he says.
Kumar agrees that organizations realize agents need context to provide better insights. “In many cases, it has to do with ensuring the existing data had clear semantics and relationships,” she says.
If you want to make sure the purchase return window an AI chatbot promises customers is based on your own policy, not a wish list from a user forum, you need rich context. It’s not just metadata. Organizations need to have semantics, data lineage, and ontologies. “Many are also building knowledge and ontology graphs,” adds Kumar. “By ensuring the systems understand what the data means, it’ll be able to provide a better response.”
And if you’re going to the expense of fine tuning, which needs relevant and domain- or task-specific examples, you don’t want noise, duplication, or irrelevant content in your data. You can, of course, exclude poor data if it’s annotated and verified earlier, but you can also improve model performance with extra information, Aronchick points out. “The augmentation of the existing data makes the data you pull out more valuable,” he says.
Expanso recently won an Edge AI award for fine tuning a base level model with only about 3,200 images by augmenting them with metadata. “The reason it worked on that few is because I could tell it deterministically what was in the frame,” he adds. “It’s labeling at the point of capture instead of paying somebody to label it later. What if I developed models for predictive analytics of store behavior on a per city, region, or country basis? If I’m able to take the raw point of sale information and augment it with additional metadata, I’m turning this into a much easier thing to fine tune.”
Or you might even avoid the expense of fine tuning entirely, suggests Das. “You get the short-term advantage by fine-tuning and getting great performance at much cheaper cost on a smaller model, and it gets stripped away in a couple of months as a new model shows up,” he says. “You have to always run that calculus of when’s the right threshold to fine tune an existing model, distil it, and then run it for a fair amount of time to recoup the costs of fine tuning.”
Although regulated or slow-moving industries will see benefits from fine tuning a model they can run for six to 12 months on data with higher quality and better provenance, many organizations may use the improved data quality to get good results without fine tuning.
“We’re taking a more knowledge graph-oriented approach to grounding the model, and betting on the fact that because the knowledge graph is changing often, it’s better to keep it as a runtime artifact than a baked-in one.”
Entrar
