DCL Learning Series

Marianne Calilhanna

Welcome to the DCL Learning Series. Today's webinar is titled "Before You Feed the Machine: Cleaning Content for AI Workflows." My name is Marianne Calilhanna and I'm the vice president of marketing at Data Conversion Laboratory. I'm really happy you're here and I'm very excited for this conversation. Before we begin, I'm going to take care of the usual housekeeping stuff. This webinar is being recorded and it will be available in the on-demand section of our website at dataconversionlaboratory.com. My colleague Leigh Anne is working behind the scenes. So if you have any issues with this webinar platform, any other questions outside of what we're going to be speaking about, just send a chat and she will be happy to sort you out. And of course we're going to try to save time at the end of this for questions, but please feel free to submit any question, comment, anything you want via the chat box or the question box here in this platform. All right. I am happy to introduce my colleague, Christopher Hill, who is here today. Chris and I have a long history of working together. Chris, why don't you tell everyone a little bit about yourself? 

Christopher Hill

Yeah, my name's Christopher Hill. I'm a technical product manager and project manager at DCL. So I manage a couple of projects mainly in research integrity, but I also manage one of our products that I'll highlight a little bit of – I think we'll have a tiny bit of time at the end to showcase. And that's a product that is actually relevant to today's conversation, called Harmonizer. My history: I go back a long ways in content management and technical publishing. I've been doing this kind of work since the early days of XML, when I used to teach XML courses and develop training courses around XML. So I've been doing this a while and it's been interesting to grow in this industry.

 

Marianne Calilhanna

Thanks, Chris. So, let's go on to the next slide please. 

Christopher Hill

Do you see it?

Marianne Calilhanna

Yes. Yeah. 

Christopher Hill

Good. 

Marianne Calilhanna

AI is only as good as the content it consumes. And content quality isn't a nice-to-have; it really, truly is the foundation that AI is built on. And we have some quotes here from some industry resources that I think are really relevant to what we're going to talk about today. You can see here on the first box that Anthropic tells us that how you structure your inputs directly shapes the quality of your outputs. And XML tagging isn't just a formatting choice. It's a signal to the model about what matters, how to interpret it, and that whole idea of "structure in, clarity out." And then that middle box is from the Association for Computing Machinery, who take it further into retrieval systems and when we start talking about RAG and thinking about RAG. So when your documents carry rich, structured metadata, your AI actually finds the right information at the right time.

4:00

And without that structure, that's where hallucinations can start to creep in. And then the SAS quote just puts a sharp point on the business stakes that we're talking about. You know, bad data doesn't just slow AI down. We know it kills AI projects and ultimately it can drive customers away. And then that's not a technical problem. That's a revenue problem. So whether we're talking here about prompt clarity, retrieval accuracy, or business outcomes, every thread leads back to the same place. And that's the quality and structure of your content. And that's what the concept of content debt puts at risk. 

Christopher Hill

Yeah. I find it interesting and I'm glad you highlight from some of the industry leaders this because I think a big thing I've noticed in AI discussions is people still, they kind of gloss over the need for quality data to feed the LLMs. The big headlines in large language models are these know-everything, consume-everything models. But in business practice, your LLM is going to be consuming a much smaller base of information, a much smaller content set. And in those smaller content sets, you can't always rely on volume of right answers overwhelming all the bad information that might be out there. So I think in the excitement about LLMs, it's easy to forget the basics of content management. 

Marianne Calilhanna

Yeah. And sometimes it's just not as –

Christopher Hill

Not as sexy. [Laughs]

Marianne Calilhanna

Right. AI is the shiny new tool, right? 

Christopher Hill

Exactly.

Marianne Calilhanna

Let's go to the next slide and share a little experiment that we did. As Chris and I were meeting and talking about this presentation, this webinar, we each opened an LLM on our independent laptops. I'm on one side of the country, Chris is on the other side of the country. And we both opened Claude and we asked the exact same question and you can see that question here. And that is: "What is the best format to store my technical documentation in?" And if you click to the next slide, we can see our responses. So we have Chris's response here on the left and mine on the right. Let's walk through these responses and I'm incredibly interested in your observation, Chris, from the technical side. 

Christopher Hill

Sure. So I was actually floored when I asked this question. I was not expecting the answer I got. So if you look at this screen here, you'll see that the machine, the LLM Claude told me there's no single best format and said it depends on what you're going to do and that's all fine and good. And then it got into some detail and it started to tell me what formats I should be looking at. And it started with markdown, which is kind of popular in certain areas of technical documentation. It's not my first choice as a professional in the kind of work I do, but it certainly is popular for READMEs and API references like it says here.

8:00

So that didn't surprise me too much. But then I started going down the list and it goes into restructured text, this .rst format, especially if I'm using Sphinx, which I have actually never heard of Sphinx. I haven't worked in the Python ecosystem projects world. So I was a little surprised at that. And then I get to this thing called AsciiDoc, which I also hadn't heard of, but since then I've read a little bit about it and it was interesting. But what really surprised me is nowhere in this answer did it mention XML. Nowhere in my answer did it mention data, which I go to data conferences several times a year. I've done that for decades. And those are the first things that I would list if I was providing this answer to somebody. So actually it was in doing that query that I asked Marianne to ask the same question because I thought, how on earth is Claude not surfacing what I consider the mainstream technical documentation formats? So, Marianne, why don't you share what happened when you then asked the question? 

Marianne Calilhanna

And I think the response that I received is the response I would have imagined to receive. If I asked any of my colleagues, and I think I stated at the beginning, I'm going to frame this, I am a marketing person. I'm not the technical person here, but I've lived and worked in this XML before that SGML world my entire career. So I'm fairly fluent in the world of markup languages, the different standards around XML. This response mentions – references JATS, which is the standard for scholarly publishing. Of course, DITA for tech docs. So if I didn't know or have an inclination of the answer and I received this from an LLM, I'd have a good path to explore on how I want to start thinking about storing and managing my technical content. 

Christopher Hill

I'm not sure I would be that lucky.

Marianne Calilhanna

Vast differences that we each experienced. 

Christopher Hill

Yeah. And that was a real surprise to me. I understand why it's happening and we'll talk a little bit about that. But I think I was really surprised at how dramatic these two differences are. I mean, not to even mention XML or DITA or JATS or any of these major standards in my answer was just, well, it was a little surprising.

Marianne Calilhanna

Yeah. And Chris, if you hit the next slide, because you can also see that my response is exactly what you see on the screen. It all fits into what we've captured. You had sort of a two-page response. And again, with all of that, there was no mention of XML, which was pretty surprising for two people who have worked in the structured content world for 30 plus years. 

Christopher Hill

Yeah. And I'm not going back to Word as my technical document standard. That's not my first recommendation. 

Marianne Calilhanna

Yeah. 

Christopher Hill

But it was on this list. It was on my list. 

Marianne Calilhanna

What do you think, Chris? Why are we getting such vastly different responses?

12:00

This is hard for people like us who work at DCL where we're one of the leading XML service providers in the industry. It's our bread and butter. It's deterministic programming. We do something and we expect the same answer, a right answer, a correct answer. And this is just –

Christopher Hill

There's a lot going on with these LLMs when they construct an answer. LLMs, it's pretty much a mystery and it's designed to be a mystery that the vendors have created of these black boxes that you ask a question and you get a magic answer and we kind of assume that that's a good answer. I would be very concerned if I had actually been asking this question for real and then spent a week trying to figure out what technical platform I was going to acquire or what software I was going to use or anything based on this answer. It would've led me down a lot of false pathways. I think when you think about this, when you're taking all the information in the world which these LLMs are based on, there's a lot of information about all these subjects out there on the web.

And so when you have lots of different information or conflicting information or options in the responses, LLMs try to narrow down an answer and give you an answer. And it will do that by both taking into account who's looking. So who's reading this? It does its best to give you a very solid answer that's going to be confident because it wants you to use the LLM more. And one of the things we have to think about when we're using AI is what is its goal? And you might think naively that AI has the goal of giving you the right answer. That is not the goal of AI. The goal of AI, at least as they're trained by the big companies, is to get you to use more AI. It's a corporate goal, not your goal. Now your goal of getting the right answer, presumably if the AI gives good answers, if it gives right answers, if it leads you to success, then you're going to use it more and it will achieve the goal of more use. But that's one of the side effects that go along with increasing user use of an AI machine.

So if you think of it in those terms, the AI has a lot of things it does where it tries to figure out who you are. It then tries to match up some information it thinks you're going to like. That's a part of its training. And then it's also going to add a little bit of randomness to the thing. So what it does is it tries to say "Okay, I've got all these potential answers. How am I going to come up with an answer? Well, I'll get rid of some of them and I'll sort of randomize the answer." And you'll see this at work if you go into your own AI, if you go into your favorite LLM and you type a question and then you type it again from a different context or at a different time, you might get a very different answer.

16:00

This is kind of an extreme example of that, but you can probably do the same thing just by checking today, asking a question and then asking the same question tomorrow and see how different the answers are. It's that randomization factor that's worked into an AI that can cause these sorts of differences. So one of the things to think about then is if the AI is having to make decisions between conflicting information, what is it going to do? It doesn't know which one's right. It's going to grab one and it's going to grab one maybe that's more common or it's going to grab one that maybe it just randomly grabs. And so that really leads us, I think, into the next subject.

Marianne Calilhanna

Yeah. And it's really important when we think about organizations starting to use AI, implement AI, maybe it's a chatbot on their website or they have all of their content, whatever that may be, maybe it's a journal's entire publication history, and then they want to put some kind of AI overlay on top of that. So it is somewhat of a closed – or it is a closed – system. That's where we're going to talk about this notion of content debt and what that means and what that entails and why it's important to address that for AI in these places and business. And so when we say "content debt," it's this accumulated weight of inconsistent, redundant, and poorly structured information. And it's that which silently undermines these AI initiatives before they even begin. Because AI systems, they really depend on these things you see listed here. They depend on clean data, consistent terminology, low redundancy, clear structure, and accurate metadata. And let's face it, most content collections are deeply fragmented, they're inconsistent and they are full of redundant text. 

Christopher Hill

And when you think about that, think of an LLM, think of if you were answering a question and you had a pile of documents. If those documents are fairly consistent and you read all those documents and you said, okay, now I can answer questions about this subject. If the documents are consistent, if they're relatively all along the same lines, you're probably going to give pretty good answers if that's how you were operating. But if I gave you a pile of documents that were inconsistent or that had lots of different subjects and didn't really point in one clear direction, then when you answer questions about that subject, you can imagine your answers are going to have that randomness, that perhaps even inconsistency in how you answer the questions because the data you're relying on is inconsistent and it's duplicated in slightly different ways or with different tones or sometimes with totally different objectives or conclusions. I think of the year I was born was – I'll give away my age – 1968, and the movie 2001: A Space Odyssey came out. And even then, what was the problem with the computer in 2001: A Space Odyssey? Well, the computer was given

20:00

conflicting goals. It was told to not tell the users about one thing and to have a certain mission objective privately and then to have a different objective to the users and the public. And what happened was very bad, of course, in 2001. We're hoping not to create that world in reality, but a mini version of that is going on when an LLM is given these conflicting that's not consistent.

Marianne Calilhanna

Yeah. Let's keep moving on if we can. We do have a bit to cover. So when organizations think about an AI initiative, a lot of what they see is what you see above that surface, the waterline. It's all that exciting stuff. It's what some of the C-suite likes to hear, automation, personalization, all these shiny new capabilities. And that's what gets budget and attention these days. But underneath are these things that really can undermine whether these projects succeed. And this is where this notion of content debt lingers at this accumulated weight of the inconsistent, poorly structured content that so many organizations have and that's been building up for years. And what this does is kind of creates these two categories of problems. Can you go back real quick? 

Christopher Hill

Oops, sorry about that.

Marianne Calilhanna

And these two categories can be put in these buckets of hidden costs. So like we've said, duplicate content, inconsistent terms, pieces of content that need updating in three different places instead of one. And that cost doesn't disappear when you add AI to the mix. It just gets buried deeper, it gets multiplied faster. And then the second bucket or issue revolves around accuracy and trust because messy, inconsistent data or content, it doesn't just cost more to maintain. It produces AI output that you can't trust. Inconsistent terminology confuses models in the same way it can confuse readers. So redundant conflicting content means the AI has no reliable source of truth from which to draw.

So before you feed the machine, you do have to know what's below that waterline. And that's what we're talking about today. So, Chris, do you want to go ahead and hit that next slide now? And so that stuff that's below the waterline, what do you do with it? And the instinct is often treated as a cleanup project. That might mean a one-time push to fix the mess before moving on to the real work of AI, but I think that's the wrong mental model. Content maintenance isn't a task you just check off before automation begins. It's really a prerequisite for any of this to work at all. So Chris, your thoughts on this, on building this trusted content foundation.

24:00

Anything you might add? 

Christopher Hill

Well, it's kind of funny. None of this is new. This is the same thing. I like what you said earlier, which was it's the same thing your readers wanted – was to be able to navigate and find and rely on the information that you give to them. The LLM needs the same thing. It's not any different than any other reader. The only difference is the LLM is going to confidently pretend to know stuff based on what you told it, which is through that training data and may not even tell you that it's making guesses or assumptions or totally hallucinating answers as it constructs those responses. And that again works in that stochastic way that these models work. These models work on a statistical calculation to basically predict what words should go into an answer based on statistical models that are trained from all of the data.

So whereas a reader, if I read a bunch of your documents and I saw a bunch of inconsistencies and then you asked me a question about it, I'd be probably willing to say "You know, Marianne wasn't very clear on the papers she gave me, so I'm not sure of this answer, but I think..." and then I could say something. LLMs don't do that. The way they work is they provide these statistical answers that just go through and say in a very straightforward way the answer. So in some ways, all those things, cleaning, standardizing, enriching, validating, and then governing your content, that becomes even more critical when you have an LLM as the audience for your content because the LLM is going to translate that with a lot more confidence than might be warranted.

 

Marianne Calilhanna

Yeah. And we've talked in the past too about versions. We've seen it ourselves. We have multiple versions of documents. We try to be as organized as possible, but you have multiple versions of information, and how can an LLM determine which supersedes the other? There's a lot there. I think the real thing to think about here is that content maintenance, it's just an ongoing strategy in an organization that's really critical. 

Christopher Hill

Absolutely. Yep. Yeah. And I think I've heard, especially in some management circles, they think, okay, and I read about this in the last year where we're going to lay off all the librarians or the technical publishers or writers or these people and have LLMs do everything. I think a lot of those projects that tried to do that wild jump are struggling with these very issues because again, the LLMs have to be trained on something that's true and reliable. And they're not yet to the place where LLMs are doing that work. They're not able to make those judgements yet. Now, maybe in the future we'll be in a world where the

28:00

LLMs will gain this super intelligence to be able to do this kind of thing. But today that is not where we are and people need to go in with their eyes open about if I put all my data into an LLM and then rely on it to answer my user's questions, you need to be very clear-eyed and realistic about the quality of the data you fed it.

Marianne Calilhanna

Yeah. Or if I'm implementing a rag environment and I'm going to have the LLM just look at all of what's in my repository, what if I have five different versions? Which really is the real source of truth? 

Christopher Hill

And you got to remember that LLMs, at least the mainstream work in large language models, is looking at the whole global universe of data and thinking that, okay, over the entire internet, there's going to be maybe more right answers about a lot of subjects than wrong answers. And I can use things like how popular the information is, what source is it coming from and those things to help rely on it. All those types of things don't come into play if you're focusing in on a subset of content within your organization. Within your organization, you might have some areas where you say, okay, this is more reliable information than maybe this other stuff that's in a Wiki or some other source, but it's a lot less of that capability of using the sheer volume to come to some truth. It really will augment the noise levels in your system as you narrow down the set of content that's being taken into consideration.

Marianne Calilhanna

So, let's share – at DCL, we really do try to make our webinars a learning series, but it would be remiss if we didn't share a little bit about a tool that we use quite often with our customers. And it's the product that Chris does manage. So Chris, why don't you talk about Harmonizer and how it addresses this content debt?

Christopher Hill

Sure. So we developed Harmonizer. Actually, it began its life even in the '90s or maybe even the late '80s. So in the last century. So it's been around a long time. 

Marianne Calilhanna

The 1900s. [Laughs]

Christopher Hill

That's right. It's been a tool that's evolved to basically take and analyze content and help you address that content debt. So the way it does that is it says "Okay, I'm going to look at content in general. I'm going to look at the text, the actual text in your content. And I'm going to compare all the text in a content set with all the other text in the content set." So this can be hundreds of Word documents or PDFs or a mix-and-match. You can throw in XML. You can really take any format that has text in it that's extractable and we can include that in a Harmonizer report. And the report does a many-to-many comparison of all the blocks of text with all the other blocks of text and then tries to group them together into match groups. And these match groups are used to help you find this content deck. So really the goal here is

32:00

it takes any of those formats, tries to find the duplicated content. It helps you develop a reuse strategy. So oftentimes knowing why is our content inconsistent? Even answering that question can be difficult. A Harmonizer report can put that kind of at your fingertips in one place and it allows you to then scan these match groups and get an idea of why things are differing. Is it just different language? Is it totally different information? Is it conflicting? We can look at all of those types of things. So really you can use it for a lot of different tasks in the content maintenance world, but I always think of it as kind of a way. It's a utility that you can use to help give you some tangible action items as far as content maintenance goes.

So how does it work? Well, it's pretty simple, at least in concept. Of course, there's a lot going on under the hood, but it runs on a pretty big server because what it does is it takes any content set that you want to feed it in whatever formats. It pulls out all that text. It uses a natural language processing algorithm to see how similar those blocks of text are. So it can do things like if maybe the phrasing is different, it may still find matches and group things together even though if I said before I went to the store, I put on my coat because it was raining. And then in another place I might write it was raining outside so I put on my coat before I went to the store. Those are two different ways of saying basically the same thing. Harmonizer would match those with a high confidence level even though the words are out of order and different. And that's using those natural language processing techniques. And then what it does is it boils all that down into a Harmonizer report. And I think the easiest way to talk about the Harmonizer report is to just look at one. So I'll pull up a Harmonizer report. This one was done with some PDFs of some Gateway 2000 computer manuals. So I needed something that I wouldn't get sued for using. I think we haven't been using Gateway computers for quite some time, I don't think.

So these old manuals were all compared together. And in this case, these were PDFs, so there's a little bit of noise in them, but you'll get the gist of what this is doing. So what I did is I fed it several Gateway manuals and I can see which ones I fed it here on the report information page. So I fed it these seven Gateway manuals and it pulled out blocks of text and it tells you how many blocks it pulled out. And then it tells me how many fell into an exact match, a close match or were unique. So it categorized every block of text as an exact or a closer unique match. And then it summarized it here for me. So I can see that in those seven Gateway manuals, 74% of the content was exactly the same or at least had an exact match. So that's a huge amount of content that was the same in these computer manuals. And that's probably not terribly surprising if the nature of computer manuals. Typically, if they're relatively manufactured in the same generation,

36:00

they're going to have a lot of the same information about the computer. And then you'll see that about 15% of that content in those manuals was a close match. So they differed in some way. And then 11% was totally unique to the manual. o there were blocks of text that occurred in exactly one manual in exactly one place and was never repeated. So this is kind of the overview of what the report tells me. So I can then go in and look and I'll open this matches page of the report here. And here you'll see that on the left when this opens, there we go, you can see all the blocks of text that were extracted from the PDFs. So there's that first PDF and then there's additional PDFs down there and you'll see this is all the text that was pulled out in order. And then that text is put together into these match groups. So if I look at some of these manuals, you'll see on the right the first match group here is probably a title. In this case, a title of the – it just says "Front."

And I ran this report with a minimum word threshold of one. So anything gets included, including a single label. So this is probably the label on a diagram of the computer. And actually if I go open this, you'd see it was that. This is all labeling all the parts of the computer. And then there is some information here like the battery charge indicator, which tells me different ways that it talks about the battery charge indicator. It says that the LED shows the battery activity and status. As we go through the report, you'll see that some of these things are grouped together in complex ways. So here's one that tells you when the LED lights up on the different computers. And you'll see that on the Gateway 305, 350 and 675, it says the LED only lights up when your notebook is connected to AC power or the battery charge is very low. So the LED is going to light up in those two situations for these three.

For these others, it chopped off the very low part. I should have used a different example there, but on this last one, you'll see that this one actually has more information. They included seafront on page 10. So they included a reference on the 520 manual to point me to the information. You can see the different ways the LED works in these manuals by going to the sections about the actual color of the LED. So here Harmonizer said "Okay, these are all similar within a certain percentage of similarity. The only variation is the word green off or blue. And here are the places where it used green. Here are the places where it said off, and here are the places where it said blue." 

Marianne Calilhanna

So different versions of the Gateway had a different LED color for when the battery was fully charged. 

Christopher Hill

Exactly. And this would also be conveniently a way that I could quickly check and make sure that my manuals are accurate. If I was the subject matter expert and I already knew what the LED did on all these machines, I could check. Or if I'm a tester, I could actually look at this and say "Okay, let's make sure that the LED is green when it's fully charged on these three models. And then on this model, make sure it turns off and make sure it turns blue on the 675." 

40:00

Marianne Calilhanna

Chris, in the interest of being able to get everyone back to work, we do have some questions coming through.

Christopher Hill

Yeah. Let's entertain those.

Marianne Calilhanna

Let's focus on some questions because if we go back to the slides, we did want to offer anyone who would be interested in seeing this with a sample set of your own content. Chris is going to offer to run a Harmonizer report because we do find that when you look at this with your own content, it's going to be very meaningful to you. your content, right? 

Christopher Hill

Exactly.

Marianne Calilhanna

You can scan that QR code, or Leigh Anne's going to also put the URL to where you can request a personal demo. So we've got a few questions we're going to try to get through. If we don't hit them all, we'll be in touch after this. But one question is on that content maintenance slide, how do you enrich your content with metadata? Could AI add the metadata too or is that too risky? And no one likes the answer.

Marianne Calilhanna

It depends; it depends. 

Christopher Hill

That's probably what we're going to say, right?

Marianne Calilhanna

So at DCL, we work in the world of professional production. We work in a world where there is a lot of throughput. So to have an AI analyze your content and then get source files and input metadata, that would be convoluted. And I think it would also be there's a QA in whole component there that probably wouldn't be addressed or would need some engineering to really address and make sure that you can trust those results.

Christopher Hill

Yeah. So like any project, so I worked in one of my previous jobs, I worked on semantic enrichment, which actually used a different kind of AI, not an LLM, but to create metadata where there wasn't any. And it did that by looking at words and word clouds and those types of more deterministic approaches to artificial intelligence. And when you add metadata, you can do that with AI tools and they can be very effective at that, but you have to validate the approach and make sure that it works with the model you're using, with the content that you're enriching. And the only way to do that is to do it in a context where you have, as Marianne said, a QA process involved. So you have to be able to sample your data to do some sample enrichment that is then checked by a person or by an expert. And then once you're confident in that, you can scale that up using an AI tool. That tends to be the safer approach. You could just throw it in there and say "Add metadata to all of this," but whether or not your prompts were clear enough or the data was sufficient enough for it to do that well would be a total unknown to you if you didn't go through that formal process.

Marianne Calilhanna

Yeah. Another question came in, how can you validate data for accuracy in a fast and efficient manner? And that, too – I think there's a lot of ways to interpret that question.

44:00

When I read something like that, I think about making sure that you use industry standards. So depending on – we mentioned that first query, my response from Claude referenced some industry standard XML schemas, DTDs, JATS in the scholarly publishing world, DITA in tech doc world. So when you have that content in a structured XML format, that inherently does some validations. Do I have my authors properly cited in reference and citation elements? So structuring content itself in XML is a way of validation.

Christopher Hill

And the XML or having a structured format underneath also allows you to produce reports on that information. So one of the things we do in our projects is we'll have, I've got some high volume research integrity projects that generate an enormous amount of data. We boil those down into very consumable reports for validation. So every periodically we'll check to make sure that we're remaining accurate in our assessments. To maximize the efficiency of that validation, we put together these very quick to go down reports doesn't have to open hundreds of files or go through hundreds of reports. We can automate all of that validation process. I won't get into it in too much detail, but that's a big part of some of the projects that we do is building in those validation processes that make validation more efficient.

Marianne Calilhanna

Well, I do want to be conscious of everyone's time. We are so thankful and grateful that you took time out of your day to spend with Chris and I. And so thank you for attending this webinar. We've come to the 45-minute cap. So just to close it out, I do want to share that the DCL learning series comprises webinars like this. We have a monthly newsletter, our blog, and you can access many other webinars related to content structure, XML standards and more in the on-demand section of our website at dataconversionlaboratory.com. We hope to see you at future webinars and have a great rest of your day. 

Christopher Hill

Thank you.

Marianne Calilhanna

Thank you.