Here be fossils: AI and knowledge management

This blog may be bad, but it's going to be human. It's also going to be better for having thought about AI. (Which is a little inevitable as it's about AI).

I think, and have long thought, that language models being 'smart' is the most amazing feat of humanity. By speaking and writing, we encode knowledge in words, and by virtue of statistically replicating speech we have also replicated the know-how entombed within it, like an intellectual fossil.

But if that somewhat romantic view of LLMs isn't for you, here's a far colder one: LLMs only do two things well, knowledge management and knowledge delivery. Everything else about their use follows from there.

The Verge's Nilay Patel often asks 'where is the killer consumer AI product'. I think it'll be a long time coming, because most people, in their day-to-day non-work lives, don't need much knowledge management. Where they do (most often it's parents) the main improvements would just be a combination of a good calendar and good address book. (The other main sell of consumer AI, 'smart glasses' reminding you of peoples' names... you got it, that's knowledge management too).

It's a big blocker to consumer AI and AI in the arts that people are more than just knowledge management systems. But recognising what tasks are mainly made up of knowledge management helps to understand what it can be helpful for.

Coding, the clear non-consumer success story of communication model AI so far, is a near-perfect knowledge management job. Not only are the systems, which provide the right roads for bytes to travel down, a form of abstract knowledge, but a far greater amount of information is recorded in one place than for many other tasks or jobs.

Many laptop jobs have the 'knowledge' required for them scattered farer and wider than code. A codebase is, by nature, self-documenting. It's even, quite often, double-documenting, with 'human-readable' comments and docstrings aimed at the developers who (used to) write and maintain it. Some of that documentation provides knowledge not encoded in the code too, about other systems or business objectives. So, when code is a form of language, and a codebase contains so much knowledge, it's no surprise LLMs worked wonders there first.

Meanwhile, LLMs have (against the odds) made a lot of inroads in digital visual design too. On the face of it that's a little confusing, but it's just another case of knowledge encoded in text: only this time it's things like 'what good spacing looks like' being encoded in a million `<padding>` tags.

As much as we may associate the visual with the artists, user interface design is heavily a job of knowledge delivery. This is what our company does, this is how to navigate our app, that is how you sign up to subscribe, (that, in a disused lavatory in the cellar, is how you unsubscribe). It may start off with designers creating cathedrals of pixels, but then you run into legibility, internationalisation, general accessibility, SEO (or AIO), the variable boundaries of developer talents/tempers.

There is a question of taste in all this, which most often rears its head in design but is also present in code as well. Taste isn't purely about knowledge management, but it does course through it. The way that Christopher Nolan chooses to structure his films' narratives, for example, is both a matter of taste and a matter of controlling the flow of information to the viewer.

However, when you use language models, the output from them flows through the 'taste' that has been baked into the model. This is where you get the AI tell-tale signs, the em-dashes, the 'this not that', the "you're right to push back on that".

I find it helpful to think about language models as having a gravity. There are certain areas where the gravity is stronger: generally around guardrails, but also where the model company believes they're trying to be helpful. And I do think that, nowadays, the base level 'taste' of these models is not that bad. Annoying, sure; ubiquitous, absolutely; but it raises the floor for a lot of users. There's a reason for the gravity to exist.

But I think gravity is also a useful metaphor because, to break free of it, you need a certain amount of momentum and power, and that mirrors the way that you're less likely to shed Claude-speak if you don't have expertise at a particular task.

For example, Impeccable is a plug-in to 'detect design slop' and turn your AI designs into something less slop-y; their folder of reference documents has over 30 markdown files for agents to make use of, and that's only part of their system. Not all of the documents are long, but most are several hundred words, collectively stretching to several thousand in total. This is the kind of effort required to help non-designers reliably and routinely break free of AI design gravity.

Clearly, it is possible to break out of AI gravity, but it usually requires pre-existing expertise. If you know the terms for certain code conventions, and write your style guides with focus and good prioritisation, there is much more chance the code-gen bots will follow them. I suspect that linguists would manipulate AI writing (i.e. make it good) easier than most other people.

Of course, the question is whether they'd want to. For wholly understandable reasons, the kinds of people drawn to mastering the components of language may not be the kinds of people interested in models created by light-fingered hoovering of as much language as possible.

Yet, people gotta work. And so we come back to knowledge management. Coding is not just a success story because a codebase is a very strange but thorough form of information storage, it's also an area where LLMs have been successful because it makes up so much of the modern economy. While many people may build software for fun, very few people code for fun.

The practice of essay-writing, for example, is almost the reverse of that situation. Not only that, but there's a question as to whether knowledge delivery is even the point of the work. I'm not just talking about taste. A key motivation for essay writing -- which has parallels in journalistic reporting too -- is the knowledge management inside the writer's brain.

You could argue that this makes writing a selfish act, but nonetheless it's the truth. There's fair reason for it to be self-serving, though, as a human's ability to think intellectually can only increase with practice, with use, something that isn't the same for tapping into an LLM's API. At this stage in AI progress, my use of Claude is not going to make Claude smarter; but my writing of this essay myself will make me smarter.

Yet surely only the hard-line purist writers would deny that there are boring, rote parts of the writing process. Transcription, storage of information, verifying that extracts, quotes, and passages are cited in research files or a finished document: all of these are things which take up time and, I'd argue, provide minimal value to the writer themselves. Note that I didn't say 'research' itself, an exclusion that even has me wondering whether 'transcription' should be on this list.

The act of 'manual' research feels similar to the act of hand-writing notes in class. Although taking notes on a laptop, or merely listening to a lecture, may appear to do the same job, being closer, more proximate, more literally hands-on with the information seems to help retain it better. And, as a rash of programmers experiencing 'cognitive debt' will tell you, your grasp of things can quickly atrophy when you're off-loading knowledge management.

Writers and artists might be interested in hearing from the software engineering community on this.

A long-established term, 'technical debt' describes maintenance or upgrades that should ideally be done on a system but are held off on to prioritise other things. Some debt is fine (like responsibly using a credit card is perfectly fine), but too much risks Bad Things happening, particularly as debt can cause avalanching problems if part of it is called in and can't be paid.

'Cognitive debt' is a riff on the term, for when developers don't fully understand the software system. Again, some cognitive debt is fine (like how most of us use day-to-day electronics without understanding how they work), but too much...

Cognitive debt is at play when somebody uses AI to produce a report but has not read it, and crumbles when asked about it. The use of AI is the spending on the credit card, and the immediate follow-up from a colleague or teacher is the debt being called on to be paid. If the amount of thinking that's been off-loaded to the AI is too large, the cognitive debt too big, well, you're in bailiffs territory.

And perhaps this is yet another reason why LLMs have been successful in the domain of coding and are still far clunkier elsewhere. In software, the knowledge management and knowledge delivery is sort of self-contained. The code itself does not engage with other people, it produces a set of instructions for a product, which is the thing people actually engage with. But in your other office jobs, or in your day-to-day carbon-based human lives, the person is responsible for going to meetings, holding negotiations, synthesising reports and competitor analysis. Cognitive debt is more immediately called in to be paid.

I've tried out using LLMs 'for writing'. Not the writing itself, because the default tone of voice hits me like cutlery on crockery, and I have no need for spending time in making it more palatable. But its use in research and planning was appealing to me: trawling through the crumbling empire of the open web isn't something that feels worth spending my time on; and with the right data structures to follow, I could (tell Claude to) turn this research and my transcribed voice notes into the skeleton of an essay.

It started off well. I may have had to tweak its instructions for citations, after finding some absent or inaccurate links when checking up on the research, but it didn't do as badly on that front as you might fear. However, as the plan grew in length it became harder to manage. And the cognitive debt that I'd slowly been building up in relation to the research made it harder for me to play with the structure.

Turns out that if you want to play with the structure of an essay, you need to have a good grasp of all the components that will go into it. Not dissimilar, coincidentally, to how you need to have a good grasp of a codebase to manage a refactor of it.

Large language models are an incredible feat of human achievement, but like stealing fire from the gods they bring a curse along with them. They're fantastic for knowledge management and knowledge delivery, but will you need that knowledge in the future? Do you have a means, do you have a practice, of gaining that knowledge further down the line if you offload it in the here and now?

For it to be safe to use a friendly omniscient AI-bot to help you, you probably need the answers to those questions to either be 'No' and 'Who Cares' or 'Yes' and 'Yes'.

There's a metaphor people like about using AI as a student: it's bringing a forklift to the gym. Sure, it helps you shift weights, but you don't get stronger. There's something in this, on multiple levels. Firstly, it is a different way of describing the same effects that the concept of 'cognitive debt' tries to capture.

Secondly: there are real uses for forklifts! (And it's indicative of the extent to which students experience the education system in a transactional manner, as a prelude to the jobs market and, by proxy, being able to support themselves in their adult lives).

But AI can also be a gym. By taking on certain jobs, it can give you more reps at other things. I've had far more experience of project management and software design (both visual and code structure) than I'd otherwise have had without it. An adapted version of my Claude research assistance has helped me read far more than I'd otherwise have done, by taking on and taking out time spent on online searches, basic summaries, and thematic grouping of work: which I can then go and read and take notes on myself.

The important thing in all of this has been to stay conscious of my grip of things. Would I be able to explain what's going on here? Would I be able to dive into the relevant part of the work quickly and easily if needed? How much cognitive debt have I built up?

What, out of the things I'm doing, is knowledge management? What is knowledge delivery? How much of each will I need to engage with in future? Where is the knowledge stored? What is the output I want to produce, and what are the skills I want to strengthen?

They say that using AI is like bringing a forklift to the gym. You need a forklift to build the gym in the first place.