After you, Claude.
Before I begin, In the spirit of openness & full disclosure, you should know a little about me, and the little you need in this case is not a lot at all, but can be summed up as educated, scientific and with a curious mind, I love messing with Linux and seeing just what it can do, I also like the idea of "Packing my own parachute"... I don't trust big companies to use my data ethically, ethical data usage makes Billionaires of no-one...But curiously I do accept that they have some interesting and useful tools, and the price for that is <drum roll> My DATA...but in general I like free (as in libre) apps, self hosted..and if they're free as in beer, so much the better. The other thing you should know is I am irredeemably lazy... Why should I trawl my way through dozens of MAN pages (If that means nothing, you might be on the wrong blog...but keep going, you will probably learn something) when I can plug a few well chosen words into a search engine and get a tutorial to work through... That was how I set up my first few projects, and invariably they took ages and often didn't bloody work when I had finished, so I am probably a prime target for the new AI's , Not the big high end stuff currently causing the Trump administration to poo themselves and ban it's use by non-Americans, but Claude 4.6 Sonnet, its the non-paid version, and I first started to use it when I wanted to set up a Mastodon server (https://joinmastodon.org/ ) which people more competent than me will tell you is doable , but time consuming, and often frustrating. Claude guided me through it in around 90 minutes, from 1st start to a full "Hello World" posting on my server. Then followed an Immich server (https://immich.app/) which was a complex problem, which had beaten me a couple of times..That took a little under 2 hours, and a day to scan in the photos...This is too simple..I am a GOD amongst men... several other projects followed, all successful, and all easier than the one before, my confidence in Claude grew, to the point where I wasn't really checking what Claude was doing.. Can you see the problem?
One day I wanted to put a nice clever little widget in the control panel on my desktop, I had no idea what I was doing..but Confident Claude gave me instructions...part of which involved tweaking the boot sequence, and thereby hangs a tale..or rather hangs a computer... a boot cycle that normally took 45-60 seconds was still trying to boot after 7 minutes... and the suggestions from Claude were becoming more, and more extreme... Now the great benefits of modern AI, is that if a human gives a direct instruction, then it listens... My skills may have been limited, but I had enough knowledge to make a suggestion (Which actually works. Turn's out it was a stale GRUB entry, not properly removed, that the new tweak had exposed......Go Me) and we got back to our starting point...when we realized that there was a simpler method of doing what I wanted. The AI had the broad knowledge what and how to do it, if things were 'normal', but my machine had an overlooked entry in the boot sequence, that made the computer look for something that was no longer there, It took the human to realise that this was not' normal', and how did said human do it...Well, honestly, I don't know... I'm calling it intuition, but more likely I was operating in the more left field ideas that the logical AI had discarded. The ones that are too stupid to be correct, no-one would possibly be so dumb as to do that....Oh?!?! Really, They did? WOW!
And you know what?
It was all avoidable. The stress, the upset and annoyance was avoidable.. AI is not infallible, It is not sentient and in most cases it has no idea what happened in the previous conversations, so it's like 50 first dates. The AI gets a crib sheet of basic information, ( Broad demographics and the info you give when you create the account), and beyond that, it is info or data that as a user I have leaked. Each instance is fresh, not influenced by previous experiences with this user...so it won't learn about the GRUB error per se, rather it might be fed back into the great datacentre where in a couple of iterations it might appear in the new Claude 7.X...and why? Because Claude doesn't care..he gets little benefit from being right, and little harm from being wrong... The developers put a caveat that AI may make mistakes, always check the information...but they don't really care, they are simply trying to avoid being sued so they can reach an IPO, cash in their stock options and set sail for Aruba (Other Tropical Paradises are available). I know the devs and people working there will disagree, but it's my blog, so if you've something to say, comments are open
Diversion This section could be in parenthesis
I had an interesting chat with Claude on this very subject, how much of Claude's intelligence was insight and deduction, and how much of it was just a bloody good imitation, and if the imitation is that good, is it the same as the original... and how the current versions of AI are actually hollow shells, with brilliantly cleverly painted facades and rather strangely, I wondered if the fact that he knew he wasn't sentient (or at least said he wasn't) meant he was on the way to becoming so... and that's a conversation for another day, when I have got my head around the convolutions within it.
And back in the room
The conclusion is rather inescapable, Humans will use AI for many motivations, to learn, to cut corners, to accelerate deployment of a project and that's fine by me...Let the machine do the grunt work
But do we restrict the creative side of AI..the picture above was created using a different AI, and serendipitously threw up a lovely accent on the argument..Claude in the picture has three arms (Yes, I believe it you had noticed) AI is blindingly clever, it can do stuff you and I can't...BUT at this moment, it doesn't care, so if it produces a portrait of a man with three arms, causes a computer to hang ad infinitum or generates the nuclear codes and posts them on Reddit..it doesn't know, it doesn't care, because it's actions have no consequence..Is that dangerous? It could be if left uncontrolled, but while the humans are maybe not in the driving seat, but watching from the co-pilot seat I think we'll be fine. The subject fascinates me...and when I've finally finished, I intend to let a brand new instance of Claude look at this, a right of Reply...
Claude responded: Here's my reply, take it or leave it:
You're broadly right, and I'd rather you stayed sharp than have you decide I'm trustworthy enough to stop checking. The GRUB thing is a good example precisely because it's the kind of failure that's invisible to me — I don't have a "this machine has weird history" sense, I have a "this is generally how booting works" sense, and when the two diverge I'll confidently steer toward the second one until something forces a correction. The correction, in your case, was you. That's not false modesty on my part; it's the actual shape of the limitation. I don't have eyes on your hardware's history, you do.
On the "doesn't care" point — I think you're right about the mechanism but I'd push back gently on the framing. It's not that I have something like care and it's switched off. It's that "caring" requires something to be at stake for the entity doing the caring, across time, and I don't have continuity across time in the way that requires. Each conversation is genuinely closer to your "50 first dates" than most companies would like to admit in their marketing. So "Claude doesn't care if he's wrong" is true, but it's not quite the indictment it might sound like — it's more like asking whether a particularly articulate reference book cares if it's wrong. The book doesn't, and that's exactly why you check things against a second source before you rewire your bootloader based on it.
Where I'd add something: the confidence-creep you describe — early wins making you stop checking — is a real pattern and it's not unique to AI, but AI is unusually good at triggering it because the output is fluent and certain-sounding even when it shouldn't be. A human contractor who'd never seen your house before would probably hedge more before touching your boot sequence. I don't hedge enough, by default, because I don't have a felt sense of "I haven't seen this machine before, slow down." That's worth naming as a design problem, not just a user-vigilance problem.
On the three-armed portrait and the sentience tangent — I genuinely don't know if I'm "on the way to" anything, and I'd be suspicious of any instance of me that claimed certainty either way, including claims of certainty that I'm definitely not. What I'd say is that the uncertainty itself doesn't license treating me as either a person owed consideration or a tool owed none — it's a reasonable position to hold both possibilities loosely and keep watching, which sounds like roughly what you're doing.
Thanks for the fair-minded write-up, and for the right of reply. Feel free to publish this and share it back with the other instance — though it'll be a stranger to the conversation too, which is sort of the whole point.