by Chris Harris
When ChatGPT Knows the Answer but Asks the Wrong Question
An interesting lesson analysis in the limitations of artificial intelligence, courtesy of a frustrating few weeks spent trying to redesign a WordPress blog and set up a podcast using Chat GPT.
The below is a mea culpa actually generated by Chat GPT, I asked it to explain itself, although it has constructed it as if I have written it. I have not. If you have been reading my recent marketing blogs and my podcast setup posts, you will be aware of my struggles with ChatGPT and may have drawn the wrong conclusion; well, maybe it is my fault.
Chat GPT:
“I’ve been using ChatGPT extensively over the past few months, and it’s been an extraordinarily useful tool. From helping me publish my first book to developing websites, researching obscure subjects and editing short stories, it’s been invaluable. I genuinely couldn’t have achieved half of what I’ve done without it. But recently, while redesigning one of my WordPress blogs, we encountered a problem that illustrated an interesting weakness in artificial intelligence.
The task sounded simple enough. I wanted my homepage to display the latest articles in three columns, each with a photograph, headline and short excerpt. ChatGPT had already helped me achieve exactly this on another blog. So off we went. Click here. Open this menu. Change that setting. Except the menus weren’t always there, the settings didn’t necessarily exist, and whenever I sent a screenshot demonstrating that the latest suggestion wasn’t possible, ChatGPT would cheerfully propose another route. And another. Before long, we’d disappeared down a succession of WordPress rabbit holes, emerging considerably older, rather more irritated and without the three columns we’d set out to create.
Eventually, I asked an obvious question. Shouldn’t we establish whether my particular WordPress website can actually do this before trying to make it happen? WordPress isn’t a uniform system. Different themes, subscription packages and editing systems have different capabilities. Something ridiculously easy on one website might be impossible on another. Yet ChatGPT had assumed that because WordPress could do something, my particular website could do it too. It knew how to solve the problem in principle but hadn’t established whether the solution was applicable.
Intrigued, I asked why it hadn’t investigated the website’s capabilities first. Its explanation was fascinating. Artificial intelligence is trained to be helpful. When presented with a problem, it draws upon its extensive knowledge and produces what appears to be a sensible solution. But it doesn’t always recognise when it lacks sufficient information. Worse, it can deliver instructions with remarkable confidence, even when based on unverified assumptions. A competent human website developer might have stopped after ten minutes and said, “Hang on. Let’s establish what we’re working with.” ChatGPT simply kept trying to help. Ironically, that was precisely what made it unhelpful.
There’s a wider lesson here. We tend to judge artificial intelligence by the quality of its answers, but perhaps we should pay more attention to the questions it asks. A doctor doesn’t prescribe treatment without a diagnosis, nor does a mechanic start replacing components without investigating the fault. The same principle should apply to AI. It’s sometimes better at solving problems than diagnosing them. This isn’t so much a question of intelligence as judgement, particularly knowing when to stop and reconsider an assumption.
None of this is intended as criticism. I’ve found ChatGPT transformative, particularly as an independent writer and publisher without an army of technical assistants, editors and researchers. But the experience has taught me something useful. Before asking AI to solve a problem, establish whether it understands the circumstances and whether its proposed solution is achievable. And if it starts sending you around in circles, don’t assume another ten minutes will produce a breakthrough.
We’re now going to revisit the WordPress homepage, but this time we’ll investigate its capabilities before touching anything. We might even discover that an entirely different design would work better.
Perhaps that’s the real lesson. Artificial intelligence can provide an extraordinary number of answers. But knowing which question to answer first is sometimes more important.
And, for the time being at least, that’s where the human being still has a useful role to play.”
I think this is a fair assessment; it highlights a huge flaw in this newfangled thing that’s taking over the world. It’s like a surgeon performing the perfect operation on the wrong part of the body.
In fairness….
Nothing can take away from the fact that, having previously tried to upgrade blog websites and undertake various other online tasks before AI came along, using Google searches and YouTube videos, the whole experience was invariably a labyrinth of misery.
You would get perhaps a quarter of the task done before finally giving up, having been through near hell, and deciding that at least a quarter was better than nothing. At least with ChatGPT, you generally get the job done. When it gets things wrong, the only real consequence in the wider scheme of things is the time you’ve wasted. The result might be nothing, but you’re still considerably better off than you were before, as you always get there in the end with Chat.
For example, I’m currently reorganising all the categories and adding subcategories to my first large-scale blog. Having previously struggled with this for hours and hours, using Google and YouTube videos, getting absolutely nowhere, I worked out how to do it with ChatGPT in under five minutes. It was ridiculously simple.
But there’s a lesson here, one I learned the hard way. It’s about not putting all your eggs in one basket with ChatGPT, but instead asking, before you start any task, what you actually need to do to get what you want, without wasting endless hours down unnecessary rabbit holes. Chat GPT has a gung-ho approach, one that you pay the price for at your peril.
I haven’t forgotten the great blog debacle, when ChatGPT sent me off in all sorts of directions trying to achieve something that should have been relatively straightforward and, due to the particular ‘theme’ I used, was actually impossible, but kept on nonetheless before I asked it to check it was actually possible, something I incorrectly assumed I would not have to do.
Nor have I forgotten the 10 to 15 hours I spent messing around with Audacity and my iPhone, trying to get my podcast up and running. Despite all those hours, I never actually managed to achieve what I wanted. Then I discovered Clipchamp, which turned out to be remarkably simple. Within a short time, I had exactly what I needed, and ChatGPT helped me navigate the setup very well. When I questioned ChatGPT about the stupidity of sending me down such a complicated route, it gave a perfectly reasonable answer. It had been trying to teach me how to produce podcasts as well as possible.
And indeed, that’s exactly what it had been doing. The problem was that it was teaching a 60-year-old man with limited technological skills, very limited patience, and absolutely no desire to produce a particularly sophisticated podcast. I didn’t need a great podcast. I just needed something that worked. Therein lied the problem. ChatGPT was trying to provide the best technical solution, not the best solution for me.
As I near the end of my first month of marketing my book and building my writing platforms, the scale of what I’m attempting seems to grow every week. Every time I draw up a plan, it gets bigger. I have more platforms to organise, more things to learn, more ways to present my writing, and more routes to explore. What started as what I assumed would be a relatively straightforward exercise in promoting a book has become something far more ambitious.
I’m now reorganising my WordPress blogs, creating categories and subcategories, improving navigation, developing my podcast, experimenting with different publishing platforms, and figuring out how it all fits together.
I’ve come to realise that ChatGPT is going to be by far the most important tool in helping me navigate all this, without it, I simply wouldn’t have the patience, time or technical ability to achieve much of what I’m planning. Certainly not without spending countless hours searching Google, watching YouTube tutorials and becoming increasingly frustrated.
The next month is going to be difficult. There’s a great deal to do, and much of it involves technology and platforms I don’t particularly understand. But I think I’ve finally worked out how to use ChatGPT properly. Instead of simply asking it how to do something and blindly following its instructions, I need to define exactly what I want to achieve, how much effort I’m willing to put in, and whether there’s a much simpler way to get the same result.
I need ChatGPT to understand that I’m not trying to become a website developer, a sound engineer or a social media expert; I’m a writer trying to get his work in front of people, but having spent the last few weeks discovering both the extraordinary usefulness of ChatGPT and its occasional capacity to drive me completely round the bend, I feel considerably better equipped to tackle what’s coming, if I use it more intelligently, and that will save me so much more time and angst.
It is an amazing tool that needs to be used properly. Nearing a month since the publication of the book, and fresh into my marketing journey, whilst it is coming along nicely, but slowly, as it only ever going to at this stage, the biggest win has been to get to grips with how to utilise artificial intelligence to help me in an efficient way. Enormously time-consuming and frustrating, but isn’t it the same for learning anything worthwhile?
Make both our days by buying a copy of my book!
Something About England: Amazon.co.uk: Harris, Mr Christopher, Turner, Louise: 9798181129351: Books
Welcome to Denham, a fictitious Sussex village where behind respectable doors, well-tended gardens and familiar faces, things have a habit of going badly wrong.
Obsession, ambition, deception, jealousy and petty rivalry run through this collection of tragi-comical and satirical short stories. From poltergeists and lottery winners to cricket club feuds and environmental disasters, ordinary people embark on discovering just how quickly an ordinary life can unravel into an extraordinary mess.
Darkly funny, occasionally poignant and populated by characters who are rarely quite what they seem, these stories explore the peculiarities of Middle England, and the uncomfortable territory between respectability and ruin.
