Why You Can't Just Use ChatGPT & Co. to Create E2E Tests

Georg Dörgeloh May 20, 2025

As a business analyst, requirements engineer, or product owner, you’re probably familiar with this situation: You describe what seems like a straightforward test case and think, “This can’t be that difficult… just click here, enter something there, and check the result.” Then comes sprint planning, where you discover that test automation will take far more development effort than expected - and the implementation gets cut or drastically simplified.

Then ChatGPT burst onto the scene. Suddenly everyone thought, “AI will fix everything!” Finally, we wouldn’t have to wrestle with flaky tests and endless debugging loops.

But as with most things that sound too good to be true, the devil is in the details. In this article, I’ll show you why current AI tools aren’t yet the solution for your test automation - and what you can do about it anyway.

The Dream: “ChatGPT, Write Me a Test!”

Imagine you could simply say to ChatGPT or Claude: “Create a test that checks whether a user can log in and update their profile” - and get a working test back. No more communication barriers between you and the developers. “I write my tests myself now. And they test what I actually mean.”

The Reality: Unfortunately, this doesn’t work reliably yet. Why?

Problem 1: The AI Can’t See Your Application

ChatGPT can generate test code for you, but the AI has no idea what your application actually looks like. It works blindly and has to guess which buttons are where and what they’re called.

It’s like trying to explain to someone over the phone how to assemble complex IKEA furniture - without them being able to see the instructions or the parts.

Problem 2: Every Application is Different

Even if the AI writes perfect code, it only fits a hypothetical standard application. Your real application has different button names, different field labels, and a different structure.

The Temptation: “Let the AI Test Directly!”

The next logical thought: Why not let the AI control the browser directly? You describe the test case in natural language, and the AI executes it independently.

This actually works! Modern AI models can analyze screenshots and navigate complex websites. Tools like Claude Desktop with Playwright MCP or similar solutions can navigate impressively well through the application and do exactly what you describe.

But: The Hidden Costs

What initially looks like the perfect solution has a crucial catch:

• Every test run is expensive: Each test requires dozens of AI calls that quickly add up
• Every test run is slow: What normally takes seconds can take to minutes
• Tests are unpredictable: The AI might approach the same test differently each time
• Scaling impossible: Imagine wanting to run 50 tests in your CI pipeline

It’s like hiring a brilliant but painfully slow and expensive QA employee who starts from scratch with every test and never learns from experience.

What You Should Know as a Domain Expert

Your Expertise is the Key

Even though the technology isn’t perfect yet: You as a domain expert are indispensable. You know the business requirements, understand the business processes, and can define the best test scenarios. No AI can replace that.

The Best Approach: Hybrid Solution

The most promising solution combines your domain expertise with AI support:

1. You describe the test in natural language (you do this best)
2. AI executes the test once and “learns” the application in the process
3. The system automatically generates stable test code for the future
4. Generated tests can be executed repeatedly reliably, reproducibly and quickly
5. When changes occur you can have the test regenerated without having to program

What to Look for in Tools

If you’re evaluating AI-based test tools, pay attention to these points:

• User-friendliness: Can you really create tests without programming knowledge?
• Cost efficiency: Are tests executed without additional AI costs after creation?
• Integration: Does the tool fit into your existing processes (Jira, Azure DevOps, etc.)?
• Maintainability: Can you independently adapt tests when requirements change?

The Future: What’s Coming Your Way

Even though current tools still have limitations, this will change quickly. In the near future, you’ll probably be able to:

• Generate tests directly from Jira stories
• Get automatic validation of your user stories for testability
• Operate self-service test automation without waiting for developer resources

Conclusion: AI as a Bridge, Not a Magic Wand

AI test automation isn’t the magic button that creates perfect tests at the push of a button. But it can be the bridge between your domain expertise and technical implementation.

Your added value remains: You define the best test cases, understand the requirements, and know the critical business processes.

AI supports: With technical implementation and makes test automation accessible to you.

The future of test automation isn’t about replacing human expertise, but enhancing it with intelligent AI systems. As a domain expert, you’re indispensable in this process - the technology just makes it easier for you to translate your expertise into automated tests.