Just-in-Time Testing: How AI and LLMs Are Changing the Way We Generate Software Tests

Introduction: What If We Could Generate Tests When They Are Needed?

Automated testing is nothing new in software engineering.

For years, QA engineers have used automation frameworks such as Selenium, Cypress, Playwright, and JUnit to execute test cases, validate application behaviour, and identify defects before software reaches production.

What is changing is how we create those tests.

Large Language Models (LLMs) are introducing a new approach: instead of relying entirely on engineers to manually write test cases or maintain predefined automation scripts, AI can analyse code changes, understand features, and generate tests to validate the software being developed.

But there is a problem.

Generating a test is relatively easy. Generating a test that correctly understands the intended behaviour of a system is much harder.

An LLM might identify a line of code that looks suspicious and generate a test around it. The test may execute successfully, but it might not represent a genuine software defect. It could produce a false positive because the model does not understand the wider business context.

This is where the concept of Just-in-Time Testing becomes interesting.

In January 2026, Meta researchers published a paper titled Just-in-Time Catching Test Generation at Meta, exploring how AI-assisted test generation can identify potential defects as code changes are introduced.

The research offers an interesting perspective on the future of software testing: moving beyond simply generating more tests towards generating more relevant tests.

1. What Is Just-in-Time Catching Test Generation?

Traditional automated testing generally follows a familiar process:

Developers write or modify application code.
QA engineers create test cases based on requirements.
Automation engineers implement automated scripts.
Tests execute during development or CI/CD.
Failed tests are investigated to determine whether they indicate a defect.

Just-in-Time Catching Test Generation introduces a different idea.

Rather than generating tests only to confirm that existing functionality continues to work, the system generates tests in response to code changes, looking for behaviours that could expose newly introduced defects.

There is an important distinction here:

Hardening tests: Tests designed to pass when the current implementation is behaving correctly. They help protect existing functionality.
Catching tests: Tests intended to fail when a code change introduces a bug. Their failure is the signal that something may be wrong.

In other words, a catching test is not necessarily expected to pass when it is generated. Its purpose is to challenge the new implementation and expose potential problems before the code reaches production.

This changes the way we think about test generation.

Instead of asking:

“Can AI generate a test for this feature?”

We can ask:

“Can AI identify a test that exposes a defect introduced by this particular code change?”

That is a much more interesting engineering problem.

2. The Context Problem: Why LLM-Generated Tests Can Produce False Positives

One of the biggest challenges with LLM-powered testing is understanding context.

LLMs are very good at analysing code patterns, identifying relationships, and generating plausible test scenarios. However, code alone does not always explain what a feature is supposed to do.

Consider a simple example.

A developer changes a function responsible for processing customer refunds.

The function contains the following condition:

if (refund_amount > available_balance)

An LLM examining only the source code might generate a test that expects the transaction to fail whenever the refund amount exceeds the available balance.

Technically, that may seem reasonable.

However, imagine the business requirement allows a refund to exceed the customer’s current account balance because the refund is funded through a separate payment settlement process.

Without that additional context, the generated test might incorrectly flag the implementation as defective.

The test is technically plausible but behaviourally incorrect.

This is a false positive.

The problem is not necessarily the quality of the generated test code. It is the information available to the model when deciding what the software should do.

This leads to an important distinction: context-unaware versus context-aware test generation.

3. Two Approaches to AI-Powered Test Generation

Meta’s research explores different approaches to generating catching tests, including approaches that reason primarily from code changes and approaches that incorporate additional information.

For QA engineers, we can use the following conceptual distinction to understand why context matters.

A. Context-unaware test generation

In this approach, the LLM primarily examines the source code and code changes.

It may analyse:

Modified functions and methods.
Conditional statements.
Input and output values.
Dependencies between functions.
Potential edge cases.
Changes in control flow.

The model then generates test cases based on what it can infer from the implementation.

Advantages:

Fast to generate.
Can be integrated directly into code review or CI/CD workflows.
Does not require extensive business documentation.
Useful for identifying technical edge cases.

Limitations:

May misunderstand intended business behaviour.
Can produce tests based on assumptions.
May flag valid implementation decisions as defects.
Has limited understanding of user expectations when requirements are not available.

This approach is useful for technical validation, but it does not necessarily establish whether the application meets business requirements.

B. Context-aware test generation

Context-aware test generation provides the LLM with additional information about the feature and its intended behaviour.

Instead of looking only at source code, the model can also consider:

Feature descriptions.
Business requirements.
Acceptance criteria.
User stories.
System workflows.
API contracts.
Expected business rules.
Related documentation.

The model can then generate tests based on both the implementation and the intended functionality.

For example, instead of simply analysing a refund-processing function, the LLM could also receive the following requirement:

“Customers can receive a full refund even when their current account balance is insufficient, because refunds are processed through the original payment settlement channel.”

With this additional information, the model has a better chance of understanding why a particular implementation is valid.

Advantages:

Better alignment with business requirements.
More meaningful test scenarios.
Reduced risk of tests based on incorrect assumptions.
Greater potential for generating user-oriented acceptance scenarios.

Limitations:

Requires reliable and up-to-date documentation.
Additional context needs to be collected and maintained.
Conflicting requirements can still confuse the model.
More information does not automatically guarantee correct reasoning.

The important lesson is that context improves the quality of test generation, but it does not eliminate the need for human review.

4. How Does This Relate to UAT and Integration System Testing?

This is where the concept becomes particularly interesting for QA professionals.

Different testing levels have different objectives. They also require different types of context.

Consider User Acceptance Testing (UAT) and Integration System Testing (IST).

Although both validate software functionality, they approach quality from different perspectives.

User Acceptance Testing (UAT): Is the Software Doing What the Business Needs?

UAT is highly context-dependent.

It focuses on whether the software supports real business processes and meets the expectations of its intended users.

For example, consider an employee salary application.

A UAT scenario might be:

“As an employee, I want to access a portion of my earned salary before payday so that I can manage unexpected expenses.”

To validate this scenario, the test needs to understand:

Employee eligibility.
Salary calculation rules.
Withdrawal limits.
Payroll schedules.
Approval requirements.
Business restrictions.
Expected user experience.

Simply examining the source code would not provide enough information to validate the entire business scenario.

The LLM needs to understand what the feature is intended to achieve.

Integration System Testing (IST): Are the Technical Components Working Together?

IST focuses more on technical interactions between integrated components and systems.

Using the same salary application, an IST scenario might validate:

Whether the application sends the correct API request.
Whether the payroll service returns the expected response.
Whether authentication tokens are handled correctly.
Whether transaction data is transferred between services.
Whether errors are handled when an external service is unavailable.

These tests still require context, including API contracts, interface specifications, and integration rules.

However, their primary focus is technical system behaviour rather than whether the overall business process satisfies the end user.

Comparing UAT and IST

Dimension	UAT	IST
Primary focus	Business requirements	System integration
Main question	Does it meet user expectations?	Do integrated components work correctly?
Context required	Business rules, user stories, workflows	API contracts, interfaces, technical specifications
Test perspective	User or business perspective	System or technical perspective
Example	Employee can access eligible earned salary	Payroll API returns the correct transaction response
LLM input	Feature requirements and acceptance criteria	Source code, API specifications and integration contracts

An important clarification: IST does not mean context-unaware testing. It also requires context, but the context is generally more technical.

Similarly, UAT is not automatically context-aware just because a human executes it. The quality of UAT still depends on clear requirements and realistic scenarios.

The real distinction is the type and depth of context available to the test-generation process.

5. What Did Meta’s Research Actually Demonstrate?

The paper, Just-in-Time Catching Test Generation at Meta, reports an industrial-scale implementation designed for large backend systems.

Some of its reported findings are particularly interesting:

The researchers analysed 22,126 generated tests.
Code-change-aware methods improved candidate catch generation by 4x compared with hardening tests.
They reported a 20x improvement compared with coincidentally failing tests.
Rule-based and LLM-based assessors reduced human review workload by 70%.
Engineers received 41 candidate catches.
Eight were confirmed as true positives.
Four of those eight would have led to serious failures had they remained undetected.

These findings demonstrate why just-in-time test generation is worth exploring.

However, there is an important distinction between generating candidate catches and confirming genuine defects. The 41 reported candidates were not all confirmed bugs.

The research also highlights the importance of false-positive assessment. Tests that fail do not automatically indicate that the software is defective.

The engineering challenge is to identify which failures represent genuine problems and which are caused by incorrect assumptions or irrelevant test scenarios.

This is particularly important in large-scale systems, where unnecessary test failures can slow down development teams and create additional investigation work.

6. How QA Engineers Can Apply This Concept

You do not need Meta’s infrastructure to start experimenting with LLM-powered test generation.

QA engineers can begin with a much smaller workflow.

Imagine a developer submits a pull request that changes a customer registration feature.

A practical AI-assisted workflow could look like this:

Step 1: Analyse the code change

Provide the LLM with the modified source code or pull request diff.

Ask it to identify potential changes in behaviour, edge cases, and affected dependencies.

Step 2: Provide feature context

Include the relevant user story, acceptance criteria, and business rules.

Do not assume that the source code alone explains the intended functionality.

Step 3: Generate candidate tests

Ask the LLM to generate:

Positive test scenarios.
Negative test scenarios.
Boundary-value tests.
Integration scenarios.
Regression scenarios.
Potential defect-catching tests.

Step 4: Assess the generated tests

Ask the LLM to explain why each test should pass or fail.

Check whether the expected result is supported by the actual requirements.

Step 5: Execute and review

Run the tests in an appropriate test environment.

Investigate failures rather than automatically treating every failed test as a defect.

Step 6: Refine the test suite

Keep useful tests, remove incorrect assumptions, and update the supporting context when requirements change.

This approach allows QA engineers to use AI as a test-generation assistant without surrendering responsibility for software quality.

7. What Does This Mean for the Future of QA Automation?

For a long time, automation engineering has focused heavily on writing scripts, maintaining locators, managing test data, and improving execution speed.

LLMs may change some of that work.

The next generation of QA automation engineers will need to understand more than automation frameworks.

They will need to understand:

Requirements engineering: How to interpret user stories and acceptance criteria.
Software architecture: How components interact and how code changes affect system behaviour.
Business processes: Why a feature exists and what users expect from it.
AI-assisted testing: How to generate, assess, and refine LLM-generated tests.
Test quality: How to distinguish genuine defects from false positives.
Critical thinking: How to challenge an AI-generated test instead of accepting it at face value.

The most valuable QA engineers may not be the ones who generate the largest number of automated tests.

They may be the ones who understand which tests matter, why they matter, and whether the results represent real risks to the business.

Conclusion: Context Is the Missing Piece in AI-Powered Testing

Automated test generation is not a new idea. What makes LLM-powered testing interesting is its ability to interpret code, requirements, and feature descriptions to create tests with less manual effort.

Meta’s just-in-time catching test research demonstrates how this idea can be applied at an industrial scale, with the goal of identifying defects as code changes are introduced.

But one lesson stands out:

A test generated from code is not necessarily a test that understands the feature.

Context matters.

A technically correct test can still be a bad test if it validates the wrong expectation. Likewise, a test that understands the business requirement but ignores technical integration risks is incomplete.

The future of AI-powered testing is therefore not simply about asking an LLM to generate more test cases.

It is about giving AI the right context, validating its assumptions, and using human engineering judgement to determine whether a test genuinely protects software quality.

For QA engineers, this is an opportunity to move beyond writing automation scripts and become stronger contributors to the entire software development lifecycle.