Client Overview

The client is a development agency based in Mexico. Agency AI acted as a subcontractor to develop AI solutions for them.

Project Background

The client's initial problem was the need for a system that would automatically generate unit tests based on a user story within a code base. The main objectives of the project were to generate unit tests relevant to the user's story and automate the coverage testing to ensure that the tests work properly.

Challenges

The client faced the challenge of finding the proper code based on a user story. The technology constraint was a significant issue due to the client's preference for deploying everything on Azure. This led to complications when searching the code base. The Azure Code Search system provided by Azure for basic code search within a repository was not very effective, and setting up a search problem to do vector embeddings across the entire code base would be costly and complex.

Solution

The client would create a ticket in their Azure DevOps board. Agency AI would then create a technical report which would revise the user story based on the code it found and what files need to be tested. A test case plan would then be created, which would, based on the technical report, outline what the test cases should be. Finally, the test cases would be created and a pull request would be opened, placing them in the proper place in the directory.

The Agency Swarm framework was utilized to develop a code search agent, a technical report agent, and a unit test generation agent.

Key Features and Functionality

Three AI agents were developed:

1. Code Search Agent: Searches through the code for relevant code snippets based on the user story.
2. Technical Report Agent: Generates a technical report based on the code snippets found, outlining what parts of the code should be tested in the unit test.
3. Unit Test Generation Agent: Creates unit test files that can then be run for a coverage report.

The solution integrated with various Azure APIs and utilized the Azure deployment for assistance API.

Results and Impact

The client is still testing the solution. The success of the project will be demonstrated by the creation of proper unit tests and the total coverage generated.

Lessons Learned

The unit test generation aspect of the project worked well. The most complex part of the project was finding the proper code snippets. A key lesson learned was not to overcomplicate the system for running the code in the initial MVP build.

Conclusion

The project aimed to automate unit test generation and testing for various tickets within the client's QA system. There are ongoing plans with the client for future development.