A Plan-Do-Check-Act Framework for AI Code Generation

Summary: A Structured Approach to AI Code Generation with PDCA

This article introduces a Plan-Do-Check-Act (PDCA) framework for improving the quality and reliability of code generated by AI tools. The author argues that while AI code generation offers the potential for faster development, it often leads to issues with code quality, integration, and delivery. The PDCA framework, adapted from established software engineering practices, provides a structured approach to human-AI collaboration that prioritizes planning, testing, and continuous improvement. The author details the components of the PDCA cycle, including working agreements, detailed planning, test-driven implementation, completion analysis, and retrospection, and presents experimental results demonstrating its effectiveness compared to an unstructured approach. The article concludes that a structured approach like PDCA is crucial for realizing the full potential of AI in software development.

Detailed Explanation

Introduction: The Promise and Challenges of AI Code Generation

AI code generation tools are gaining traction as a means to accelerate software development. However, the author highlights a growing concern: these tools often fall short of delivering on their promise, leading to problems such as reduced delivery stability and increased code duplication and defects. The article posits that a structured approach is needed to harness the benefits of AI while mitigating these risks.

The Plan-Do-Check-Act (PDCA) Framework

The core of the article is the proposed PDCA framework, which is presented as a solution to the challenges of AI code generation. This framework is inspired by the Deming cycle and is adapted for human-AI collaboration. It emphasizes a cyclical process of continuous improvement, with each phase building upon the previous one.

Key Components of the PDCA Cycle:

• Working Agreements: These are foundational commitments made by the developer to guide the AI and maintain quality. They establish norms for the collaboration and ensure accountability.
• Planning Analysis: This initial phase involves a thorough analysis of the business objective, existing code patterns, and alternative approaches. It aims to provide context and direction for the AI.
• Planning Task Breakdown: The AI is instructed to break down the overall objective into smaller, testable tasks. This step ensures a structured approach to the implementation.
• Do (Test-Driven Implementation): This phase focuses on implementing the plan using a test-driven development (TDD) approach. The AI is guided to write failing tests first and then implement the code to make the tests pass. This ensures that the code meets the required behavior.
• Check (Completion Analysis): The AI reviews the implemented code, tests, and documentation to ensure completeness, adherence to the plan, and the absence of regressions.
• Act (Retrospection): This final phase involves a retrospective analysis of the entire process. The developer and AI reflect on what worked well, what could be improved, and how to refine the prompts and collaboration for future iterations.

Experimental Results and Analysis

The author conducted an experiment to compare the PDCA approach with an unstructured approach to AI code generation. The results indicate that the PDCA approach, while requiring more upfront planning, leads to:

• Reduced Token Usage: The PDCA approach achieved significantly lower token consumption compared to the unstructured approach, indicating more efficient use of the AI model.
• Fewer Lines of Code: The PDCA approach resulted in less overall code being generated, suggesting more focused and efficient implementations.
• Improved Test Coverage: The PDCA approach yielded more tests and better test coverage, leading to more robust and reliable code.
• Reduced Debugging Effort: The PDCA approach required less troubleshooting and fewer interventions after the initial code generation.

The experiment also revealed that the PDCA approach resulted in fewer lines of produced code, more comprehensive test coverage, and a higher number of implemented methods and classes. Furthermore, the author found that the PDCA approach, despite requiring more initial steps, led to a better overall developer experience, with fewer issues and more predictable outcomes.

Future Directions and Considerations

The author discusses areas for further refinement of the PDCA framework, including:

• Dynamic Model Selection: Experimenting with different AI models (e.g., Claude, Sonnet, Haiku) based on the complexity of the task.
• Adaptability to Complexity: Developing lighter versions of the framework for simpler tasks and more rigorous approaches for complex ones.
• Prompt Engineering: Continuously refining the prompts used in each phase of the PDCA cycle to optimize performance and guidance.

Conclusion

The article concludes that a structured approach to AI code generation, such as the PDCA framework, is essential for realizing the full potential of these tools. By prioritizing planning, testing, and continuous improvement, developers can leverage AI to accelerate development while maintaining code quality and reducing risks. The author emphasizes the importance of adapting the framework to individual needs and project complexities.

References

• GitHub Actions: https://github.com/features/actions
• GitClear: https://www.gitclear.com/
• LinearB: https://linearb.com/
• Alistair Cockburn’s Crystal Approach: (Link provided in the original text)
• Piya & Sullivan, 2024: “Best Practices for Test Driven Development Using Large Language Models”
• Ning et al., 2010: “PDCA process application in the continuous improvement of software quality”
• Wagner, et al. 2017: “Do Code Clones Matter?”
• Mondal, et al. 2019: “An empirical study on bug propagation through code cloning”
• InfoQ: https://www.infoq.com/

AI Disclosure

The author discloses that they used Claude to assist with content development, including brainstorming, outlining, and refining the text. They confirm that they personally reviewed all research sources and made all final content decisions, taking full responsibility for the accuracy and originality of the article.