By Vinithra S P, Multi-Cloud Certification Architect at VMware

Companies from all industries constantly face challenges to deliver software faster, with higher quality, and reduced risk and costs. At a conference this year, I talked about how Continuous Quality (CQ) processes and automation along with an emerging set of best practices can enable both product owners and partner companies to meet rapid release timelines with quality outcomes. The continuous quality process ensures consistency through the reuse of robust certification tests in both internal and external partner environments. This approach benefits not only the product owners, partner development teams, but also the extended teams that develop solutions/products that integrate with VMware. Using the common set of certification tests allows the ecosystem to avoid redundant investment in developing quality tools and processes and helps establish a consistent quality standard. Consequently, stakeholders in the ecosystem can have more autonomy when they develop their products and solutions. Through continuous quality efforts in the entire ecosystem, companies can deliver innovative products with agility and high quality that delights customers. The USP of the idea presented is that these partner programs and certification are geared towards device manufacturers and OEMs that often require physical devices, specific hardware to execute the required testing.

This was my first conference that I attended in-person post-pandemic and I realized just how much I’ve missed the energy of the room and networking in person with colleagues from all over the world. The presentation was well received, and I was really delighted to find the audience really excited about the challenges unique to our ecosystem and the solution. Our collaborative exchange of best practices, research findings and learnings not only broadened our perspectives but also fueled our passion for driving CI/CD excellence across the industry. We had the participants nominate the challenges they face in their organizations and out of all the topics we chose to brainstorm on the ‘Economy of tests’.

Here are some of the top takeaways and insights shared in the discussion:

• Mutation testing measures test effectiveness but only works for small tests.
• Focus on test suite minimization. Maintaining new tests is a cost – even if we add coverage.
• Culprit finding can identify tests that have broken.
• Test efficacy / effectiveness -Effective tests are generally less flaky. Most organizations take the approach on quarantining flaky tests, but with high standards and motivation to get tests back online.
• Cost of running tests including hardware costs, complex scheduling should be taken into account.

Thanks to the conference committee, participants and my team, I learned from speakers and industry experts who care fervently about their work, ideas, research findings and want to share the same with you in an accessible, usable way. There is also the reciprocal benefit of sharing these ideas and insights with colleagues, brainstorming on bringing back these learnings to work and seeing how we could put these takeaways into action.