Testing, Tools, and Frameworks Resources

DZone's Featured Testing, Tools, and Frameworks Resources

Hints for Unit Testing With AssertJ

By Arnošt Havelka

CORE

Unit testing has become a standard part of development. Many tools can be utilized for it in many different ways. This article demonstrates a couple of hints or, let's say, best practices working well for me. In This Article, You Will Learn How to write clean and readable unit tests with JUnit and Assert frameworks How to avoid false positive tests in some cases What to avoid when writing unit tests Don't Overuse NPE Checks We all tend to avoid NullPointerException as much as possible in the main code because it can lead to ugly consequences. I believe our main concern is not to avoid NPE in tests. Our goal is to verify the behavior of a tested component in a clean, readable, and reliable way. Bad Practice Many times in the past, I've used isNotNull assertion even when it wasn't needed, like in the example below: Java @Test public void getMessage() { assertThat(service).isNotNull(); assertThat(service.getMessage()).isEqualTo("Hello world!"); } This test produces errors like this: Plain Text java.lang.AssertionError: Expecting actual not to be null at com.github.aha.poc.junit.spring.StandardSpringTest.test(StandardSpringTest.java:19) Good Practice Even though the additional isNotNull assertion is not really harmful, it should be avoided due to the following reasons: It doesn't add any additional value. It's just more code to read and maintain. The test fails anyway when service is null and we see the real root cause of the failure. The test still fulfills its purpose. The produced error message is even better with the AssertJ assertion. See the modified test assertion below. Java @Test public void getMessage() { assertThat(service.getMessage()).isEqualTo("Hello world!"); } The modified test produces an error like this: Java java.lang.NullPointerException: Cannot invoke "com.github.aha.poc.junit.spring.HelloService.getMessage()" because "this.service" is null at com.github.aha.poc.junit.spring.StandardSpringTest.test(StandardSpringTest.java:19) Note: The example can be found in SimpleSpringTest. Assert Values and Not the Result From time to time, we write a correct test, but in a "bad" way. It means the test works exactly as intended and verifies our component, but the failure isn't providing enough information. Therefore, our goal is to assert the value and not the comparison result. Bad Practice Let's see a couple of such bad tests: Java // #1 assertThat(argument.contains("o")).isTrue(); // #2 var result = "Welcome to JDK 10"; assertThat(result instanceof String).isTrue(); // #3 assertThat("".isBlank()).isTrue(); // #4 Optional<Method> testMethod = testInfo.getTestMethod(); assertThat(testMethod.isPresent()).isTrue(); Some errors from the tests above are shown below. Plain Text #1 Expecting value to be true but was false at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) at com.github.aha.poc.junit5.params.SimpleParamTests.stringTest(SimpleParamTests.java:23) #3 Expecting value to be true but was false at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) at com.github.aha.poc.junit5.ConditionalTests.checkJdk11Feature(ConditionalTests.java:50) Good Practice The solution is quite easy with AssertJ and its fluent API. All the cases mentioned above can be easily rewritten as: Java // #1 assertThat(argument).contains("o"); // #2 assertThat(result).isInstanceOf(String.class); // #3 assertThat("").isBlank(); // #4 assertThat(testMethod).isPresent(); The very same errors as mentioned before provide more value now. Plain Text #1 Expecting actual: "Hello" to contain: "f" at com.github.aha.poc.junit5.params.SimpleParamTests.stringTest(SimpleParamTests.java:23) #3 Expecting blank but was: "a" at com.github.aha.poc.junit5.ConditionalTests.checkJdk11Feature(ConditionalTests.java:50) Note: The example can be found in SimpleParamTests. Group-Related Assertions Together The assertion chaining and a related code indentation help a lot in the test clarity and readability. Bad Practice As we write a test, we can end up with the correct, but less readable test. Let's imagine a test where we want to find countries and do these checks: Count the found countries. Assert the first entry with several values. Such tests can look like this example: Java @Test void listCountries() { List<Country> result = ...; assertThat(result).hasSize(5); var country = result.get(0); assertThat(country.getName()).isEqualTo("Spain"); assertThat(country.getCities().stream().map(City::getName)).contains("Barcelona"); } Good Practice Even though the previous test is correct, we should improve the readability a lot by grouping the related assertions together (lines 9-11). The goal here is to assert result once and write many chained assertions as needed. See the modified version below. Java @Test void listCountries() { List<Country> result = ...; assertThat(result) .hasSize(5) .singleElement() .satisfies(c -> { assertThat(c.getName()).isEqualTo("Spain"); assertThat(c.getCities().stream().map(City::getName)).contains("Barcelona"); }); } Note: The example can be found in CountryRepositoryOtherTests. Prevent False Positive Successful Test When any assertion method with the ThrowingConsumer argument is used, then the argument has to contain assertThat in the consumer as well. Otherwise, the test would pass all the time - even when the comparison fails, which means the wrong test. The test fails only when an assertion throws a RuntimeException or AssertionError exception. I guess it's clear, but it's easy to forget about it and write the wrong test. It happens to me from time to time. Bad Practice Let's imagine we have a couple of country codes and we want to verify that every code satisfies some condition. In our dummy case, we want to assert that every country code contains "a" character. As you can see, it's nonsense: we have codes in uppercase, but we aren't applying case insensitivity in the assertion. Java @Test void assertValues() throws Exception { var countryCodes = List.of("CZ", "AT", "CA"); assertThat( countryCodes ) .hasSize(3) .allSatisfy(countryCode -> countryCode.contains("a")); } Surprisingly, our test passed successfully. Good Practice As mentioned at the beginning of this section, our test can be corrected easily with additional assertThat in the consumer (line 7). The correct test should be like this: Java @Test void assertValues() throws Exception { var countryCodes = List.of("CZ", "AT", "CA"); assertThat( countryCodes ) .hasSize(3) .allSatisfy(countryCode -> assertThat( countryCode ).containsIgnoringCase("a")); } Now the test fails as expected with the correct error message. Plain Text java.lang.AssertionError: Expecting all elements of: ["CZ", "AT", "CA"] to satisfy given requirements, but these elements did not: "CZ" error: Expecting actual: "CZ" to contain: "a" (ignoring case) at com.github.aha.sat.core.clr.AppleTest.assertValues(AppleTest.java:45) Chain Assertions The last hint is not really the practice, but rather the recommendation. The AssertJ fluent API should be utilized in order to create more readable tests. Non-Chaining Assertions Let's consider listLogs test, whose purpose is to test the logging of a component. The goal here is to check: Asserted number of collected logs Assert existence of DEBUG and INFO log message Java @Test void listLogs() throws Exception { ListAppender<ILoggingEvent> logAppender = ...; assertThat( logAppender.list ).hasSize(2); assertThat( logAppender.list ).anySatisfy(logEntry -> { assertThat( logEntry.getLevel() ).isEqualTo(DEBUG); assertThat( logEntry.getFormattedMessage() ).startsWith("Initializing Apple"); }); assertThat( logAppender.list ).anySatisfy(logEntry -> { assertThat( logEntry.getLevel() ).isEqualTo(INFO); assertThat( logEntry.getFormattedMessage() ).isEqualTo("Here's Apple runner" ); }); } Chaining Assertions With the mentioned fluent API and the chaining, we can change the test this way: Java @Test void listLogs() throws Exception { ListAppender<ILoggingEvent> logAppender = ...; assertThat( logAppender.list ) .hasSize(2) .anySatisfy(logEntry -> { assertThat( logEntry.getLevel() ).isEqualTo(DEBUG); assertThat( logEntry.getFormattedMessage() ).startsWith("Initializing Apple"); }) .anySatisfy(logEntry -> { assertThat( logEntry.getLevel() ).isEqualTo(INFO); assertThat( logEntry.getFormattedMessage() ).isEqualTo("Here's Apple runner" ); }); } Note: the example can be found in AppleTest. Summary and Source Code The AssertJ framework provides a lot of help with their fluent API. In this article, several tips and hints were presented in order to produce clearer and more reliable tests. Please be aware that most of these recommendations are subjective. It depends on personal preferences and code style. The used source code can be found in my repositories: spring-advanced-training junit-poc More

Fundamentals of Functions and Relations for Software Quality Engineering

By Stelios Manioudakis

CORE

Understanding the fundamentals of functions and relations is paramount. Grasping these core concepts lays the groundwork for effective software development and testing. We will delve into the basics of functions and relations, exploring their significance in software engineering and their implications for ensuring software quality. We will highlight basic scenarios for testing to kickstart more intricate testing activities. Effective testing is not just about covering every line of code. It's about understanding the underlying relationships. How do we effectively test the complex relationships in our software code? Understanding functions and relations proves an invaluable asset in this endeavor. This article explores these mathematical concepts, weaving their definitions with practical testing applications. By leveraging this knowledge, you can design more targeted and efficient test strategies, ultimately strengthening your software's quality. Functions A function associates elements of sets and it is a special kind of relation. Our code contains functions that associate outputs with inputs. In the mathematical formulation of a function, the inputs are the domain and the outputs are the range of the function. Formally, a function f from set A to set B can be defined as a subset of the Cartesian product A × B. This technical definition essentially ensures that each element in A maps to a unique element in B. In simpler terms, a well-behaved function never links a single input to multiple different outputs. This characteristic is crucial for testing, as non-deterministic functions (producing unpredictable outputs for the same input) pose unique challenges. It's worth noting that while code can be viewed as functions in a broad sense, not all are "pure" functions. Pure functions have no side effects, meaning they solely rely on their inputs to produce outputs without altering any external state. In practice, our code involves side effects (such as modifying databases or interacting with external devices), complicating the pure function interpretation. Basic Function Types We will cover specific types of deterministic functions. Onto functions demand specific testing strategies due to their "every input has an output" requirement. By recognizing their characteristics and potential edge cases, software testers can create more effective test suites that uncover hidden issues and ensure reliable software behavior. Into functions require a nuanced testing approach due to their selective mapping nature. By comprehending their characteristics and potential edge cases, you can craft targeted test suites that ensure robust and reliable software behavior. Mastering the intricacies of one-to-one functions empowers you to craft effective test suites that safeguard your software against hidden mapping errors and ensure data integrity. With a keen eye for unique outputs and potential collisions, you can confidently navigate the testing terrain and contribute to building reliable and secure software. By harnessing the power of equivalence classes, thorough edge case testing, and an understanding of potential performance limitations, you can conquer the challenges of many-to-one functions. This empowers you to craft test suites that ensure your software accurately processes diverse inputs and delivers consistent, reliable outputs. Onto Functions An onto function maps every element in its domain (set of inputs) to exactly one element in its range. Imagine a function converting weekdays to corresponding weekend days (Mon -> Sat, Tue -> Sun). Every weekday has a unique weekend counterpart, satisfying the onto property. Testing implications: Coverage: Testing onto functions requires ensuring all possible inputs are covered. Missing even one input could lead to untested scenarios and potential bugs. Edge cases: Pay close attention to boundary values at the edges of the domain. For instance, testing the weekend conversion function with Sunday might reveal unexpected behavior if it doesn't map to Monday (assuming a cyclical mapping). Inverse function existence: If an inverse function exists (mapping weekend days back to weekdays), testing it can indirectly validate the onto function's correctness. Performance and scalability: For large domains, testing every possible input might not be feasible. Utilizing equivalence classes or randomized testing can balance coverage with efficiency. Examples: User authentication: Mapping usernames to unique user profiles is typically an onto function. Testing should involve diverse usernames to ensure all valid ones have corresponding profiles. Error code mapping: Different error codes might map to specific error messages. Onto functions ensure every error code has a corresponding message, requiring comprehensive testing of all potential error codes. Data encryption/decryption: Onto functions can be used to ensure every encrypted message has a unique decryption key. Testing involves encrypting various messages and verifying they decrypt correctly. Into Functions Imagine a function that converts Celsius temperatures to Fahrenheit. While every Celsius value has a corresponding Fahrenheit equivalent, not every Fahrenheit value has a corresponding Celsius counterpart (e.g., -40°F has no exact Celsius equivalent). This function exemplifies an into function, mapping some, but not all, elements in its domain to the range. Testing Implications: Focus on covered elements: Testing into functions primarily focuses on ensuring all valid inputs that do have outputs are covered. Unlike onto functions, missing some inputs might be permissible based on the function's design. Edge cases and invalid inputs: Pay close attention to invalid inputs that fall outside the domain. The function's behavior for these inputs should be well-defined, whether returning a specific error value or throwing an exception. Partial testing strategies: Since not all inputs have outputs, exhaustive testing might be unnecessary. Consider equivalence partitioning to group similar inputs with likely similar behavior, optimizing test coverage without redundancy. Inverse function considerations: Unlike onto functions, inverses for into functions are generally not possible. However, if a related function exists that maps elements back to the domain, testing its correctness can indirectly validate the into function's behavior. Examples: File extension to content type mapping: Not all file extensions have corresponding content types (e.g., a ".custom" extension might not be recognized). Testing involves verifying known extensions but also validating the function's handling of unknown ones. User permission checks: Certain actions might require specific user permissions. An into function could check if a user has the necessary permission. Tests would focus on valid permissions but also include cases where permissions are absent. Data validation functions: These functions might check if data adheres to specific formats or ranges. While valid data should be processed, testing should also include invalid data to ensure proper error handling or rejection. One-to-One Functions Imagine a function that assigns unique identification numbers to students. Each student receives a distinct ID, ensuring no duplicates exist. This function perfectly embodies the one-to-one principle: one input (student) leads to one and only one output (ID). Testing Implications: Unique outputs: The heart of testing lies in verifying that distinct inputs always produce different outputs. Focus on creating test cases that cover diverse input scenarios to expose any potential mapping errors. Inverse function potential: If an inverse function exists (mapping IDs back to students), testing its correctness indirectly validates the one-to-one property of the original function. Edge cases and collisions: Pay close attention to potential "collision" scenarios where different inputs might accidentally map to the same output. Thorough testing of boundary values and special cases is crucial. Equivalence classes: While exhaustive testing might seem necessary, consider grouping similar inputs (e.g., student age ranges) into equivalence classes. Testing one representative from each class can optimize coverage without redundancy. Examples: User login with unique usernames: Each username should map to a single user account, ensuring one-to-one functionality. Test with diverse usernames to uncover potential duplicate mappings. Generating unique random numbers: Random number generators often aim for one-to-one mappings to avoid predictability. Testing involves generating large sets of numbers and verifying their uniqueness. Hashing algorithms: These functions map data to unique "hash" values. Testing focuses on ensuring different data produces distinct hashes and that collisions (same hash for different data) are highly unlikely. Many-to-One Functions Imagine a function that categorizes books by genre. Here, multiple books of different titles and authors could belong to the same genre (e.g., Sci-Fi). This exemplifies a many-to-one function, where several inputs map to a single output. Testing Implications: Focus on valid mappings: While multiple inputs might share an output, your primary focus is ensuring valid inputs indeed map to the correct output. Test diverse input scenarios to catch errors in the mapping logic. Equivalence classes: Grouping similar inputs based on shared characteristics (e.g., book themes) allows you to test one representative from each class, optimizing coverage without redundantly testing every possible combination. Edge cases and invalid inputs: Pay close attention to how the function handles invalid inputs or those falling outside its defined domain. Does it return a specific error value, ignore them, or exhibit unexpected behavior? Inverse function considerations: In many cases, inverse functions don't exist for many-to-one functions. However, if a related function maps outputs back to specific input subsets, testing its correctness can indirectly validate the original function's behavior. Examples: Product discount functions: They usually map different product quantities (e.g., 1 item, 3 items, 5 items) to a single discount percentage (e.g., 10% off for bulk purchases). Ensure discounts apply correctly for various quantities within and outside designated ranges. Test edge cases like single-item purchases and quantities exceeding discount thresholds. Shipping cost calculators: They often map different combinations of origin, destination, and package weight to a single shipping cost. Cover diverse locations, weight ranges, and shipping options. Verify calculated costs against established pricing tables and consider edge cases like remote locations or unusual package sizes. Search algorithms: Search queries could return various relevant results. Test with diverse queries and ensure the returned results indeed match the query intent, even if they share the same "relevant" category. Relations: Beyond Simple Mappings While functions provide clear input-output connections, not all relationships in software are so straightforward. Imagine tracking dependencies between tasks in a project management tool. Here, multiple tasks might relate to each other, forming a more complex network. This is where relations come in: Reflexive, symmetric, transitive: Relations can exhibit specific attributes like reflexivity (a task relates to itself), symmetry (if task A depends on B, then B depends on A), and transitivity (if A depends on B and B depends on C, then A depends on C). These properties have testing implications. For instance, in a file system deletion operation, transitivity ensures that deleting a folder also deletes its contents. Equivalence relations and partitions: Relations can sometimes group elements into equivalence classes, where elements within a class behave similarly. Testers can leverage this by testing one element in each class, assuming similar behavior for others, saving time and resources. Transitive Dependency Relation Consider a project management tool where tasks have dependencies. This forms a more complex network of relationships, represented by relations. These go beyond simple input-output mappings: Python def can_start(task, dependencies): """Checks if a task can start given its dependencies (completed or not).""" for dep in dependencies: if not dep.is_completed(): return False return True # Transitive relation: A depends on B, B depends on C, implies A depends on C task_A = Task("Write requirements") task_B = Task("Design prototype") task_C = Task("Develop code") task_A.add_dependency(task_B) task_B.add_dependency(task_C) assert can_start(task_A) == False # Task A can't start while C is incomplete task_C.mark_completed() assert can_start(task_A) == True # Now A can start as C is complete This can_start function utilizes a transitive relation. If task A depends on B, and B depends on C, then A ultimately depends on C. Testing involves checking various dependency combinations to ensure tasks can only start when their transitive dependencies are fulfilled. Testing Basic transitive dependency: Ensure the function accurately reflects the transitive nature of dependencies. Test scenarios where task A depends on B, B depends on C, and so on, ensuring A can only start when C is completed. Circular dependencies: Verify the function's behavior when circular dependencies exist (e.g., A depends on B, B depends on A). Handle them appropriately, either preventing circular dependencies altogether or flagging them for manual evaluation. Multiple dependencies: Test cases where a task has multiple dependencies. Ensure the function only allows the task to start when all its dependencies are complete, regardless of their number or complexity. Reflexive Relation Consider a user login system where a user needs to be logged in to perform certain actions. Python def is_authorized(user, action): """Checks if a user is authorized to perform an action.""" return user.is_logged_in() and user.has_permission(action) Every user is considered authorized to perform the action of "logging in" (regardless of other permissions). This establishes a reflexive relation, where every user is related to the action of "logging in" (user -> "log in"). Testing Verify that is_authorized(user, "log in") is always True for any user object, regardless of their login status or permissions. Test edge cases like newly created users, users with specific permission sets, and even invalid user objects. Symmetric Relation Consider a social media platform where users can "follow" each other. Python def are_friends(user1, user2): """Checks if two users are friends (follow each other).""" return user1.follows(user2) and user2.follows(user1) In a friendship, the relationship flows both ways. If user A follows user B, then user B must also follow user A. This establishes a symmetric relation, where user A and user B are related in the same way ("follows") if the friendship exists. Testing Verify that are_friends(user1, user2) is True only if are_friends(user2, user1) is also True. Test various scenarios like mutual follows, one-way follows, and users who don't know each other. Consider edge cases like blocked users, deactivated accounts, and privacy settings affecting visibility. Equivalence Classes and Partitioning Relations can group elements with similar behavior into equivalence classes. Testers can leverage this by testing one element in each class, assuming similar behavior for others. Python def get_file_type(filename): """Classifies a file based on its extension (text, image, etc.).""" extension = filename.split(".")[-1].lower() if extension in {".txt", ".md"}: return "text" elif extension in {".jpg", ".png"}: return "image" else: return "unknown" # Equivalence classes: Test one file from each class text_files = ["readme.txt", "report.md"] image_files = ["photo.jpg", "banner.png"] for file in text_files: assert get_file_type(file) == "text" for file in image_files: assert get_file_type(file) == "image" In the get_file_type example, testing one file from each equivalence class (text and image) efficiently covers different file extensions without redundant testing. This principle applies to various scenarios, like testing error handling for different input types or user roles with similar permissions. Visualizing Relationships for Clarity Visualizing functions and relations can significantly enhance understanding and test design. Two popular ways to visualize are the following. Function mapping diagrams: Draw arrows connecting inputs to outputs, highlighting one-to-one, many-to-one scenarios. Relation network diagrams: Represent elements as nodes and connections as edges, indicating reflexivity, symmetry, and transitivity. Wrapping Up By understanding functions and relations both conceptually and practically, we gain valuable tools for effective software development and testing. Functions and relations provide a foundational framework for organizing and reasoning about the intricate relationships between different parts of our code, ultimately leading to more robust and reliable software. Remember, effective testing is not just about covering every line of code but about understanding the underlying relationships that make your software tick. More

Unleashing the Power of Git Bisect

By Shai Almog

CORE

Be Punctual! Avoiding Kotlin’s lateinit In Spring Boot Testing

By Severn Everett

Top Visual Regression Testing Tools

By Yogesh Solanki

Self-Healing Test Automation: A Key Enabler for Agile and DevOps Teams

Test automation is essential for ensuring the quality of software products. However, test automation can be challenging to maintain, especially as software applications evolve over time. Self-healing test automation is an emerging concept in software testing that uses artificial intelligence and machine learning techniques to enable automated tests to detect and self-correct issues. This makes test automation more reliable and cost-effective and reduces the amount of time and resources required to maintain test scripts. In this article, we will discuss the benefits of self-healing test automation, how it works, and how to implement it in your organization. What Is Self/Auto-Healing Test Automation? Self-healing test automation is a new approach to test automation that uses artificial intelligence (AI) and machine learning (ML) to make test scripts more robust and adaptable. With self-healing test automation, test scripts can automatically detect and repair themselves when changes are made to the application under test, including shifting layouts and broken selectors. This makes it possible to automate tests for complex applications with frequently changing user interfaces without having to constantly maintain and update the test scripts. Why Is Self-Healing Test Automation Necessary? Test automation scripts can easily break when changes are made to the user interface. This is because test automation scripts are typically designed to interact with specific elements on the screen, such as buttons, text fields, and labels. When these elements change, the script may no longer be able to find them or interact with them correctly. This can lead to test failures and false positives, which can be time-consuming and frustrating to resolve. Also, the user interfaces are constantly evolving, with new features and bug fixes being added frequently. This means that test automation scripts need to be updated regularly to adapt to these changes. However, updating test automation scripts can be a manual and time-consuming process, which can be challenging to keep up with the pace of change. Self-healing test automation addresses this fragility of traditional test automation scripts by adapting to changes in the user interface automatically to make test scripts more robust and adaptable. Self-healing test scripts can automatically detect and repair themselves when changes are made to the application under test, which can help to reduce test maintenance costs, improve test quality, and increase test coverage. How Does Self-Healing Mechanism Work? Step 1: The self-healing mechanism gets triggered whenever “NoSuchElement” or a similar error occurs for an element mentioned in the automation script. Step 2: The algorithm analyzes the test script to identify the root cause of the error. Step 3: The algorithm uses AI-powered data analytics to identify the exact object in the test script that has changed. An object can be any interface item like a webpage, navigation button, or text box. Step 4: The algorithm updates the test script with the new identification parameters for the affected object(s). Step 5: The updated test case is re-executed to verify that the remediation has been successful. How Self-Healing Test Automation Adds Value to Your Software Delivery Process Leveraging self-healing capabilities allows test automation to adapt to changes, improving test coverage, reducing maintenance efforts, and enabling faster feedback. Saves Time and Effort Self-healing test automation can save organizations a significant amount of time and effort in software testing. Traditional test automation approaches require manual intervention to fix errors or failures that occur during test execution. This can be a time-consuming and error-prone process, especially when dealing with large and complex test suites. Self-healing test automation eliminates the need for manual intervention, allowing tests to recover automatically from failures or errors. Improves Test Coverage Self-healing test automation can help to improve test coverage by allowing testers to focus on writing new tests rather than maintaining existing tests. This is because self-healing tests can automatically adapt to changes in the software under test, which means that testers do not need to spend time updating their tests every time the software changes. As a result, testers can focus on writing new tests to cover new features and functionality. Self-healing automation can improve test coverage by 5-10% by eliminating unnecessary code, resulting in shorter delivery times and higher returns on investment. Prevents Object Flakiness Object flakiness is a common problem in test automation, especially for GUI testing. Object flakiness occurs when a test fails because it is unable to locate an object on the page. This can happen for a variety of reasons, such as changes to the UI, changes to the underlying code, or network latency. Self-healing test automation can detect and prevent object flakiness by analyzing test results and identifying patterns that indicate flaky tests. By preventing object flakiness, teams can reduce the number of false positives and negatives, improving the overall accuracy and reliability of test results. Faster Feedback Loop Self-healing test automation also enables a faster feedback loop. With traditional test automation approaches, tests are often run manually or through a continuous integration pipeline. However, with self-healing test automation, tests can be run continuously, providing immediate feedback on the quality of the application under test. This enables teams to identify and fix issues faster, improving the overall quality and reliability of the application. Conclusion In Agile methodology, applications are continuously developed and tested in short cycles. This can make it difficult to maintain test cases, as the application is constantly changing. Self-healing test automation can help to overcome this challenge by automatically updating test cases when the application under test changes.

By Rohit Wadhwani

Probability Basics for Software Testing

Have you ever felt like building a castle out of sand, only to have the tide of unexpected software bugs wash it all away? In everyday work in software development, unforeseen issues can spell disaster. But what if we could predict the likelihood of these problems before they arise? Enter the realm of probability, our secret weapon for building robust and reliable software. Probability plays a crucial role in software testing, helping us understand the likelihood of certain events like encountering specific paths within the code and assessing the effectiveness of test coverage. This article starts from scratch. We define probability theoretically and practically. We'll then dive into conditional probability and Bayes' theorem, giving basic formulas, examples, and applications to software testing and beyond. Laying the Foundation: Defining Probability We begin with the fundamental question: what exactly is probability? In the realm of software testing, it represents the likelihood of a particular event occurring, such as executing a specific sequence of statements within our code. Imagine a coin toss: the probability of landing heads is 1/2 (assuming a fair coin). Similarly, we can assign probabilities to events in software, but the complexities inherent in code demand a more robust approach than counting "heads" and "tails." Beyond Laplace's Marble Bag: A Set-Theoretic Approach While the classic definition by Laplace, which compares favorable outcomes to total possibilities, works for simple scenarios, it becomes cumbersome for intricate software systems. Instead, we leverage the power of set theory and propositional logic to build a more versatile framework. Imagine the set of all possible events in our code as a vast universe. Each event, like encountering a specific path in our code, is represented by a subset within this universe. We then formulate propositions (statements about these events) to understand their characteristics. The key lies in the truth set of a proposition – the collection of events within the universe where the proposition holds true. Probability Takes Shape: From Truth Sets to Calculations Now, comes the magic of probability. The probability of a proposition being true, denoted as Pr(p), is simply the size (cardinality) of its truth set divided by the size of the entire universe. This aligns with Laplace's intuition but with a more rigorous foundation. Think about checking if a month has 30 days. In the universe of all months (U = {Jan, Feb, ..., Dec}), the proposition "p(m): m is a 30-day month" has a truth set T(p(m)) = {Apr, Jun, Sep, Nov}. Therefore, Pr(p(m)) = 4/12, providing a precise measure of the likelihood of encountering a 30-day month. The Universe Matters: Choosing Wisely Selecting the appropriate universe for our calculations is crucial. Imagine finding the probability of a February in a year (Pr(February)) – simply 1/12. But what about the probability of a month with 29 days? Here, the universe needs to consider leap years, influencing the truth set and ultimately, the probability. This highlights the importance of choosing the right "playing field" for our probability calculations and avoiding "universe shifts" that can lead to misleading results. Imagine we're testing an e-commerce application and only consider the universe of "typical" transactions during peak season (e.g., holidays). We calculate the probability of encountering a payment gateway error to be low. However, we haven't considered the universe of "all possible transactions," which might include high-value orders, international payments, or unexpected surges due to flash sales. These scenarios could have a higher chance of triggering payment gateway issues, leading to underestimated risks and potential outages during crucial business periods. Essential Tools in Our Probability Arsenal Beyond the basic framework, there are some key facts that govern the behavior of probabilities within a specific universe: Pr(not p) = 1 - Pr(p): The probability of an event not happening is simply 1 minus the probability of it happening. Pr(p and q) = Pr(p) * Pr(q) (assuming independence): If events p and q are independent (meaning they don't influence each other), the probability of both happening is the product of their individual probabilities. Pr(p or q) = Pr(p) + Pr(q) - Pr(p and q): The probability of either p or q happening, or both, is the sum of their individual probabilities minus the probability of both happening together. These principles, combined with our understanding of set theory and propositional logic, can empower us to confidently manipulate probability expressions within the context of software testing. Conditional Probability While probability helps us estimate the likelihood of encountering specific events and optimize testing strategies, conditional probability takes this a step further by considering the influence of one event on the probability of another. This concept offers valuable insights in various software testing scenarios. Understanding the "Given" Conditional probability focuses on the probability of event B happening given that event A has already occurred. We represent it as P(B | A). This "given" condition acts like a filter, narrowing down the possibilities for event B based on the knowledge that event A has already happened. Basic Formulas for Conditional Probability Here are some key formulas and their relevance to software testing. 1. Unveiling the Definition (Set Membership) P(B | A) = P(A ∩ B) / P(A) Imagine events A and B as sets representing specific scenarios in our software (e.g., A = invalid login attempt, B = system error). The intersection (∩) signifies "both happening simultaneously." This translates to the probability of event B occurring given event A, represented by P(B | A), being equal to the ratio of the elements in the intersection (A ∩ B) to the elements in set A alone. In general, P(A ∩ B) might represent encountering a specific bug under certain conditions (A), and P(A) could represent the overall probability of encountering that bug. Example: Analyzing login errors, we calculate P(error | invalid login) = P({invalid login ∩ system error}) / P({invalid login}). This reveals the likelihood of encountering a system error specifically when an invalid login attempt occurs. 2. Relationship with Marginal Probabilities (Set Union and Complement) P(B) = P(B | A) * P(A) + P(B | ~A) * P(~A) This formula relates the unconditional probability of event B (P(B)) to its conditional probabilities given A and its opposite (~A), along with the marginal probabilities of A and its opposite. It highlights how considering conditions (A or ~A) can alter the overall probability of B. Example: Imagine testing a payment processing system. We estimate P(payment failure) = P(failure | network issue) * P(network issue) + P(failure | normal network) * P(normal network). This allows us to analyze the combined probability of payment failure considering both network issues and normal operation scenarios. 3. Total Probability (Unveiling Overlap, Complement and Difference) P(A ∪ B) = P(A) + P(B) - P(A ∩ B) This formula, though not directly related to conditional probability, is crucial for understanding set relationships in software testing. It ensures that considering both events A and B, along with their overlap (A ∩ B), doesn't lead to overcounting possibilities. The union (∪) signifies "either A or B or both." Example: Imagine you're testing a feature that allows users to upload files. You're interested in calculating the probability of encountering specific scenarios during testing: Events A: User uploads a valid file type (e.g., PDF, DOCX) B: User uploads a file larger than 10MB You want to ensure you cover both valid and invalid file uploads, considering both size and type. P(A ∪ B): This could represent the probability of encountering either a valid file type, a file exceeding 10MB, or both. P(A): This could represent the probability of encountering a valid file type, regardless of size. P(B): This could represent the probability of encountering a file larger than 10MB, regardless of type. P(A ∩ B): This could represent the probability of encountering a file that is both valid and larger than 10MB (overlap). 4. Independence (Disjoint Sets) P(B | A) = P(B) if A ∩ B = Ø (empty set), meaning that A and B are independent (no influence on each other). This special case applies when knowing event A doesn't change the probability of event B. While often not the case in complex software systems, it helps simplify calculations when events are truly independent. Example: Imagine testing two independent modules. Assuming no interaction, P(error in module 1 | error in module 2) = P(error in module 1), as knowing an error in module 2 doesn't influence the probability of an error in module 1. Application to Risk Assessment Suppose a component relies on an external service. We can calculate the probability of the component failing given the external service is unavailable. This conditional probability helps assess the overall system risk and prioritize testing efforts towards scenarios with higher potential impact. Application to Test Case Prioritization Consider complex systems with numerous possible error states. We can estimate the conditional probability of encountering specific errors given certain user inputs or system configurations. This allows testers to prioritize test cases based on the likelihood of triggering critical errors, optimizing testing efficiency. Application to Performance Testing Performance bottlenecks often manifest under specific loads. We can use conditional probability to estimate the likelihood of performance degradation given concurrent users or specific data sizes. This targeted testing approach helps pinpoint performance issues that occur under realistic usage conditions. Beyond the Examples These are just a few examples. Conditional probability has wider applications in areas like: Mutation testing: Estimating the probability of a test case revealing a mutation given its specific coverage criteria. Statistical testing: Analyzing hypothesis testing results and p-values in the context of specific assumptions and data sets. Machine learning testing: Evaluating the conditional probability of model predictions being wrong under specific input conditions. Remember: Choosing the right "given" conditions is crucial for meaningful results. Conditional probability requires understanding dependencies between events in our software system. Combining conditional probability with other testing techniques (e.g., combinatorial testing) can further enhance testing effectiveness. Bayes' Theorem The definition of conditional probability provides the foundation for understanding the relationship between events. Bayes' theorem builds upon this foundation by allowing us to incorporate additional information to refine our understanding in a dynamic way. It allows us to dynamically update our beliefs about the likelihood of events (e.g., bugs, crashes) based on new evidence (e.g., test results, user reports). This dynamic capability may unlock numerous applications for our testing approach. Demystifying Bayes' Theorem: Beyond the Formula Imagine we suspect a specific functionality (event B) might harbor a bug. Based on our current understanding and past experiences (prior probability), we assign a certain likelihood to this event. Now, we conduct a series of tests (evidence A) designed to uncover the bug. Bayes' theorem empowers us to leverage the results of these tests to refine our belief about the bug's existence (posterior probability). It essentially asks: "Given that I observed evidence A (test results), how does it affect the probability of event B (bug) being true?" While the formula, P(B | A) = [ P(A | B) * P(B) ] / P(A), captures the essence of the calculation, a deeper understanding lies in the interplay of its components: P(B | A): Posterior probability - This represents the updated probability of event B (bug) given evidence A (test results). This is what we ultimately seek to determine. P(A | B): Likelihood - This signifies the probability of observing evidence A (test results) if event B (bug) is actually true. In simpler terms, it reflects how effective our tests are in detecting the bug. P(B): Prior probability - This represents our initial belief about the likelihood of event B (bug) occurring, based on our prior knowledge and experience with similar functionalities. P(A): Total probability of evidence A - This encompasses the probability of observing evidence A (test results) regardless of whether event B (bug) is present or not. It accounts for the possibility of the test results occurring even if there's no bug. Visualizing the Power of Bayes' Theorem Imagine a scenario where we suspect a memory leak (event B) in a specific code change (A). Based on past experiences, we might assign a prior probability of 0.1 (10%) to this event. Now, we conduct tests (evidence A) that are known to be 80% effective (P(A | B) = 0.8) in detecting such leaks, but they might also occasionally yield positive results even in the absence of leaks (P(A) = 0.05). Applying Bayes' theorem with these values: P(B | A) = [0.8 * 0.1] / 0.05 = 1.6 This translates to a posterior probability of 64% for the memory leak existing, given the observed test results. This significant increase from the initial 10% prior probability highlights the power of Bayes' theorem in updating beliefs based on new evidence. Application to Test Effectiveness Analysis Bayes' theorem can be a useful tool for analyzing the effectiveness of individual test cases and optimizing our testing resources. Let's delve deeper into this application: 1. Gathering Data Identify known bugs (B): Compile a list of bugs that have been identified and fixed in our system. Track test case execution: Record which test cases (A) were executed for each bug and whether they successfully detected the bug. 2. Calculating Likelihood For each test case-bug pair (A, B), calculate the likelihood (P(A | B)). This represents the probability of the test case (A) detecting the bug (B) if the bug is actually present. We can estimate this likelihood by analyzing historical data on how often each test case successfully identified the specific bug or similar bugs in the past. 3. Estimating Prior Probability Assign a prior probability (P(B)) to each bug (B). This represents our initial belief about the likelihood of the bug existing in the system before any new evidence is considered. This can be based on factors like the bug's severity, the code complexity of the affected area, or historical data on similar bug occurrences. 4. Applying Bayes' Theorem For each test case, use the calculated likelihood (P(A | B)), the prior probability of the bug (P(B)), and the total probability of observing the test result (P(A)) to estimate the posterior probability (P(B | A)). This posterior probability represents the updated probability of the bug existing given that the specific test case passed. 5. Interpreting Results and Taking Action High posterior probability: If the posterior probability is high, it suggests the test case is effective in detecting the bug. Consider keeping this test case in the suite. Low posterior probability: If the posterior probability is low, it indicates the test case is unlikely to detect the bug. We might consider: Refactoring the test case: Improve its ability to detect the bug Removing the test case: If it consistently yields low posterior probabilities for various bugs, it might be redundant or ineffective. Example Imagine we have a test case (A) that has successfully detected a specific bug (B) in 70% of the past occurrences. For illustrative purposes, we assign the sample value for the prior probability of 20% to the bug existing in a new code change. Applying Bayes' theorem: P(B | A) = [0.7 * 0.2] / P(A) Since P(A) depends on various factors and might not be readily available, it's often ignored for comparative analysis between different test cases. There are three main reasons for this. The first is normalization. P(A) represents the overall probability of observing a specific test result, regardless of whether the bug is present or not. This value can be influenced by various factors beyond the specific test case being evaluated (e.g., overall test suite design, system complexity). The second reason is the focus on relative performance. When comparing the effectiveness of different test cases in identifying the same bug, the relative change in the posterior probability (P(B | A)) is crucial. This change signifies how much each test case increases our belief in the bug's presence compared to the prior probability (P(B)). The third reason is simplification. Ignoring P(A) simplifies the calculation and allows us to focus on the relative impact of each test case on the posterior probability. As long as all test cases are subjected to the same denominator (P(A)), their relative effectiveness can be compared based solely on their posterior probabilities. By calculating the posterior probability for multiple test cases targeting the same bug, we can: Identify the most effective test cases with the highest posterior probabilities. Focus our testing efforts on these high-performing tests, optimizing resource allocation and maximizing bug detection capabilities. Remember: The accuracy of this analysis relies on the quality and completeness of our data. Continuously update our data as we encounter new bugs and test results. Bayes' theorem provides valuable insights, but it shouldn't be the sole factor in test case selection. Consider other factors like test coverage and risk assessment for a holistic approach. Wrapping Up Probability is a powerful tool for our testing activities. This article starts with probability basics, continues with conditional probabilities, and finishes with Bayes' theorem. This exploration of probability provides a solid foundation to gain deeper insights into software behavior, optimize testing efforts, and ultimately contribute to building more reliable and robust software. Software testing is about predicting, preventing, and mitigating software risks. The journey of software testing is a continuous pursuit of knowledge and optimization, and probability remains our faithful companion on this exciting path. Remember, it's not just about the formulas: it's about how we apply them to better understand our software.

By Stelios Manioudakis

CORE

Mastering Test Code Quality Assurance

Over the years, many articles have highlighted the importance of unit and integration tests and their benefits. They enable quick and accurate identification of errors, simplify the debugging process, support safe refactoring, and prove invaluable during code reviews. These tests can also significantly reduce development costs, help catch mistakes early, and ensure the final product aligns well with its specifications. As such, testing is often viewed as a central part of the development process. However, within the developer community, it's become clear in recent years that merely having unit and integration tests isn't enough. A growing number of blog posts and articles emphasize the need for well-structured and formatted tests. So, why is this aspect so crucial? Best Practices In short, poorly formatted tests or those exhibiting anti-patterns can significantly hamper a project's progress. It's not just my perspective. Many articles stress the significance of well-structured testsand provide best practices and insights on this topic. One element that frequently emerges as pivotal in these discussions is the naming of tests. Two articles in particular, Anatomy of a Good Java Test and Importance of Unit Testing underscore the crucial role of effective test naming. They advise against using the word "test" in test names, suggesting that appropriate naming can clearly describe the test's objective or what it intends to verify. Additionally, the article Clean Unit Testing highlights not only the naming of test methods but also the importance formaintainability of correct naming and ordering test variables. Branching out from naming assertions is another cornerstone in testing best practices. Take, for instance, the article 7 Tips for Writing Better Unit Tests in Java that highlights the advantage of using assertions over print statements. Other industry experts often emphasize limiting the number of assertions and correctly positioning them within a single test. The AAA pattern (Arrange, Act, Assert) is the perfect example of this intention: positioning assertions at the end of the test method ensures clarity and readability for other developers. Moreover, the transparency of the assertions themselves is also important. For instance, they should comewith descriptive messages. In fact, there are more suggestions to keep in mind: Appropriate usage of mocks and stubs. Avoiding "if" statements in test blocks. Focusing on a single case in each unit Making tests as isolated and automated as possible. Maintaining high test and code coverage. Testing negative scenarios and borderline cases, in addition to positive ones. Avoiding non-deterministic results and flaky tests Avoiding unit-test anti-patterns Yet, the realm of best practices is ever-evolving, and this list isn't exhaustive. New best practices continue to emerge. For example, the recent idea about the layout of tests highlights the importance of structuring both unit and integration tests within the source code. It's not just about refactoring tests anymore but also about organizing them systematically within the source code. In summation, as you can see, the community provides a variety of best practices for creating quality tests. The real question, however, is: Are these principles just theoretical, or are there practical solutions that can help us achieve such quality? Gap Identification Yes, I'm referring to static analyzers. Let's briefly examine the most widely used ones, even though there are many similar tools available. I will focus only on rules and checks that help to address at least some of the best practices discovered previously. Checkstyle Checkstyle is a development tool that helps programmers write Java code that adheres to a coding standard. In other words, Checkstyle is a static code analysis tool (linter) used in the Java world. Although Checkstyle doesn't provide features specifically tailored for tests, many of its features areapplicable to test code, just as they are to production code. It can assist with Javadoc comments, indentation, line length, cyclomatic complexity, etc. However, to the best of my knowledge, the only feature related to tests is the ability to enforce the test names convention by developing a specific checker. So, yes, before using it, you need to develop your own checker first.Thus, while Checkstyle is a general tool that focuses solely on Java code, it doesn't specifically address issues with tests. It doesn't consider specific rules related to assertion checks, identification of anti-patterns, or maintaining the layout of tests - all of which are essential to keep tests consistent and clear in line with industry requirements and best practices. PMD PMD is one more source code analyzer similar to Checkstyle. It finds common programming flaws like unused variables, empty catch blocks, unnecessary object creation, and so forth. While it supports many different languages, we are only interested in Java. PMD, compared with Checkstyle, has many more rules that check test quality, for example (but not limited to): JUnitAssertionsShouldIncludeMessage requires JUnit assertions to include a message. JUnitTestContainsTooManyAsserts checks if the JUnit or TestNG test contains too many assertion statements. JUnitTestsShouldIncludeAssert checks that JUnit tests include at least one assertion. TestClassWithoutTestCases checks that test classes have at least one testing method. UnnecessaryBooleanAssertion checks that JUnit assertions are used correctly without assertTrue(true) statements (line-hitter anti-pattern detection.) Here is a short example of test violations that PMD can find: Java public class Foo extends TestCase { public void testSomething() { // [JUnitAssertionsShouldIncludeMessage] Use the form: // assertEquals("Foo does not equals bar", "foo", "bar"); // instead assertEquals("foo", "bar"); } //[TestClassWithoutTestCases] Consider adding test methods if it is a test: public class Bar extends TestCase {} public class MyTestCase extends TestCase { // Ok public void testMyCaseWithOneAssert() { boolean myVar = false; assertFalse("should be false", myVar); } //[JUnitTestsShouldIncludeAssert] //Bad, don't have any asserts public void testSomething() { Bar b = findBar(); b.work(); } //[JUnitTestContainsTooManyAsserts]: //Bad, too many asserts (assuming max=1) public void testMyCaseWithMoreAsserts() { boolean myVar = false; assertFalse("myVar should be false", myVar); assertEquals("should equals false", false, myVar); //[UnnecessaryBooleanAssertion] Bad, serves no real purpose - remove it: assertTrue(true); } However, all these checks are designed primarily for JUnit assertions and, in some cases, for AssertJ. They don't support Hamcrest assertions, which are widely adopted in the industry. Also, while PMD can check method names, these checks are relatively simple. They focus on aspects such as method name length, avoiding special characters like underscores, and adhering to camel case naming conventions. Consequently, these checks are primarily intended for production code only and don't examine specific test name patterns. Moreover, to the best of my knowledge, PMD doesn't identify structural mistakes or verify the correct placement of methods. Thus, PMD provides a rather limited set of checks for tests. Sonar Qube SonarQube is also a widely used tool for checking code quality. SonarQube has a lot of rules similar to PMD that can be applied to tests, for example: TestCases should contain tests. Literal boolean values and nulls should not be used in assertions. Assertions should not compare an object to itself. Test assertions should include messages. Test methods should not contain too many assertions. Similar tests should be grouped in a single Parameterized test. At the time of writing this text, there are around 45 rules specifically designed for tests. As you might have noticed, SonarQube has more rules than PMD, although many of them overlap. However, to the best of my knowledge, SonarQube doesn't check Hamcrest assertions and doesn't maintain the layout of tests. It also doesn't show much concern about checking test anti-patterns. Others Actually, there are other tools available for detecting issues related to test quality. Some notable ones include: SpotBugs checks for correct usage of setUp/tearDown methods, empty test cases, andimproper use of assertions. ErrorProne examines test signatures and forbids the use of "test" in test names, identifies redundant methods without @Test and @Ignore and offers some other test-related checks. MegaLinter and Qulice primarily combine previously mentioned linters like PMD and Checkstyle. Essentially, they just bundle checks from other linters. Coverity is a proprietary tool that has numerous checks, including those for assertions and various resource leaks. However, some users argue that its features are similar to those PMD and SpotBugs. Jtest is another proprietary tool that has a comprehensive set of features. This includes checks for assertion statements, initialization methods, and more. The complete list of checks can be found here. There are numerous other tools, including Checkmarx Glossary, Klocwork, CodeSonar, among many others, that we simply can't cover in this article. In summary, tools like Checkstyle, PMD, SonarQube, and others offer numerous rules to ensure test code quality. However, noticeable gaps exist in their ability to tackle certain test-related issues. Checkstyle is primarily designed for Java production code, and its features for tests are limited. This often requires users to develop their own checkers for specific scenarios. PMD has a robust set of rules for JUnit assertions, yet it doesn't support popular frameworks like Hamcrest or method naming patterns. SonarQube provides an extensive rule set, which overlaps with PMD in many areas. However, it lacks some vital test checks, including those for Hamcrest assertions and test anti-patterns. Other tools have their own limitations, or they are proprietary. Significantly, none of the aforementioned tools focus on the proper placement and naming of test classes. Thus, even though these tools provide a foundation for test code quality, there's a notable gap in terms of aligning with industry test standards and best practices. Introducing jtcop To address the aforementioned gaps, we developed a new static analyzer called jtcop that focuseson test quality in Java projects. It is a simple Maven plugin that checks tests for common mistakes and anti-patterns. We use it in our projects, and it has helped us maintain consistent and clear tests. It also speeds up PR reviews significantly by preventing recurring comments about issues like improper test placement or naming. Although, we don't think our rules are the only good way to set up tests, so feel free to share your ideas and suggestions by submitting tickets and PRs. In the following, I'll explain how jtcop fits into the landscape of static analysis tools, which checks it utilizes, and how it can assist you in youreveryday programming. Test Names I'm sure you know there are many ways to name your test. For example, you can find various test naming conventions or even some threads that have lengthy discussions on how to do it correctly. Here is just a short summary of how you can name your tests: Pattern Example methodName_stateUnderTest_expected add_negativeNumbers_throwsException() when_condition_then_expected when_ageLessThan18_then_isUnderageIsTrue() given_precondition_when_action_then_result given_userIsAdmin_when_deleteIsCalled_then_deleteSuccess() test[methodName] testAdd() or testIsUnderage() should_expectedBehavior_when_condition should_throwException_when_negativeNumbersAreAdded() methodName_expected add_returnsSum() or isUnderage_returnsTrue() canAction canDeleteUser() or canCalculateSum( methodName_doesExpectedBehavior add_doesReturnSum() or isUnderage_returnsTrue() verbCondition (or verbResult) calculatesSum() or deletesSuccessfully() jtcopprefers the last pattern: Test names should use the present tense without a subject. For example, if you're testing a class Animal with a method eat(), the test name should be eats(). If you need to add more context, do it after the verb – for instance, eatsApplesOnly(). Test names should use camelCase. Name shouldn't use the word "test", as it is redundant. The @Test annotation is sufficient. Special characters like _ and $ are forbidden. Correct Names Incorrect Names eats() testEats() eatsApplesOnly() TestEatsApplesOnly() runsQuickly() _runsQuickly() jumpsOverFence() jumps_over_fence() drinksWater() drinks$Water() sleepsAtNight() sleepsZZZ() chewsGum() test_chewsGum() listensToMusic() listens_To_Music() walksInPark() WalksInPark() barksLoudly() barks__loudly() This style has been chosen by many developers and is widely used in numerous projects. If you prefer a different pattern for test naming, just let us know, and we'll be happy to add it to the plugin. Corresponding Production Class Now, let's imagine we have a test class named SumTest.java with the test method checksSum(). But what if the test occasionally fails? Most would attempt to locate the issue and find the original class where the problem occurred. But which class is it? The first guess would likely be Sum.java, right? Yet, you might not find it, perhaps because the production class is named something like Addition.java or Calculator.java. This mismatch in naming conventions can lead to significant confusion and longertroubleshooting times. In other words, if you have a test class named SumTest.java and the corresponding production class is Addition.java, it can be very confusing. The more appropriate name for the test class would be AdditionTest.java. Essentially, the name of the test class isn't merely a label; it serves as a pointer to the production class, helping developers pinpoint potential issues.This is where jtcop comes into play. It helps ensure that your tests are consistent with your production classes and suggests appropriate naming conventions for them, effectively addressing the problem described. If you're further interested in this issue, you can read about it here.The only exception in this case is integration tests. They are usually named like AdditionIT.java or AdditionIntegrationTest.java. However, they should be placed in a separate package, such as it, and have an appropriate suffix like IT or ITCase. Test Methods Only The next check is rather strict and is still considered an experimental feature. However, the rule itself is simple: test classes should contain methods that are only annotated with the @Test annotation. You might wonder what to do with initialization methods or common code shared among different test cases. The answer isn't straightforward and this rule is designed to guide you with it. There aren't actually many options available. I'm referring to methods such as static initialization methods, setup methods @BeforeEach and @AfterEach annotations, JUnit extensions, and Fake Objects. The approach you choose for initializing your tests will determine their quality. Static Methods The first idea that comes to mind is using static methods. Developers often use static methods to configure a common setup for several tests in the class. Here's a simple example: Java @Test void calculatesSum(){ Summator s = init(); Assertions.assertEquals( 2, sum(1, 1), "Something went wrong, because 1 + 1 != 2" ); } private static Summator init(){ Summator s = new Summator(); // Setup return s; } At first glance, it might seem like a good solution, but it does have inherent problems. When such a method is used within a single class, it's usually not a major concern, even though static methods typically lead to low cohesion and tight coupling. However, issues arise when you begin to use it across multipleclasses or try to consolidate such methods into a centralized TestUtils.java class. In this case, the approach with static methods can become problematic: It can lead to confusion for developers since TestUtils.java doesn't correspond to any class in the production code. TestUtils.java might be considered an anti-pattern. Thus, jtcop deems static methods in tests and utility classes as dangerous and prohibits them. If you attempt to run jtcop against the previous code sample, you'll receive the following warning message: Shell All methods should be annotated with @Test annotation. SetUp and TearDown Methods The next widely-used approach involves the so-called "setUp" methods. By "setUp" methods, I'm referring to those annotated with @BeforeAll, @BeforeEach, @AfterAll, or @AfterEach. An example of using these annotations is as follows: Java Summator s; @BeforeEach void setUp(){ s = new Summator(); // Setup } @Test void calculatesSum(){ Summator s=init(); Assertions.assertEquals( 2, sum(1,1), "Something went wrong, because 1 + 1 != 2" ); } This approach makes the situation even worse for many reasons. The most obvious reason, familiar to most developers, is the need to "jump" between test methods and the initialization part. Then, over time, as the codebase grows and changes and as the number of test cases in the test class increases, developers may become unaware of the setup/teardown that happens for each test and may end up with setup code that is unnecessary for certain tests, thus violating the principle of keeping tests minimal and setting up only what is needed. Next, using such methods can introduce another problem. They can lead to ashared state between tests if not managed properly. This harms test isolation, an extremely important quality of any test, which in turn can result in flaky tests. Moreover, using @BeforeAll and @AfterAll use static methods, which inherit all the disadvantages of the previous approach. Hence, jtcop doesn't allow the use of such setUp/tearDown methods. Test Extensions Now, let's examine the approach supported by jtcop. JUnit 5 offers Test Extensions that allow for the creation of custom extensions. These extensions. can be used to configure setup and teardown logic for all the tests in a class. Java @ExtendWith(SummatorExtension.class) public class SumTest { @Test void calculatesSum(Summator s) { Assertions.assertEquals( 2, s.sum(1, 1), "Something went wrong, because 1 + 1 != 2" ); } class SummatorExtension implements ParameterResolver { @Override public boolean supportsParameter(ParameterContext pctx, ExtensionContext ectx) { return pctx.getParameter().getType() == Summator.class; } @Override public Object resolveParameter( Summator s =new Summator(); // Setup return s; } Extensions offer a way to craft more modular and reusable test setups. In this scenario, we've bypassed the need for utility classes, static methods, and shared states between tests. These extensions are easily reused across a multitude of test classes and standalone unit tests. What's more, theseextensions often have insight into the current test class, method, annotationsused, and other contextual details, paving the way for versatile and reusablesetup logic. Fake Objects Another method for test configuration and setup that jtcop supports is the use of Fake objects, as recommended here. These are positioned with other production objects, yet they provide a unique"fake" behavior. By leveraging these objects, all setup can be handled directly in a test, making the code cleaner and easier to read. Java abstract class Discount { // Usually we have rather complicated // logic here for calculating a discount. abstract double multiplier(); static class Fake extends Discount { @Override double multiplier() { return 1; } } public class PriceTest { @Test void retrievesSamePrice() { Price p = new Price(100, new Discount.Fake()); Assertions.assertEquals( 100, p.total(), "Something went wrong; the price shouldn't have changed" ); } Fake objects often sit alongside production code, which is why jtcop doesn't classify them as test classes. While mixing production and test code might seem questionable, Fake objects aren't exclusively for testing; you might sometimes integrate them into your production code, too. Many projects have embraced the use of Fake objects, finding it a practical way to set up tests. Additionally, this strategy eliminates the need for using Mock frameworks with intricate initialization logic. Test Assertions jtcop also underscores the need to validate assertions in tests. Several tools out there offer similar checks. Yet, many of them focus solely on JUnit assertions or only catch high-level errors. jtcop supports both Hamcrest and JUnit assertions and adheres to stricter guidelines for assertions. To paint aclearer picture, let's dive into a few code snippets. Java @Test void calculatesSum(){ if(sum(1, 1) != 2){ throw new RuntimeException("1 + 1 != 2"); } } This code snippet lacks any assertions, meaning jtcop will warn about it. Check out the next snippet as a proper replacement, and note the use of the Hamcrest assertion. Java @Test void calculatesSum(){ assertThat( "Something went wrong, because 1 + 1 != 2", sum(1, 1), equalTo(2) ); } Pay attention to the explanatory messages in the assertion Something went wrong, because 1 + 1 != 2 from the code above. They're essential. Without such messages, it can sometimes be challenging to understand what went wrong during test execution, which can puzzle developers. For instance, consider this real example. I've simplified it for clarity: Java @Test void checksSuccessfully(){ assertThat( new Cop(new Project.Fake()).inspection(), empty() ); } Now, suppose this test fails. In that scenario, you'll receive the following exception message: Shell Expected: an empty collection but: <[Complaint$Text@548e6d58]> Not very informative, right? However, if you include an explanatory message in the assertion: Java void checksSuccessfully(){ assertThat( "Cop should not find any complaints in this case, but it has found something.", new Cop(new Project.Fake()).inspection(), empty() ); } With this inclusion, you're greeted with a far more insightful message: Shell java.lang.AssertionError: Cop should not find any complaints in this case, but it has found something. Expected: an empty collection but: <[Complaint$Text@548e6d58]> In a perfect world, we'd offer more details — specifically, some context. This sheds light on initialization values and provides developers with valuable hints. Line Hitters The last feature I'd like to spotlight is the Line Hitter anti-pattern detection. At first glance, the tests cover everything and code coverage tools confirm it with 100%, but in reality the tests merely hit the code, without doing any output analysis. What this means is that you might stumble upon a test method in a program that does not really verify anything. Take this for instance: Java @Test void calculatesSum(){ sum(1, 1); } This typically happens when a developer is more into their code coverage numbers than genuinely ensuring the robustness of the test. There are tools that can spot when assertions are missing in tests. But, as you know, developers might always find a way around: Java @Test void calculatesSum(){ sum(1,1); assertThat( "I'm just hanging around", true, is(true) ); } Yep, that’s our "Line Hitter" again, only this time, it's wearing the disguise of an assertion statement. Luckily, jtcop can detect such tests and flag them as unreliable. Setting up jtcop To get started with jtcop, simply add the plugin to your build configuration file. If you're using Maven, here's how you can do it: XML <build> <plugins> <plugin> <groupId>com.github.volodya-lombrozo</groupId> <artifactId>jtcop-maven-plugin</artifactId> <version>1.1.1</version> <executions> <execution> <goals> <goal>check</goal> </goals> </execution> </executions> </plugin> </plugins> </build> By default, the plugin operates in the verify phase, so there is no need to specify it. However, if you wish to modify it, simply add the desired phase to the execution section. Then, to run jtcop, use the mvn jtcop:checkcommand. If you stumble upon an issue, say, a test lacking a corresponding productionclass, you'll get a clear error message: Shell [ERROR] Test SumTest doesn't have corresponding production class. [ERROR] Either rename or move the test class ./SumTest.java. [ERROR] You can also ignore the rule by adding @SuppressWarnings("JTCOP.RuleAllTestsHaveProductionClass") annotation. [ERROR] Rule: RuleAllTestsHaveProductionClass. [ERROR] You can read more about the rule here: <link> Similarly, for the "Line Hitter" pattern previously mentioned: Shell [ERROR] Method 'calculatesSum' contains line hitter anti-pattern. [ERROR] Write valuable assertion for this test. [ERROR] You can also ignore the rule by adding @SuppressWarnings("JTCOP.RuleLineHitter") annotation. [ERROR] Rule: RuleLineHitter. [ERROR] You can read more about the rule here: <link> By default, jtcop will halt the build if it detects issues with your tests. If you only want to use it to highlight problems without interrupting the build, you can configure jtcop to display only warning messages by adjusting the failOnError property. XML <configuration> <failOnError>false</failOnError> </configuration> However, I highly recommend keeping the default setting to maintain high-quality tests. Experimental Features As I mentioned earlier, some features are still experimental. To try them out, just add the following configuration to your pom.xml file: XML <configuration> <experimental>true</experimental> </configuration> Once done, all experimental features will be active in your project, ensuring cleaner and more organized tests. Benefits jtcop has already helped us in several ways: Code Review: The primary issue addressed by jtcop is the frequent appearance of comments such as "place this test class here," "rename this test method," or "that's a testing anti-pattern." `jtcop` saves time and aids developers in resolving these issues before even making a PR into arepository. Onboarding: Another advantage we've observed is that well-structured and appropriately named test methods not only facilitate code understanding and maintenance but also reduce the time spent explaining or documenting code style guides. As a result, we often receive well-formatted pull requests from new team members with little to no additional guidance. Consistency: jtcop ensures our tests remain consistent across numerous projects. So, when you delve into a project that uses jtcop, it becomes significantly easier to comprehend its workings and start contributing to it. Overall, integrating `jtcop` has significantly streamlined our processes, enhancing collaboration and understanding across our development projects. Future Plans Looking ahead, we're preparing to enhance jtcop with additional rules. One of our primary focuses is to address several anti-patterns like the ones highlighted in this StackOverflow thread. Just to name a few: The Mockery: Tests that have too many mocks. Excessive Setup: Tests that demand extensive setup. Wait and See: Tests that need to pause for a specific duration before verifying if the tested code works as intended. It's worth noting that these are just a few examples; there's a broader spectrum of anti-patterns we're considering. Additionally, we've also encountered issues with projects that have many tests written in various styles. At times, it's incredibly tedious to address these issues manually. Thus, another viable avenue is developing an application that will automatically solve most of these problems. So, if you have ideas or suggestions, please don't hesitate to open an issue or submit a pull request in our repository and share your thoughts with us. We're always eager to get feedback or contributions from the community. Feel free to fork it if you want and craft your own test checkers that fit your needs, or simply use jtcop as is.

By Volodya Lombrozo

Test Automation Guilds: Advancing Excellence in Testing

Have you ever found yourself in the position of a test engineer embedded in one of the Agile engineering teams? While you have daily interactions with peers, connecting with them on a profound level for the successful execution of job duties might be challenging. Although there is a shared goal to release features successfully, we often experience isolation, especially while others, like developers, find comfort within the team. In the realm of dispersed Agile teams with time zones adding an extra layer of complexity, the longing for a team to resonate with, connect with, and brainstorm on all test automation challenges is prevalent. In the expansive landscape of test automation, the creation of an automation guild is more than just collaboration; it stands as a testament to the resilience of SDETs working across diverse time zones and Agile teams. Through this guide, I aim to share the benefits and challenges overcome, the enrichment of test engineers or SDETs, and the establishment of a collective force dedicated to advancing excellence in testing. Breaking Silos In a world where time zones separate teams and Agile methodologies dictate the rhythm of development, test engineers face a unique challenge. Even though they are part of an Agile team with a shared goal, i.e., successful release, they must navigate independently without a clear direction or purpose. The guild, however, becomes a bridge across these temporal gaps, offering a platform for asynchronous collaboration. It not only allows them to demo their progress, accomplishments, and new utility that can be leveraged by others but also their challenges and blockers. It will surprise you to see how often those obstacles are common among other guild members. Now that they have each other, all heads come together to brainstorm and find common, effective solutions for any testing problem. Fostering Through Training and Contribution As important as regular guild meet-ups and collective commitment are, continuous learning and training initiatives are equally vital to empower test engineers to contribute effectively. From workshops on emerging testing methodologies to skill-building webinars, the guild evolves into a learning haven where members grow together, ensuring each test engineer is equipped to make a meaningful impact. It enhances members’ efficiency by reducing redundant efforts. Understanding what others are working on and what tools are available for use, such as common utilities and shared definitions, enables them to save time by avoiding duplication of efforts and contribute more effectively. This isn’t just about individual efficiency; it’s a strategic move toward collective empowerment. Grow Your Network and Your Profile Within the guild, networking is not confined to individual teams. It offers the creation of a network that spans across Agile teams, allowing Test Engineers to understand overall solutions from diverse perspectives. This isn’t just about sharing knowledge; it’s about broadening domain knowledge. Turning new members into seasoned members who can then mentor new juniors, ensuring that the guild is not just a community but a mentorship ecosystem that thrives on collective wisdom. If there’s one aspect that has been repeatedly demonstrated in the guild, it would be that challenges are not roadblocks but opportunities for innovation and collaboration. The guild stands as a testament to the fact that, even in the world of test automation, where distances and time zones pose challenges, excellence can be achieved through collective strength. Automation guild is not just about crafting code; it’s about crafting a community that advances excellence in testing, collectively and collaboratively. The future, as envisioned through the chronicles, is one where Test Engineers, regardless of time zones, work seamlessly in a guild that stands as a beacon of innovation, knowledge-sharing, and collective growth.

By Priyanka Chauhan

Unit Testing: Java Streams and Lambdas

Unit testing is an indispensable practice for Java developers (any developer really). The primary benefit of unit testing a pipeline is the early detection of bugs, which is less expensive to fix. It not only improves the overall quality of the code but also makes future maintenance easy. Using lambdas specifically with streams in Java makes code concise and readable. Streams are excellent for filtering, mapping, sorting, and reducing operations. The elements in the sequence are better processed in a declarative and functional manner, something which lambdas exactly fit into. The anonymous functions that are written with lambdas not only facilitate a functional style of programming but also enhance the expressiveness of the code. Streams and lambdas were introduced in Java after JDK 8, and since then, Java developers have used these features frequently in their projects. The question around these components that the article tries to address is, "How can you unit test Java streams and lambdas?" Importance of Unit Testing Pipelines and Lambdas In unit testing, an individual piece of component of a software application is tested separately. This small unit of code typically is a function, method, or subroutine. The testing mechanism is automated so that they can be done repeatedly and quickly. The test cases are usually written by developers and integrated into the CI/CD pipeline in the development process. The code can be isolated and problems can be easily identified if we use lambda because the essence of it is to make the program functional, more modular, and reusable — something which makes it friendly for unit testing pipelines. Unit Testing Stream Pipelines Since stream pipelines combine with lambdas to form a single unit, it is not obvious how to effectively unit test the pieces of the pipeline. I have always followed these two guidelines to unit test those stream pipelines: If the pipeline is simple enough, it can be wrapped in a method call, and it is enough to unit test the method call. If the pipeline is more complex, pieces of the pipeline can be called from support methods, and the support methods can be unit-tested. For example, let's say we have a stream pipeline that maps all the letters of the word to uppercase and is wrapped in a java.util.function.Function<T,R> as below: Function<List<String>, List<String>> allToUpperCase = words -> words.stream().map(String::toUpperCase).collect(Collectors.toList()); Now, the unit test for this stream pipeline can be easily written as ordinary unit testing of the allToUpperCase. @Test public void testAllToUpperCase() { List<String> expected = Arrays.asList("JAVA8", "STREAMS"); List<String> result = allToUpperCase.apply(Arrays.asList("java8", "streams")); assertEquals(expected, result); } The stream above can be wrapped in a regular function, as seen below. Also, an ordinary unit test can be written against this function: public List<String> convertAllToUpperCase(List<String> words) { return words.stream().map(String::toUpperCase).collect(Collectors.toList()); } Unit Testing Lambdas Believe me — it is very likely that you will encounter complex unit testing in real-world programming. The unit testing with complex lambdas, similar to unit testing of stream pipelines, can be simplified with the following practices: Replace a lambda that needs to be tested with a method reference and an auxiliary method. Then, test the auxiliary method. For example, I have a stream pipeline that involves a somewhat complex lambda and a mapping class for the given string class name. public static Class[] mapClasses(final List<String> exceptions) { return exceptions.stream().map(className -> { try { return Class.forName(className); } catch(Exception ex) { LOGGER.error("Failed to load class for exceptionWhiteList: {}", className); } return null; }).toArray(Class[]::new); } Here, the key point to test is whether the expression for transforming a string class name to a Class object is working. As mentioned above, this can replace the lambda expression with a method reference, along with an auxiliary method that can be placed in a companion class, as shown below: public static Class[] mapClassesBetter(final List<String> exceptions) { return exceptions.stream().map(LambdaTester::mapClass).toArray(Class[]::new); } public static Class mapClass(String className) { try { return Class.forName(className); } catch(Exception ex) { LOGGER.error("Failed to load class for name: {}", className); } return null; } Now, the key element of the original lambda is that it can be tested directly: @Test public void testMapClass() throws ClassNotFoundException { assertEquals(null, mapClass("a")); assertEquals(null, mapClass("apple")); assertEquals(Object.class, mapClass("java.lang.Object")); } Conclusion Writing unit tests is one of the core parts of software development. With new features introduced after JDK 8, it has become easier to write code concisely and declaratively. However, the proper use of features like streams and lambda brings value and, of course, makes writing unit tests easier. If you have any additional guidelines for unit testing these features, don't stop yourself from sharing them in the comments. Until next time, happy coding! Learn more about the best Java unit testing frameworks. The source code for the examples presented above is available on GitHub.

By Yogen Rai

Munit: Parameterized Test Suite

The purpose of this use case is to explain how to define different RAML data types, define the business-related status code for request payload validation, define the single unit test case for multiple field validation with dynamic request payload, and how to use the parameterized test suite. With a parameterized test suite, we can write a reusable unit test case. It can help to write and test multiple scenarios based on different inputs with a single test case. This can be more useful when we are writing a test case to validate the request and response payload. RAML Data Types Definition Define different types of data types for a request payload. In the below example, I covered integer, string, pattern, enum, array, object, datetime, datetime-only, and time-only. YAML #%RAML 1.0 DataType type: object properties: employeeId: type: integer required: true minimum: 8 firstName: type: string required: true minLength: 1 maxLength: 10 pattern: ^[A-Za-z]* lastName: type: string required: true minLength: 1 maxLength: 10 pattern: ^[A-Za-z]* email: pattern: ^.+@.+\..+$ gender: enum: [male, female] default: male required: true dateOfBirh: type: date-only required: true addresses: type: array minItems: 1 items: type: object properties: isPermanent: type: boolean required: true street: type: string required: true minLength: 5 maxLength: 50 pattern: ^[A-Za-z ]* district: type: string required: true minLength: 3 maxLength: 20 pattern: ^[A-Za-z]* state: type: string required: true minLength: 5 maxLength: 15 pattern: ^[A-Za-z]* pinNumber: type: integer required: true minimum: 6 province: type: string required: true minLength: 1 maxLength: 10 pattern: ^[A-Za-z]* phoneNumber: type: string required: true minLength: 1 maxLength: 13 pattern: ^\s*|^(0|91)?[6-9][0-9]{9}$ created: type: datetime format: rfc3339 required: true createdDateTime: type: datetime-only required: true createdTime: type: time-only required: true Sample Request Payload Based on the RAML definition, prepare a valid request payload. JSON { "employeeId": 12345678, "firstName": "Ankur", "lastName": "Bhuyan", "email": "ankur.bhuyan@gmail.com", "gender": "male", "dateOfBirh": "2000-04-01", "addresses": [ { "isPermanent": true, "street": "teachers colony", "district": "Sunitpur", "state": "Assam", "pinNumber": 784507, "province": "Tezpur", "phoneNumber": "+919590951234" } ], "created": "2016-02-28T12:30:00.090Z", "createdDateTime": "2016-02-28T12:30:00", "createdTime": "12:30:00" } Configure Parameterized Test Suite for Valid Scenarios This is a valid test case scenario for which the parameterized inputs are defined. Based on the parameterization "name" defined, the test result name will be prepared (once the test case is executed). We have defined only one property with which the input payload for the test case will be prepared, which will vary based on the test scenario we will cover. XML <munit:config name="apikit-valid-test-suite.xml" > <munit:parameterizations > <munit:parameterization name="001" > <munit:parameters > <munit:parameter propertyName="caseNumber" value="001" /> </munit:parameters> </munit:parameterization> </munit:parameterizations> </munit:config> Configure Parameterized Test Suite for Invalid Scenarios This is an invalid test case scenario for which the parameterized inputs are defined. We can add as many numbers of the parameter (like caseNumber, expectedType, expectedCode) based on the use case. This is an example to show how we can define multiple parameters for a single test case. XML <munit:config name="apikit-invalid-test-suite.xml"> <munit:parameterizations > <munit:parameterization name="001" > <munit:parameters > <munit:parameter propertyName="caseNumber" value="001" /> <munit:parameter propertyName="expectedType" value="REQUIRED_KEY" /> <munit:parameter propertyName="expectedCode" value="ANK000001" /> </munit:parameters> </munit:parameterization> <munit:parameterization name="002" > <munit:parameters > <munit:parameter propertyName="caseNumber" value="002" /> <munit:parameter propertyName="expectedType" value="TYPE" /> <munit:parameter propertyName="expectedCode" value="ANK000002" /> </munit:parameters> </munit:parameterization>  </munit:parameterizations> </munit:config> Dynamic Payload for Munit Test This is how we can write a test case with a dynamic request payload. This will help to define a number of different input payloads based on the test scenario that can be used as a parameterized input. XML <ee:transform doc:name="payload" doc:id="7c934c10-d874-4207-be27-de6c8b1a1c5a" > <ee:message > <ee:set-payload > <![CDATA[%dw 2.0 output application/json var fileBaseName = "-invalid-payload.json" var caseNumber = Mule::p("caseNumber") var fileName = caseNumber ++ fileBaseName --- readUrl("classpath://test-data/employee/payload/$(fileName)", "application/json")]]> </ee:set-payload> </ee:message> </ee:transform> Test Output We can test multiple field validation with a single munit test by using dynamic payload and parameterized input. Code Reference To find more on the above description, please follow the full code reference here.

By Ankur Bhuyan

Run Ansible Playbook From Azure DevOps Release Pipeline

In this article, you will learn how to run Ansible Playbook from the Azure DevOps tool. By incorporating Ansible Playbooks into the Azure release pipeline, organizations can achieve streamlined and automated workflows, reducing manual intervention and minimizing the risk of errors. This enhances the efficiency of the release process, accelerates time-to-market, and ensures a standardized and reliable deployment of applications and infrastructure on the Azure platform. What Is an Ansible Playbook? An Ansible Playbook is a configuration management and automation tool to define and execute a series of tasks. It is particularly valuable in managing infrastructure as code, ensuring consistency and repeatability in the deployment and configuration of systems. In the context of Azure release pipelines, Ansible Playbooks play a crucial role in automating the deployment and configuration of resources within the Azure environment. They allow for the definition of tasks such as provisioning virtual machines, configuring networking, and installing software components. This tutorial assumes that the Ansible utility is installed and enabled for your Project in Azure DevOps. You can download and install the utility from this link, and get it enabled from your Azure DevOps Administrator. Related: Learn how to schedule pipelines in Azure DevOps. How to Run Ansible Playbook From Azure DevOps Step 1: Create New Release Pipeline Create a new release pipeline with an empty job. Step 2: Add Artifacts in Release Pipeline Job Next, add Azure DevOps in artifacts, as I am using the Azure repository to store our playbook and inventory file. I have already pushed the inventory file and tutorial.yml playbook in my Azure repo branch, ansible-tutorial. Select your project, repo, and branch to add artifacts in your release pipeline. YAML xxxxxxxxxx 1 1 #tutorial.yml 2 - hosts: "{{ host }" 3 tasks: 4 - name: create a test file for ansible 5 shell: touch /tmp/tutorail.yml Step 3: Upload and Configure Secure Key in Stage 1 for Ansible-Playbook Authentication Use the SSH key for authentication on my target machine. To pass the SSH key, I will upload it using the Download Secure file utility available. Download Secure Utility This is used for storing secure files in your release pipeline like SSH key, SSL certs, and CA certs. During execution, files are downloaded in a temp folder and their path can be accessed by calling the reference variable (shown below). These files are deleted as the release job is completed. Enter the reference name as shown below. To access the file, use the variable $(<reference name>.secureFilePath). Ex: $(pemKey.SecureFilePath) Step 4: Change File Permission We will add a shell command-line utility to change the file permission to 400 before using it in the playbook. I have used $(pemKey.secureFilePath) to access the SSH key. Step 5: Add and Configure the Ansible Task Add the Ansible task and enter the playbook path as shown below. For an inventory, the location selects the file and the file path as shown below. Use additional parameters to pass a variable to the Ansible playbook. Use additional parameters to pass variables and other command line parameters to the playbook at run time. To pass the path of the SSH key, I have used ansible_ssh_private_key_file=$(pemKey.secureFilePath). Also, you can use the variable ansible_ssh_common_args='-o StrictHostKeyChecking=no' to disable the host key checking in your Ansible playbook, if it's failing due to a host key verification error. Step 6: Save the Release Pipeline and Create a Release To Run the Playbook We can see our release completed successfully. Summary Ansible playbook ran successfully from Azure DevOps. If you want to use a username and password instead of an SSH key, you can pass the Linux creds using additional parameters using secrets variables so that the creds will be masked, or you can also use a shell command-line utility to set creds in an environment variable and Ansible will read from there.

By Virendar Kumar

A Guide To Visual Regression Testing With PlayWright and How to Get Started

I’m pretty sure that you’ve had a situation where you deployed a major UX change on your web app and missed the most obvious issues, like a misaligned button or distorted images. Unintended changes on your site can cause not only a sharp decline in user satisfaction but also a large fall in sales and customer retention. By identifying and resolving these discrepancies before the update went live, you could have prevented these outcomes. This is where visual regression testing comes in, helping you validate visual elements on your web app. Having a visual regression testing strategy prevents issues that could disrupt the user experience and cause users to bounce. In this article, we’re focusing on visual regression testing with PlayWright. We’ll break down the concept, go through the benefits of visual testing, and see how to implement it into your testing strategy. Finally, I’ll show you how to combine Checkly and PlayWright for effective visual regression testing. What Is Visual Regression Testing With PlayWright? Visual regression testing with PlayWright is the most reliable way to ensure your web app looks and behaves correctly. It does this by comparing two snapshots of an area of your choice and immediately detecting issues related to the layout, content, or design of your web app. When web apps are updated iteratively with new features or optimizations, there is a chance that unintentional visual changes will occur. If these changes go unnoticed, they can negatively affect the user experience, cause customer annoyance, or even lower engagement or sales. The main goal of visual regression testing is to provide predictable and consistent visual behavior over iterations, avoiding regressions and improving the application's overall quality. Visual Regression Testing Key Concepts To understand the concept of visual regression testing, let’s look at these key concepts related to it: Image comparison: To identify changes between older and more recent iterations of web pages, visual regression testing uses image comparison techniques. To draw attention to any visual changes or discrepancies, PlayWright takes screenshots and applies picture-diffing methods to them. Baseline establishment: Setting up a baseline entails establishing a point of reference for the visual design of the application's initial release. This baseline is used as a benchmark for comparison in later testing iterations. Automated workflows: With PlayWright, automation is essential to visual regression testing since it makes it possible to execute tests repeatedly and smoothly across a variety of devices and browsers. Version control system integration: By incorporating visual regression tests into version control systems, development teams can work together more easily, and traceability is guaranteed.Check how the feature compares snapshots here: Benefits of Visual Regression Testing Now, let’s look at the benefits and advantages of visual regression testing: Consistent UI/UX stability: Visual regression testing ensures a consistent and stable user interface and experience across different versions, guaranteeing reliability for end-users. Efficiency and cost-effectiveness: Automating visual regression tests using PlayWright saves time and resources by reducing the need for manual checks, leading to more efficient testing processes. Early identification of issues: Detecting visual defects early in the development cycle allows for swift issue resolution, minimizing the chances of releasing flawed features and enhancing overall software quality. Cross-platform compatibility: With PlayWright, visual regression testing verifies visual elements across multiple browsers and devices, ensuring uniformity and compatibility in diverse environments. Confidence in deployments: Regular visual regression testing instills confidence in software releases, decreasing the likelihood of unexpected visual regressions when rolling out updates or new features. What Does Visual Regression Testing Not Detect? Visual testing, while crucial for assessing the graphical interface and layout of web applications, does not cover certain aspects related to the functionality and underlying code. Some areas that visual testing does not encompass include: Functional testing: This testing verifies that the application operates in accordance with its specifications, ensuring that each feature, component, or interaction functions correctly and performs as intended, encompassing form submissions, user actions, data processing, and other essential operations. Security testing: Assessing security measures within the application is vital. Security testing identifies vulnerabilities, weaknesses, and potential threats, aiming to prevent data breaches, unauthorized access, or any form of security compromise. It involves examining authentication methods, data protection mechanisms, encryption, and defenses against diverse cyber threats. Accessibility testing: Ensuring the application's accessibility to all user groups, including individuals with disabilities, is crucial. Accessibility testing confirms compliance with accessibility standards like WCAG, focusing on features such as compatibility with screen readers, keyboard navigation, contrast adjustments, and other elements to ensure an inclusive user experience for diverse audiences. API and backend testing: API and backend testing involve evaluating the functionality and responses of the application's backend components, such as APIs, databases, and server-side operations. Unlike visual testing, which focuses on the front end, this testing requires distinct methodologies to directly interact with and assess the backend systems, ensuring their proper functioning and accuracy in handling data and operations. Getting Started With Visual Regression Testing Checkly natively supports PlayWright Test Runner for browser checks, so you can now use its visual snapshot testing feature with Checkly. These are two important assertions to get you started: .toHaveScreenshot() will help you visually compare a screenshot of your page to a golden image/reference snapshot .toMatchSnapshot() will compare any string or Buffer value to a golden image/reference snapshot Before getting started, make sure you’ve downloaded the newest version of Checkly: Checkly CLI v4.4.0 or later. Checkly Agent v3.2.0 or later. Step 1 Add expect(page).toHaveScreenshot() to your browser check script. Here’s an example: Step 2 Run your browser check. The first time you run it, you will get an error indicating that no golden image/reference snapshot exists yet. A snapshot doesn't exist at /tmp/19g67loplhq0j/script.spec.js-snapshots/Playwright-homepage-1-chromium-linux.png. Step 3 Generate a golden image snapshot by clicking the “Run script and update golden image” option in the “Run script” button. This step will generate a golden image. You can check your golden image in the “Golden Files” tab in the editor. You can now save your check, and on each check run, the golden image will be compared to the actual screenshot. When your check fails due to a visual difference, you will see the difference between the golden image and the actual screenshot in your check result. To find out how to configure visual regression testing with Checkly, please check our docs. Conclusion Mastering the intricacies of visual regression testing and API testing is paramount for delivering resilient and high-performing software applications. While visual testing ensures the visual integrity of your application, API testing guarantees the reliability of backend functionalities. To enhance your testing strategy and safeguard against regressions, consider combining PlayWright with other monitoring tools like Checkly. More Resources The official PlayWright guide on visual comparison and snapshot testing The .toHaveScreenshot() API reference The .toMatchSnapshot() API reference

By Sara Miteva

CORE

Ten Emerging Software Testing Trends for 2024

Anticipated to shatter records and surpass an extraordinary USD 813 billion in revenues by 2027, the global software market is set to achieve unprecedented growth. This surge is propelled by the pivotal role software products play in enabling businesses to attain a competitive edge in the digital era. As organizations strive for excellence in their digital offerings, the imperative to elevate software quality has propelled technology assessments to new heights. In this blog, we will guide you through the forefront of the industry, unveiling the most prominent trends shaping the landscape of software testing. Certainly, let's explore the future of software testing by delving into the first trend on the list: 1. Scriptless Test Automation: Accelerating SDLC With Simplicity Automation testing has long been a cornerstone in expediting the Software Development Life Cycle (SDLC). In response to the evolving demands of modern software, a prominent trend has emerged — Scriptless Test Automation. This innovative approach addresses the need for enhanced scalability beyond traditional manual testing methods. Key Attributes Accessibility for non-coders: Scriptless automation eliminates the requirement for extensive coding skills. Testers and developers can execute manual run tests without delving into complex code scripting, making it a user-friendly approach. Shortcut to high-speed testing: Offering a shortcut to accelerate testing processes, scriptless automation significantly reduces the time and effort traditionally associated with creating and maintaining test scripts. 2. Robotic Process Automation (RPA): Transforming Testing Dynamics In the realm of software testing trends, Robotic Process Automation (RPA) stands out as a transformative force, integrating artificial intelligence (AI), cognitive computing, and the Internet of Things (IoT). This cutting-edge technology has not only redefined the industry landscape but is anticipated to become a specialized testing domain by 2024. Key Attributes AI, cognitive computing, and IoT integration: RPA leverages a synergy of artificial intelligence, cognitive computing capabilities, and IoT functionalities, creating a comprehensive testing approach. Market growth: Reports indicate substantial growth in the RPA market, with projections soaring from US$ 2,599.3 Million in 2022 to an estimated USD 24,633.4 Million by the year 2033. This growth signifies its increasing significance in various industries. 3. Combined Automation and Manual Testing: Striking a Balance for Holistic Testing In the intricate landscape of software testing, a trend has emerged that emphasizes the synergy of both Automation and Manual Testing. Recognizing the unique strengths and limitations of each approach, this trend advocates for a balanced testing strategy to achieve comprehensive software quality. Key Attributes Security and speed with automation: Automation testing, a burgeoning trend, is celebrated for enhancing security and expediting testing processes. Its strengths lie in repetitive tasks, regression testing, and rapid feedback. Manual testing for varied considerations: Acknowledging that certain aspects like accessibility, user interface intricacies, and architectural nuances cannot be fully addressed by automation alone, manual testing remains a crucial component of the testing process. 4. API and Service Test Automation: Navigating the Era of Microservices The prevalence of microservice architecture in application development has given rise to the prominence of API and Service Test Automation. As client-server platforms proliferate, the need for robust testing of APIs, which operate independently and collaboratively, becomes paramount. Key Attributes Microservices and API integration: Microservice architecture's popularity underscores the significance of APIs in web services automation. APIs enable seamless communication between independently functioning microservices. Interoperability and system integration: APIs not only facilitate communication between microservices but also enable seamless integration with other systems and applications, contributing to the development of a cohesive and interoperable system. 5. More Data, Better Data: Elevating Data Testing for Quality Assurance In the realm of software testing, the meticulous examination of data is integral to ensuring quality assurance. The trend of "More Data, Better Data" underscores the significance of systematic approaches to verify and process data. This includes assessing data accuracy, relevance, believability, repetition, and the sheer number of records. Modern software testing tools are now enhancing the practicality of data collection, paving the way for the emergence of more advanced data testing tools in the pursuit of high-quality products. Key Attributes Methodical data verification: Test teams employ systematic approaches to scrutinize data across multiple parameters, ensuring its accuracy, relevance, believability, and more. Technological advancements: New software testing tools are revolutionizing the collection and analysis of data, making data testing more feasible and comprehensive. 6. Performance Testing: Ensuring Excellence in Software Functionality Performance testing emerges as a pivotal trend, solidifying its place as a crucial component in achieving optimal results for applications. Developers recognize the imperative of creating test scripts not only to safeguard the application but also to ensure its functionality and efficiency. Key Attributes Quality and efficiency: Performance testing is employed to enhance software quality, ensuring that applications operate seamlessly, meet performance benchmarks, and deliver a satisfying user experience. Cross-platform compatibility: Testing scripts are crafted to guarantee that applications function effectively across multiple operating systems and browsers, addressing the diverse technological landscape. 7. Testing Centers for Quality: A Global Shift Towards Excellence Quality Test Centers have emerged as a pivotal trend in global software testing. These centers play a vital role in fostering the creation of high-quality software applications and enhancing the entire application development phase. Going beyond conventional testing practices, these centers house competent QA teams that efficiently reduce the testing period while upholding the consistency, reliability, and effectiveness of the product. Additionally, a focus on robust test automation aligns software development with QA demands, ensuring a comprehensive and efficient Software Testing Life Cycle. Key Attributes Promotion of quality software: Quality Test Centers contribute to the creation of robust software applications, elevating the quality standards of the products in the long run. Competent QA teams: These centers boast skilled QA teams that optimize testing periods without compromising on the fundamental attributes of a product, such as consistency, reliability, and effectiveness. 8. Cyber Security and Risk Compliance: Safeguarding the Digital Realm The digital revolution brings unprecedented benefits but also introduces threats such as cyber threats and various forms of digital attacks. In response to the critical need for security testing, the trend of Cyber Security and Risk Compliance has emerged. This trend is pivotal in ensuring the security of products, networks, and systems against cyber threats and diverse risks. Key Attributes Security testing imperative: Digital dependency underscores the critical nature of security testing. Products, networks, and systems must undergo rigorous security checks to safeguard against cyber threats and ensure user safety. Standardization of security practices: Transaction processing and user safety are now standard requirements, leading to improved coding practices for secure software. 9. IoT (Internet of Things) Testing: Navigating the Proliferation of IoT Devices As the landscape of IoT development expands significantly, so does the need for robust testing. By 2028, the market revenue for IoT devices is projected to reach USD 282.25 billion. In anticipation of this growth, IoT Testing emerges as a core trend in software testing for 2024. This trend is designed to enable testers to comprehensively test and analyze risks associated with IoT instruments, with a focus on security, accessibility, software compatibility, data integrity, performance, and scalability. Key Attributes Addressing diverse risks: IoT Testing aims to address a spectrum of risks associated with IoT devices, ensuring their secure, efficient, and scalable integration into the digital landscape. Key focus areas: Testing in the realm of IoT will concentrate on crucial aspects such as security, ensuring data integrity, assessing performance, and evaluating scalability in the context of IoT devices. 11. QAOps: Bridging Development, Testing, and Operations In the holistic approach of QAOps, developers, testers, and operations teams converge to enhance collaboration and streamline processes. This practice incorporates continuous testing within the broader DevOps framework, improving Continuous Integration/Continuous Delivery (CI/CD) pipelines and fostering a seamless collaboration between testers, QA professionals, and developers. Key Attributes Collaboration across teams: QAOps emphasizes collaboration among developers, testers, and operations teams, breaking down silos and fostering a shared responsibility for software quality. Continuous testing integration: The integration of QA testing into the DevOps approach enhances CI/CD pipelines, ensuring a continuous and efficient testing process throughout the software development life cycle.

By Shormistha Chatterjee

CORE

Unveiling GitHub Copilot's Impact on Test Automation Productivity: A Five-Part Series

Phase 1: Establishing the Foundation In the dynamic realm of test automation, GitHub Copilot stands out as a transformative force, reshaping the approach of developers and Quality Engineers (QE) towards testing. As QA teams navigate the landscape of this AI-driven coding assistant, a comprehensive set of metrics has emerged, shedding light on productivity and efficiency. Join us on a journey through the top key metrics, unveiling their rationale, formulas, and real-time applications tailored specifically for Test Automation Developers. 1. Automation Test Coverage Metrics Test Coverage for Automated Scenarios Rationale: Robust test coverage is crucial for effective test suites, ensuring all relevant scenarios are addressed. Test Coverage = (Number of Automated Scenarios / Total Number of Scenarios) * 100 Usage in real-time scenarios: Provides insights into the effectiveness of test automation in scenario coverage. Cost savings: Higher automation test coverage reduces the need for manual testing, resulting in significant cost savings. 2. Framework Modularity Metrics Modularity Index Rationale: Modularity is key for maintainability and scalability. The Modularity Index assesses independence among different modules in your automation framework. Modularity Index = (Number of Independent Modules / Total Number of Modules) * 100 Usage in real-time scenarios: Evaluate modularity during framework development and maintenance phases for enhanced reusability. Cost savings: A higher modularity index reduces time and effort for maintaining and updating the automation framework. 3. Test Script Efficiency Metrics Script Execution Time Rationale: Script execution time impacts the feedback loop. A shorter execution time ensures quicker issue identification and faster development cycles. Script Execution Time = Total time taken to execute all test scripts Usage in real-time scenarios: Monitor script execution time during continuous integration for optimization. Cost savings: Reduced script execution time contributes to shorter build cycles, saving infrastructure costs. Test Script Success Rate Rationale: The success rate reflects the reliability of your automation suite. Test Script Success Rate = (Number of Successful Test Scripts / Total Number of Test Scripts) * 100 Usage in real-time scenarios: Continuously monitor the success rate to identify and rectify failing scripts promptly. Cost savings: Higher success rates reduce the need for manual intervention, saving both time and resources. 4. Assertion Effectiveness Assertion Success Rate Rationale: Assertions ensure correctness in test results. The assertion success rate measures the percentage of assertions passing successfully. Assertion Success Rate = (Number of Successful Assertions / Total Number of Assertions) * 100 - Number of Successful Script Executions: The count of test script executions that have produced the desired outcomes without encountering failures or errors. - Total Number of Script Executions: The overall count of test script executions, including both successful and unsuccessful executions. Usage in real-time scenarios: Regularly track this metric during test execution to ensure the reliability of your test results. Cost savings: Improved assertion effectiveness reduces false positives, minimizing debugging efforts and saving valuable time. 5. Parallel Execution Metrics Rationale: Parallel execution enhances test suite efficiency. Parallel Execution Utilization = (Time with Parallel Execution / Time without Parallel Execution) * 100 Real-time scenarios: Monitor parallel execution utilization during large test suites to optimize test execution times. Cost savings: Efficient use of parallel execution reduces overall testing time, leading to cost savings in infrastructure and resources. 6. Cross-Browser Testing Metrics Number of Supported Browsers Rationale: Cross-browser testing ensures compatibility across various browsers, a critical factor in user satisfaction. Cross Browser Test Success Rate = (Number of Successful Cross Browser Tests / Total Number of Cross Browser Tests) * 100 Usage in real-time scenarios: Regularly update and track the supported browsers to ensure coverage for the target audience. Cost savings: Identifying and fixing browser-specific issues in the testing phase prevents costly post-production bug fixes. Cross-Browser Test Success Rate Rationale: The success rate of tests across different browsers is vital for delivering a consistent user experience. Cross-Browser Test Success Rate = (Number of Successful Cross-Browser Tests / Total Number of Cross-Browser Tests) * 100 Usage in real-time scenarios: Regularly assess the success rate to catch potential issues with browser compatibility. Cost savings: Early detection of cross-browser issues reduces the time and resources spent on fixing them later in the development process. Conclusion In Phase 1, we've set the stage by exploring essential metrics such as test coverage, framework modularity, and script efficiency. GitHub Copilot's influence is unmistakable. But what's next? As we embark on Phase 2, expect insights into Test Script Efficiency Metrics. How does Copilot enhance script execution time and success rates? Stay tuned for more discoveries in Phase 2! The journey into GitHub Copilot's impact on test automation efficiency continues.

By SHALLABH DIXITT

Testing, Tools, and Frameworks

DZone's Featured Testing, Tools, and Frameworks Resources

Top Testing, Tools, and Frameworks Experts

The Latest Testing, Tools, and Frameworks Topics