DevOps and CI/CD Resources

DZone's Featured DevOps and CI/CD Resources

The Rise of the Platform Engineer: How to Deal With the Increasing Complexity of Software

By Mirco Hering

Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, The Modern DevOps Lifecycle: Shifting CI/CD and Application Architectures. DevOps — ✓DevSecOps — ✓Platform engineering — ? Is platform engineering just another term used for a specialization of DevOps, or is it something different? The truth is probably somewhere in the middle. DevOps and its associated DevXOps flavors have a strong cultural spice that puts the individual teams at the center. Unfortunately, in many places, DevOps has led to new problems like tool proliferation and a lack of harmonization across the enterprise. One could say that in response to the very strict silos and strong centralization of the past, DevOps has pushed the pendulum too far toward federation — and, hence, a suboptimization at the team level — to the detriment of the organization. This has been felt most by the larger, more complex enterprises that have to deal with different technology stacks and differing levels of maturity across the organization. Platform engineering has evolved as a response to this enterprise-wide challenge. Platform engineering is not a replacement for DevOps. Instead, platform engineering complements DevOps to address enterprise-wide challenges and provide a tooling platform that makes it easier for individual teams to do the right thing rather than break things while trying to maintain consistency across the organization. IT delivery has increased in complexity over the last few years, given that more applications are moving at a faster pace. This means organizations cannot rely on individuals to control the complexity; they require systemic answers supported by the proper tooling. This is the problem statement that platform engineering has the ambition to address. With this, platform engineers have become crucial for organizations, as their role holds the keys to enabling security and engineering standards. What Is a Platform Engineer? The role of the platform engineer has three different parts. Figure 1. Role of the platform engineer The most obvious one is the role of a technical architect as they have to build an engineering platform that connects all tools and enables processes. The second aspect is a community enabler, which is similar to developer relations roles at technical tooling companies. The third part is a product manager; the competing interests and demands from the developer community need to be prioritized against the technical needs of the platform (consider things like security hardening and patching of outdated components). Platform Engineer as Technical Architect In organizations with moderate or high complexity within their technology stack, the number of tools required to build, release, and maintain software is at least a dozen, sometimes more. Integrating these tools and enabling the measurement of meaningful metrics is about as tricky as integrating business applications. After all, the challenges are very similar: Different processes need to be aligned, data models need to be transformed to make them usable, and integration points need to be connected to enable the end-to-end process. The systems that run the software side of the business have become similarly challenging. The role of the platform engineer here is to look after the architecture of the tools that run the software side — the goal being to make the tools "disappear" and make the build and release of software appear easy. Platform Engineer as Community Enabler Software engineers tend to think their solutions are better than those from someone else. As such, the adoption of engineering platforms is a challenge to overcome. Telling engineers to use a specific tool has often been met with resistance. The platform engineer must be a community enabler who works with the engineers to promote the platform and convince them of the benefits. Communication goes both ways in this part of the role as the platform engineer also must listen to the problems and challenges of the platform and identify new features that are high in demand. This leads to the third part of the role. Platform Engineer as Product Manager Competing demands on the platform come from the engineers of an organization and other stakeholders like security and, of course, the platform engineers. Prioritizing these demands in a meaningful way is a difficult task as you have to find a balance between all the competing interests, especially as funding for the platform is often a challenge in itself, so speed to value is critical for the ongoing support of the platform. The platform engineer requires good negotiation skills to navigate these challenges. Overview of Platform Engineering Architecture We spoke about the role of the platform engineer, but what is in that platform that the platform engineer is building and maintaining? It is easiest to think about three layers and one target environment: The top layer is the developer experience. These are the tools the developer directly engages with — tools that drive the overall workflow, like an Agile lifecycle management tool, a service management tool, and the developer IDE, fit into this. The bottom layer comprises the infrastructure components that must be combined to build application environments. This can be from the public or private cloud and includes traditional data center technologies. In the middle is where most of the complexity sits — the software engineering platform. Here, all the processes that are required to create and deliver software are being orchestrated: CI/CD, security scanning, environment provisioning, and release management. Figure 2. Platform structure Making the Switch: How to Adopt Platform Engineering Across DevOps Teams So where should you start? One successful adoption pattern focuses on identifying developer journeys to define a minimum viable platform. Which capabilities are required to enable a developer journey to achieve an outcome? Think of a task like provisioning an environment, deploying a new API to production, or running a performance test suite. Each is a valid developer journey with multiple touchpoints that potentially require numerous tools. Once you have created the minimum viable platform for the first set of applications or technologies, adoption follows three dimensions: More applications (once the required capabilities are available), more capabilities, and more maturity, thus increasing the levels of automation and/or performance. Besides worrying about building out the platform with a reasonable approach, three other aspects should be addressed early on: Community engagement Funding Measuring outcomes from the platform Defining a community engagement strategy can be very helpful. This strategy should contain how the information will be shared with the developer community, how feature requests can be made, and how the platform's benefits will be communicated. Defining the forums, the communications, and their respective frequency is also helpful. Funding can quickly become a bottleneck, so a funding strategy should be agreed upon early in the platform engineer adoption. This can be one of several strategies, such as dedicated funding, funding for the services provided, or a service tax on all software development. Each has its own benefits and challenges, a discussion of which is beyond the scope of this article. What is essential is to have a sustainable long-term funding strategy that does not depend on stakeholders' goodwill. Last but not least, the platform engineer needs to be able to show results, which means we need to measure meaningful metrics that showcase why the company is better off with the platform in place. This is often forgotten or an afterthought. Understanding the organization's priorities and aligning the measurement framework to it can help achieve ongoing support. Unfortunately, this usually requires data alignment across multiple tools and is easiest to accomplish when thought about upfront — it becomes increasingly difficult the longer the data models of individual tools remain isolated. Conclusion Platform engineering is still pretty new, yet there is already a lot of content on it, which shows how quickly it has gained interest from organizations. There is even a dedicated conference for it, which began in 2022 and has thousands of participants. It's the early days, but current indications show that platform engineering has quickly found market adoption and a passionate community. And while this is happening, the role of the platform engineer will steadily increase in importance, which is already showing up in salaries too. Hopefully, platform engineering will continue to help organizations reduce complexity for their developers while delivering on the DevOps promise: to provide better solutions faster and more securely. This is an excerpt from DZone's 2024 Trend Report, The Modern DevOps Lifecycle: Shifting CI/CD and Application Architectures.For more: Read the Report More

A Comprehensive DevSecOps Guide: Key Considerations to Effectively Secure Your CI/CD Pipeline

By Louis-Guillaume Morand

Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, The Modern DevOps Lifecycle: Shifting CI/CD and Application Architectures. The era of digital transformation has brought about the need for faster, efficient, and more secure software development processes. Enter DevSecOps: a philosophy that integrates security practices into DevOps processes and aims to embed security into every stage of the development lifecycle — from the writing of code to application deployment in production. The incorporation of DevSecOps can lead to numerous benefits such as early identification of vulnerabilities, cost savings, and faster delivery times. Shift-Left Principle The term "shift left" refers to shifting the focus on security checks and controls toward the beginning, or "left," of the software development lifecycle (SDLC). Traditionally, security checks were performed toward the end, or "right," of the SDLC, often leading to vulnerabilities being detected late in the process, whilst the application is already deployed in production and such vulnerabilities are more expensive and time-consuming to fix. The shift-left principle offers numerous benefits: Early detection of vulnerabilities – By integrating security checks earlier in the SDLC, vulnerabilities can be detected and addressed sooner. This reduces the risk of security breaches and ensures a more secure product. Reduced costs – Addressing security issues late in the development process can be costly. By shifting left, these issues are identified and repaired early, reducing the associated costs and resources required. Improved compliance – With security integrated from the outset, it's easier to ensure compliance with industry regulations and standards. Enhanced product quality – A product built with security in mind from the beginning is likely to be of higher quality with fewer bugs and vulnerabilities. Faster time to market – By reducing the time spent on fixing security issues at later stages, products can be delivered to the market faster. This integration ensures that testing becomes an intrinsic part of the development organization's DNA, fostering a culture where software is meticulously crafted with quality considerations ingrained from the inception of the project. Figure 1. Shifting security controls to the left Key Considerations for DevSecOps Implementation Implementing DevSecOps successfully requires careful consideration of key factors that contribute to a secure and efficient development pipeline. This integration of DevSecOps into the CI/CD pipeline allows for early detection of security issues, reducing the likelihood of vulnerabilities making their way into production while also allowing developers to quickly fix these issues and learn how to avoid reproducing them in the future. Automated Security Testing Tools Because applications come in different forms (e.g., mobile, web, thick client, containerized), you may need to set up different types of controls — and even different types of tooling to secure each component of your application. Let's review the main types of tests you should use. Static Application Security Testing Static application security testing (SAST) tools analyze an application's source code (the code written by your developers) for potential vulnerabilities without executing the program. By scanning the codebase during the development phase, SAST provides developers with insights into security flaws and coding errors. A good SAST tool can detect code smells as well as any bad practices that could lead to vulnerabilities such as SQL or path injection, buffer overflow, XSS, and input validation. Software Composition Analysis Software composition analysis (SCA) is critical for identifying and managing security risks associated with open-source components used in software development, generally coming from additional packages (e.g., NPM packets for JavaScript, NuGet for .NET, Maven, gems). Most developers load a package when they need one but never check if the package has a known vulnerability. An SCA tool will warn you when your application is using a vulnerable package as well as when a fix already exists but you are not using the fixed version of the dependency. Dynamic Application Security Testing Dynamic application security testing (DAST) tools assess applications in their running state, simulating real-world attacks to identify vulnerabilities. By incorporating DAST into the testing process, DevSecOps teams can uncover security weaknesses that may not be apparent during static analysis. A DAST tool will act like a fully automated penetration testing tool that will test for major known vulnerabilities (OWASP) and for a lot of other bad practices such as information leak/exposure. Interactive Application Security Testing Interactive application security testing (IAST) tooling is the combination of a DAST tool and a SAST tool because by allowing access to the source code ("gray boxing"), it will help the DAST perform better but also limit the number of false positives. IAST is super efficient but more challenging to set up because it tends to deeply test each application. Container Scanner Containers offer agility and scalability yet also introduce unique security challenges. For example, if your application is containerized, you must perform additional controls. Mainly, scanners will analyze your Dockerfile to check if the base image contains known vulnerabilities, and they will also look for bad practices such as running as root, using the "latest" tag, or exposing dangerous ports. The following Dockerfile example contains at least three bad practices, and it may contain a vulnerability in the Node.js base image: Shell FROM node:latest WORKDIR /usr/src/app COPY package*.json ./ RUN npm install COPY . . EXPOSE 3000 22 HEALTHCHECK CMD curl --fail http://localhost:3000 || exit 1 CMD ["node","app.js"] Infrastructure-as-Code Scanner Infrastructure as Code (IaC) allows organizations to manage and provision infrastructure through code, bringing the benefits of version control and automation to the infrastructure layer. IaC scanning ensures that infrastructure code undergoes rigorous security controls, such as validating configurations, following best practices, scanning for security misconfigurations, and enforcing security policies throughout the infrastructure deployment process. Secrets Scanner A secret (e.g., API key, password, connection string for a database) should be stored in the source code (hard-coded), or in a configuration file stored within the code repository, because a hacker gaining access to the code could then access production and/or other critical environments. Secrets scanners can detect 150+ types of secrets that developers could leave in the code, and once a secret has been stored in the code (commit), it should be considered "compromised" and revoked immediately. Criteria to select the right third-party product: SAST Number of languages supported, ideally one tool for all code Accuracy of detection Dashboard to customize analysis with sets of rules SCA Number of packages recognized Automated remediation (can create a pull-request with the updated package) DAST Should be able to cover APIs as well as GUI apps Cover more than just OWASP IAST Capable of covering rich applications (e.g., with microservices) Offer remediation/advice to fix detected issues Container Scanner Up-to-date CVE database for the base image Can lint a Dockerfile and check best practices IaC Scanner Find issues in template files Supports the format of your cloud provider (e.g., ARM + Bicep for Azure, CloudFormation for AWS, Deployment Manager for Google Cloud) or Terraform if you are using it Secret scanner The number of credentials types recognized A dashboard that allows security teams to monitor detected secrets and ensure they have been revoked Custom rules to prevent false positives and/or add new formats Establishing Security Gates in CI/CD Pipeline Analysis tools are a good start, but they are useless if they are not part of a global governance. This governance must be built on well-defined security policies and on mandatory controls to ensure that the organization's data and systems are consistently protected against potential threats and vulnerabilities. Defining and Enforcing Security Policies Effective security in a CI/CD pipeline begins with the definition of clear and project-specific security policies. These policies should be tailored to the unique requirements and risks associated with each project. Whether it's compliance standards, data protection regulations, or industry-specific security measures (e.g., PCI DSS, HDS, FedRamp), organizations need to define and enforce policies that align with their security objectives. Once security policies are defined, automation plays a crucial role in their enforcement. Automated tools can scan code, infrastructure configurations, and deployment artifacts to ensure compliance with established security policies. This automation not only accelerates the security validation process but also reduces the likelihood of human error, ensuring consistent and reliable enforcement. Integration of Security Gates In the DevSecOps paradigm, the integration of security gates within the CI/CD pipeline is pivotal to ensuring that security measures are an inherent part of the software development lifecycle. If you set up security scans or controls that users can bypass, those methods become totally useless — you want them to become mandatory. Security gates act as checkpoints throughout the CI/CD pipeline, ensuring that each stage adheres to predefined security standards. By integrating automated security checks at key points, such as code commits, build processes, and deployment stages, organizations can identify and address security issues in a systematic and timely manner. These gated controls can be in different forms: Automated security controls (e.g., SAST, SCA, CredScan) Manual approval (e.g., code review) Manual testing (e.g., pen testing by specialized teams) Performance testing Quality (e.g., a query that monitors the number of defects opened in your quality tracking tool) Figure 2. Standard DevSecOps pipeline with gated security controls Continuous Monitoring and Feedback In the fast-paced world of software development, the importance of real-time monitoring for security and quick fixing cannot be overstated because even with gated controls, vulnerabilities can be found after an application has been deployed in production. Real-Time Monitoring for Security Real-time monitoring allows teams to proactively detect and respond to security threats as they emerge. By leveraging automated tools and advanced analytics, organizations can continuously monitor their applications, infrastructure, and networks for potential vulnerabilities or suspicious activities. This proactive approach not only enhances security but also minimizes the risk of security breaches and data compromises. It gives the ability to gain comprehensive visibility across the entire technology stack. DevSecOps teams can track and analyze security metrics at every layer, from application code to production environments. This visibility enables quick identification of security gaps and facilitates the implementation of targeted remediation measures, ensuring a robust defense against evolving cyber threats. Addressing Security Findings and Adapting Processes Identifying security findings is only the first step; effective DevSecOps requires a proactive approach to address and remediate these issues promptly. When security findings are identified, cross-functional teams work together to assess the impact, prioritize remediation tasks, and implement corrective measures. This collaborative effort ensures that security is everyone's responsibility and not just confined to a specific silo within the organization. Adaptability is a core tenet of DevSecOps. Organizations must foster a culture of continuous learning, where security teams regularly update their knowledge, processes, and tools based on evolving threats and industry best practices. This adaptive mindset ensures that security measures remain effective in the face of new challenges, and that the DevSecOps pipeline is continually refined for optimal security outcomes. Conclusion As software development processes continue to evolve, the need for robust security measures within the CI/CD pipeline becomes more critical. Embracing a DevSecOps approach can help organizations create a secure, efficient, and reliable CI/CD pipeline. By prioritizing security from the get-go and implementing security gates, organizations can save resources, reduce risk, and ultimately, deliver better, safer products to the market. Go and make security the foundation of your products! This is an excerpt from DZone's 2024 Trend Report, The Modern DevOps Lifecycle: Shifting CI/CD and Application Architectures.For more: Read the Report More

Integrating Software Supply Chains and DevOps: Tips for Effectively Reconciling Supply Chain Management and DevOps

By Justin Albano

CORE

Eliminate Human-Based Actions With Automated Deployments: Improving Commit-to-Deploy Ratios Along the Way

By John Vester

CORE

Achieving High Availability in CI/CD With Observability

By Lipsa Das

CORE

Software Engineering Trends in the Industry

This article identifies some basic trends in the software industry. Specifically, we will explore how some well-known organizations implement and benefit from early and continuous testing, faster software delivery, reduced costs, and increased collaboration. While it is clear that activities like breaking down silos, shift-left testing, automation, and continuous delivery are interrelated, it is beneficial to take a look at how companies strive to achieve such goals in practice. Companies try to break down the traditional silos that separate development, operations, and testing teams. This eliminates barriers and builds collaboration, where all teams share responsibility for quality throughout the software development lifecycle. This collaborative approach leads to improved problem-solving, faster issue resolution, and ultimately, higher-quality software. The concept of "shifting left" emphasizes integrating testing activities earlier into the development process. This means conducting tests as code is written (unit tests) and throughout development stages (integration tests), instead of waiting until the end. By detecting and fixing defects earlier, the overall development cycle becomes more efficient as issues are addressed before they become complex and expensive to fix later. This proactive approach ultimately leads to higher-quality software and faster releases. Embracing automation is another core trend. By utilizing automated testing tools and techniques, such as unit testing frameworks and continuous integration pipelines, organizations can significantly accelerate the testing process. This frees up valuable human resources, allowing testers to focus on more complex tasks like exploratory testing, test strategy development, and collaborating with other teams. This increases efficiency, it allows faster feedback loops and earlier identification of defects, ultimately leading to higher-quality software and faster releases. Continuous delivery, ensuring high-quality software is delivered frequently and reliably is another key trend. This is achieved through several key practices: automation of repetitive tasks, integration and testing throughout development, and streamlined deployment pipelines. By catching and addressing issues early, fewer defects reach production, enabling faster and more reliable releases of high-quality software that meets user expectations. This continuous cycle of delivery and improvement ultimately leads to increased innovation and a competitive edge. Early and Continuous Testing Early and continuous testing may lead to better defect detection and faster resolution, resulting in higher-quality software. Let's take a look at a few specific cases: 1. Netflix Challenge Netflix's challenge is releasing new features regularly while maintaining a high level of quality across various devices and platforms. Solution Netflix adopted a DevOps approach with extensive automation testing. They utilize unit tests that run on every code commit, catching bugs early. Additionally, they have automated testing frameworks for various functionalities like UI, API, and performance. Impact This approach allows them to identify and fix issues quickly, preventing them from reaching production and impacting user experience. 2. Amazon Challenge Amazon's challenge is ensuring the reliability and scalability of their massive e-commerce platform to handle unpredictable traffic spikes. Solution Amazon employs a "chaos engineering" practice. They intentionally introduce controlled disruptions into their systems through automated tools, simulating real-world scenarios like server failures or network outages. This proactive testing helps them uncover potential vulnerabilities and weaknesses before they cause customer disruptions. Impact By identifying and addressing potential issues proactively, Amazon can ensure their platform remains highly available and reliable, providing a seamless experience for millions of users. 3. Spotify Challenge Spotify's challenge is maintaining a seamless music streaming experience across various devices and network conditions. Solution Spotify heavily utilizes continuous integration and continuous delivery (CI/CD) pipelines, integrating automated tests at every stage of the development process. This includes unit tests, integration tests, and performance tests. Impact Early detection and resolution of issues through automation allow them to maintain a high level of quality and deliver frequent app updates with new features and bug fixes. This results in a more stable and enjoyable user experience for music lovers globally. These examples highlight how various organizations across different industries leverage early and continuous testing to: Catch defects early: Automated tests identify issues early in the development cycle, preventing them from cascading into later stages and becoming more complex and expensive to fix. Resolve issues faster: Early detection allows for quicker bug fixes, minimizing potential disruptions and ensuring a smoother development process. Deliver high-quality software: By addressing issues early and continuously, organizations can deliver software that meets user expectations and performs reliably. By embracing early and continuous testing, companies can achieve a faster time-to-market, reduced development costs, and ultimately, a more satisfied customer base. Faster Software Delivery Emphasizing automation and continuous integration empowers organizations to achieve faster software delivery. Here are some examples showcasing how: 1. Netflix Challenge Netflix's challenge is maintaining rapid release cycles for new features and bug fixes while ensuring quality. Solution Netflix utilizes a highly automated testing suite encompassing unit tests, API tests, and UI tests. These tests run automatically on every code commit, providing immediate feedback on potential issues. Additionally, they employ a continuous integration and delivery (CI/CD) pipeline that automatically builds, tests, and deploys code to production environments. Impact Automation reduces the need for manual testing, significantly reducing testing time and allowing for faster feedback loops. The CI/CD pipeline further streamlines deployment, enabling frequent releases without compromising quality. This allows Netflix to deliver new features and bug fixes to users quickly, keeping them engaged and satisfied. 2. Amazon Challenge Amazon's challenge is scaling deployments and delivering new features to their massive user base quickly and efficiently. Solution Amazon heavily invests in infrastructure as code (IaC) tools. These tools allow them to automate infrastructure provisioning and configuration, ensuring consistency and repeatability across different environments. Additionally, they leverage a robust CI/CD pipeline that integrates automated testing with infrastructure provisioning and deployment. Impact IaC reduces manual configuration errors and streamlines infrastructure setup, saving significant time and resources. The integrated CI/CD pipeline allows for automated deployments, reducing the time required to move code from development to production. This enables Amazon to scale efficiently and deliver new features and services to their users at an accelerated pace. 3. Spotify Challenge Spotify's challenge is keeping up with user demand and delivering new features and updates frequently. Solution Spotify utilizes a containerized microservices architecture, breaking its application down into smaller, independent components. This allows for independent development, testing, and deployment of individual services. Additionally, they have invested heavily in automated testing frameworks and utilize a continuous integration and delivery pipeline. Impact The microservices architecture enables individual teams to work on and deploy features independently, leading to faster development cycles. Automated testing provides rapid feedback, allowing for quick identification and resolution of issues. The CI/CD pipeline further streamlines deployment, allowing for frequent releases of new features and updates to the Spotify platform and keeping users engaged with fresh content and functionalities. These examples demonstrate how companies across various sectors leverage automation and continuous integration to achieve: Reduced testing time: Automated testing reduces the need for manual efforts, significantly reducing the time it takes to test and identify issues. Faster feedback loops: Automated tests provide immediate feedback on code changes, allowing developers to address issues quickly and iterate faster. Streamlined deployment: Continuous integration and delivery pipelines automate deployments, minimizing manual intervention and reducing the time it takes to move code to production. By leveraging automation and continuous integration, organizations can enjoy faster time-to-market, increased responsiveness to user needs, and a competitive edge in their respective industries. Reduced Costs Automating repetitive tasks and shifting left can reduce the overall cost of testing. There are three main areas to highlight here. 1. Reduced Manual Effort Imagine a company manually testing a new e-commerce website across different browsers and devices. This would require a team of testers and significant time, leading to high labor costs. By automating these tests, the company can significantly reduce the need for manual testing, freeing up resources for more complex tasks and strategic testing initiatives. 2. Early Defect Detection and Resolution A software company traditionally performed testing only towards the end of the development cycle. This meant that bugs discovered late in the process were more expensive to fix due to a number of reasons. By shifting left and implementing automated unit tests early on, the company can identify and fix bugs early in the development cycle, minimizing the cost of rework and reducing the chance of them cascading into later stages. 3. Improved Test Execution Speed A software development team manually ran regression tests after every code change, causing lengthy delays and hindering development progress. By automating these tests, the team can run them multiple times a day, providing faster feedback and enabling developers to iterate more quickly. This reduces overall development time and associated costs. Examples Capgemini: Implemented automation for 70% of their testing efforts, resulting in a 50% reduction in testing time and a 20% decrease in overall project costs. Infosys: Embraced automation testing, leading to a 40% reduction in manual effort and a 30% decrease in testing costs. Barclays Bank: Shifted left by introducing unit and integration testing, achieving a 25% reduction in defect escape rate and a 15% decline in overall testing costs. These examples showcase how companies across different sectors leverage automation and shifting left to achieve the following: Reduced labor costs: Automating repetitive testing tasks reduces the need for manual testers, leading to significant cost savings. Lower rework costs: Early defect detection and resolution minimize the need for rework later in the development cycle, saving time and money. Increased development efficiency: Faster test execution speeds through automation allow developers to iterate more quickly and reduce overall development time, leading to cost savings. By embracing automation and shifting left, organizations can enjoy improved resource utilization, reduced project overruns, and a better return on investment (ROI) for their software development efforts. Increased Collaboration Increased collaboration between development (Dev), operations (Ops), and testing teams. This is achieved by creating a shared responsibility for quality throughout the software development lifecycle. Here's how it works: Traditional Silos vs. Collaborative Approach Traditional Silos In a siloed environment, each team operates independently. Developers write code, testers find bugs, and operations manage the production environment. This often leads to finger-pointing, delays, and a disconnect between teams. Collaborative Approach DevOps, QAOps, and agile practices, among others, break down these silos and promote shared ownership for quality. Developers write unit tests, operations implement automated infrastructure testing, and testers focus on higher-level testing and test strategy. This nurtures collaboration, communication, and a shared sense of accountability. Examples Netflix: Utilizes a cross-functional team structure with members from development, operations, and testing working together. This allows them to share knowledge, identify and resolve issues collaboratively, and ensure a smooth delivery process. Amazon: Employs a "blameless post-mortem" culture where teams analyze incidents collaboratively without assigning blame. This builds openness, encourages shared learning, and ultimately improves system reliability. Spotify: Implements a "one team" approach where developers, operations engineers, and testers work together throughout the development cycle. This facilitates open communication, allows for shared decision-making, and promotes a sense of collective ownership for the product's success. Benefits of Increased Collaboration Improved problem-solving: By working together, teams can leverage diverse perspectives and expertise to identify and resolve issues more effectively. Faster issue resolution: Open communication allows for quicker sharing of information and faster identification of the root cause of problems. Enhanced quality: Increased collaboration creates a culture of ownership and accountability, leading to higher-quality software. Improved team morale: Collaborative work environments are often more enjoyable and motivating for team members, leading to increased productivity and job satisfaction. Strategies for Fostering Collaboration Cross-functional teams: Encourage collaboration by forming teams with members from different disciplines. Shared goals and metrics: Align teams around shared goals and success metrics that promote collective responsibility for quality. Open communication: Create open communication channels and encourage information sharing across teams. Knowledge sharing: Facilitate knowledge sharing across teams through workshops, training sessions, and collaborative problem-solving activities. By adopting DevOps, QAOps, and agile principles, organizations can break down silos, embrace shared responsibility, and cultivate a culture of collaboration. This leads to a more efficient, innovative, and, ultimately, successful software development process. Wrapping Up A number of organizations embark on a transformative journey towards faster, more reliable, and higher-quality software delivery. Through breaking down silos and forging shared responsibility, teams can leverage automation and shift left testing to enhance continuous delivery. This collaborative and efficient approach empowers organizations to deliver high-quality software more frequently, reduce costs, and ultimately gain a competitive edge in the ever-evolving technology landscape.

By Stelios Manioudakis

CORE

How To Implement Code Reviews Into Your DevOps Practice

DevOps encompasses a set of practices and principles that blend development and operations to deliver high-quality software products efficiently and effectively by fostering a culture of open communication between software developers and IT professionals. Code reviews play a critical role in achieving success in a DevOps approach mainly because they enhance the quality of code, promote collaboration among team members, and encourage the sharing of knowledge within the team. However, integrating code reviews into your DevOps practices requires careful planning and consideration. This article presents a discussion on the strategies you should adopt for implementing code reviews successfully into your DevOps practice. What Is a Code Review? Code review is defined as a process used to evaluate the source code in an application with the purpose of identifying any bugs or flaws, within it. Typically, code reviews are conducted by developers in the team other than the person who wrote the code. To ensure the success of your code review process, you should define clear goals and standards, foster communication and collaboration, use a code review checklist, review small chunks of code at a time, embrace a positive code review culture, and embrace automation and include automated tools in your code review workflow. The next section talks about each of these in detail. Implementing Code Review Into a DevOps Practice The key principles of DevOps include collaboration, automation, CI/CD, Infrastructure as Code (IaC), adherence to Agile and Lean principles, and continuous monitoring. There are several strategies you can adopt to implement code review into your DevOps practice successfully: Define Clear Goals and Code Review Guidelines Before implementing code reviews, it's crucial to establish objectives and establish guidelines to ensure that the code review process is both efficient and effective. This helps maintain quality as far as coding standards are concerned and sets a benchmark for the reviewer's expectations. Identifying bugs, enforcing practices, maintaining and enforcing coding standards, and facilitating knowledge sharing among team members should be among these goals. Develop code review guidelines that encompass criteria for reviewing code including aspects like code style, performance optimization, security measures, readability enhancements, and maintainability considerations. Leverage Automated Code Review Tools Leverage automated code review tools that help in automated checks for code quality. To ensure proper code reviews, it's essential to choose the tools that align with your DevOps principles. There are options including basic pull request functionalities, in version control systems such as GitLab, GitHub, and Bitbucket. You can also make use of platforms like Crucible, Gerrit, and Phabricator which are specifically designed to help with conducting code reviews. When making your selection, consider factors like user-friendliness, integration capabilities with development tools support, code comments, discussion boards, and the ability to track the progress of the code review process. Related: Gitlab vs Jenkins, CI/CD tools compared. Define a Code Review Workflow Establish a clear workflow for your code reviews to streamline the process and avoid confusion. It would help if you defined when code reviews should occur, such as before merging changes, during feature development, or before deploying the software to the production environment. Specify the duration allowed for code review, outlining deadlines for reviewers to provide feedback. Ensure that the feedback loop is closed, that developers who wrote the code address the review comments, and that reviewers validate the changes made. Review Small and Digestible Units of Code A typical code review cycle should involve only a little code. Instead, it should split the code into smaller, manageable chunks for review. This would assist reviewers in directing their attention towards features or elements allowing them to offer constructive suggestions. It is also less likely to overlook critical issues when reviewing smaller chunks of code, resulting in a more thorough and detailed review. Establish Clear Roles and Responsibilities Typically, a code review team comprises the developers, reviewers, the lead reviewer or moderator, and the project manager or the team lead. A developer initiates the code review process by submitting a piece of code for review. A team of code reviewers reviews a piece of code. Upon successful review, the code reviewers may request improvements or clarifications in the code. The lead reviewer or moderator is responsible for ensuring that the code review process is thorough and efficient. The project manager or the team lead ensures that the code reviews are complete within the decided time frame and ensuring that the code is aligned with the broader aspects of the project goals. Embrace Positive Feedback Constructive criticism is an element, for the success of a code review process. Improving the code's quality would be easier if you encouraged constructive feedback. Developers responsible, for writing the code should actively seek feedback while reviewers should offer suggestions and ideas. It would be really appreciated if you could acknowledge the hard work, information exchange, and improvements that result from fruitful code reviews. Conduct Regular Training An effective code review process should incorporate a training program to facilitate learning opportunities for the team members. Conducting regular training sessions and setting a clear goal for code review are essential elements of the success of a code review process. Regular trainings play a role, in enhancing the knowledge and capabilities of the team members enabling them to boost their skills. By investing in training the team members can unlock their potential leading to overall success, for the entire team. Capture Metrics To assess the efficiency of your code review procedure and pinpoint areas that require enhancement it is crucial to monitor metrics. You should set a few tangible goals before starting your code review process and then capture metrics (CPU consumption, memory consumption, I/O bottlenecks, code coverage, etc.) accordingly. Your code review process will be more successful if you use the right tools to capture the desired metrics and measure their success. Conclusion Although the key intent of a code review process is identifying bugs or areas of improvement in the code, there is a lot more you can add to your kitty from a successful code review. An effective code review process ensures consistency in design and implementation, optimizes code for better performance and scalability, helps teams collaborate to share knowledge, and improves the overall code quality. That said, for the success of a code review process, it is imperative that the code reviews are accepted on a positive note and the code review comments help the team learn to enhance their knowledge and skills.

By Joydip Kanjilal

CORE

Implementing CI/CD Pipelines With Jenkins and Docker

Continuous integration and continuous delivery (CI/CD) have become critical practices for software teams looking to accelerate development cycles and improve product quality. By automatically building, testing, and deploying application updates, CI/CD pipelines enable reliable and efficient software delivery. This article will discuss best practices for implementing robust CI/CD workflows using popular open-source tools like Jenkins and Docker. Overview of CI/CD Concepts Continuous integration (CI) refers to the practice of frequently merging developer code changes into a shared repository, triggering automated builds and tests to detect integration issues early. Common CI principles include committing code in small increments, continuously testing each change, and rapidly resolving identified problems to avoid the accumulation of technical debt. Continuous delivery (CD) extends upon CI by automating the release process all the way to production deployment using repeatable workflows. Each code change that passes the automated testing gates is considered releasable. Automated deployment allows development teams to deliver features faster and more reliably. Benefits of Adopting CI/CD Practices Implementing CI/CD pipelines provides multiple software development and delivery advantages, including: Accelerated time to market: Automated workflows enable faster build, test, and release cycles Reduced risk: Continuous testing and version control identify defects early on Reliability: Repeatability ensures software updates are consistently delivered without manual errors Developer productivity: Automation frees up developers to focus on coding rather than builds Reputation: Users and customers benefit from faster features and minimal disruption Critical Components of a CI/CD Pipeline A typical CI/CD pipeline comprises several key components connected together: Version control system: Hosts application code in repositories. Developers can collaboratively edit and track changes over time. Popular systems like Git facilitate branching and merging. Build server: Automates compiling source code into executable applications by running build scripts. Popular open-source build servers include Jenkins and Bamboo. Testing framework: Automatically runs unit, integration, and system tests to validate application integrity before release. JUnit and Selenium are commonly used. Binary repository: Stores build artifacts and dependencies in a centralized, easily accessible package. Artifactory and Nexus are common artifact repository examples. Deployment automation: Scripts and configures deployment of built and tested code changes to progressive server environments, all the way up to production. Kubernetes facilitates container deployments. Jenkins Overview Jenkins is one of the most widely adopted open-source automation servers used to set up, operate, and manage CI/CD pipelines. It is used for automating software development processes through a pipeline-oriented architecture. Key Jenkins capabilities include: Easy installation: Available both on-premises and in cloud platforms. Easily scalable. Declarative pipelines: Pipeline workflows can be defined through code using a Jenkinsfile, facilitating version control. Extensive ecosystem: A broad plugin ecosystem allows the integration of the most common developer tools into pipelines. Distributed builds: Supports distributed CI/CD by executing parallel tests and build routines across multiple machines. Simple administration: Easy for admins to manage users, access controls and Jenkins configuration. Docker Overview Docker has emerged as the de facto standard in the development and deployment of containerization technologies. Docker containers bundle application source code together with libraries, dependencies, and a lightweight runtime into an isolated package. Containers provide a predictable way to deploy applications across environments. Benefits include: Lightweight: Containers leverage the host OS instead of needing a guest OS, reducing overhead. Portability: Can run uniformly on any platform due to shared runtime environments. Scalability: Easily spawn multiple instances of containers due to low resource requirements. Isolation: Changes made inside containers do not impact the host machine or other containers. Implementing CI/CD Pipelines Using Jenkins and Docker Leveraging both Jenkins and Docker, robust CI/CD pipelines can be designed that enable continuous code integration and reliable application deployments. Here is one recommended implementation pattern: Code commits: Developers commit code changes frequently to Git repositories. Webhooks trigger Jenkins jobs upon code pushes. Jenkins CI jobs: Jenkins pulls source code and runs CI workflows - clean > build > unit tests > static analysis > create Docker image with dependencies. Docker registry: A validated Docker image is pushed and versioned in a private Docker registry. Deploy Jenkins jobs: Deployment jobs first pull images from the registry and then deploy them onwards to higher environments. Infrastructure: Docker environments for progressive test, stage, and prod application deployment need to be set up. Kubernetes is great for container orchestration. Rollback strategies: Rollback workflows are automated through Jenkins to revert to the last working version in case of production issues. This pipeline allows developers to have a fast inner DevOps feedback loop through Jenkins while Docker containers handle application encapsulation and deployment portability. Infrastructure-as-code practices help manage environment sprawl. Best Practices for Effective Jenkins and Docker CI/CD Based on industry-wide learnings, here are some best practices to follow: Standardize pipelines through templatized Jenkinsfiles checked into source control. Leverage Docker multi-stage builds to keep images lean. Abstract environment differences using Docker runtime configurations over custom image builds. Scale Jenkins dynamically using the Kubernetes plugin for on-demand build agents. Implement Git hooks for commit syntax linting and automated tests before pushing code. Integrate security scans in the pipeline and analyze images for vulnerabilities. Enable traceability by integrating build numbers into application UIs and logs. Simulate production load, traffic, and access environment during later testing stages. Only build container images once through registries. Avoid image sprawl. Conclusion Implementing a high-performing CI/CD pipeline requires integrating disparate systems like code repositories, build servers and container technologies while ensuring automated test coverage through all phases. Jenkins and Docker provide open-source solutions to deliver robust pipelines that augment developer productivity, release reliability and operations efficiency. Standardizing pipelines, branching strategies, and environments provides consistency across the SDLC. By following industry best practices around CI/CD processes, test automation, and architectural decentralization, teams can accelerate innovation cycles dramatically.

By Igboanugo David Ugochukwu

Enhancing DevOps With AI: A Strategy for Optimized Efficiency

In the ever-evolving landscape of software development, the integration of Artificial Intelligence (AI) into DevOps practices emerges as a transformative strategy, promising to redefine the efficiency and effectiveness of development and operational tasks. This article explores the synergy between AI and DevOps, outlining its potential benefits, challenges, and practical applications through code examples. We aim to provide a comprehensive overview catering to professionals seeking to leverage AI to enhance their DevOps processes. The Convergence of AI and DevOps DevOps, a compound of development (Dev) and operations (Ops) emphasizes the continuous integration and delivery of software, fostering a culture of collaboration between developers and IT professionals. The incorporation of AI into DevOps, or AI-driven DevOps, introduces intelligent automation, predictive analytics, and enhanced decision-making into this collaborative framework, aiming to optimize workflow efficiency and reduce human error. Benefits of AI in DevOps Automated problem solving: AI algorithms can automate routine tasks, from code generation to testing, freeing human resources for more complex problem-solving tasks. Predictive analytics: AI-driven analytics can predict failures and identify bottlenecks in the development cycle, enabling preemptive action to mitigate risks. Enhanced efficiency: By automating workflows and optimizing processes, AI reduces the time to market for new software releases. Practical Application: AI-Powered Continuous Integration Tool To illustrate the practical application of AI in DevOps, consider an AI-powered Continuous Integration (CI) tool. This tool utilizes Machine Learning (ML) algorithms to automate the testing and integration of code changes, improving the efficiency and reliability of software development processes. Code Example: AI-Powered CI Tool Python from sklearn.ensemble import RandomForestClassifier from ci_tools import fetch_changes, run_tests, integrate_changes # Load dataset of past code changes and their impact (successful or failed builds) code_change_data, build_outcomes = load_dataset('code_changes.csv') # Train a Random Forest classifier to predict the outcome of code changes classifier = RandomForestClassifier(n_estimators=100) classifier.fit(code_change_data, build_outcomes) # Fetch the latest code changes new_changes = fetch_changes() # Predict the outcome of the new changes predictions = classifier.predict(new_changes) # Automatically integrate changes predicted to be successful for change, prediction in zip(new_changes, predictions): if prediction == 'success': integrate_changes(change) else: print(f'Change {change.id} flagged for review due to predicted failure.') In this example, a RandomForestClassifier from the scikit-learn library predicts the success of new code changes based on historical data. Successful changes integrate automatically, while those predicted to fail are flagged for review. This process exemplifies how AI can automate and optimize the CI pipeline, reducing manual oversight and accelerating the development cycle. Challenges and Considerations Adopting artificial intelligence (AI) in DevOps has benefits but also brings difficulties. Data quality: The effectiveness of AI models depends on the quality and relevance of the training data. Model complexity: Developing and maintaining sophisticated AI models requires expertise in data science and ML. Ethical and privacy concerns: The use of AI must adhere to ethical guidelines and privacy regulations, particularly when processing sensitive data. Conclusion The fusion of AI with DevOps represents a frontier of opportunity for software development teams. By automating routine tasks, predicting system failures, and optimizing workflows, AI-driven DevOps can significantly enhance the efficiency and reliability of software delivery processes. However, realizing this potential requires careful consideration of data quality, model complexity, and ethical concerns. As the field matures, the integration of AI into DevOps will undoubtedly become a standard practice, reshaping the landscape of software development for the better.

By Bhargavi Gorantla

Leveraging Feature Flags With IBM Cloud App Configuration in React Applications

In modern application development, delivering personalized and controlled user experiences is paramount. This necessitates the ability to toggle features dynamically, enabling developers to adapt their applications in response to changing user needs and preferences. Feature flags, also known as feature toggles, have emerged as a critical tool in achieving this flexibility. These flags empower developers to activate or deactivate specific functionalities based on various criteria such as user access, geographic location, or user behavior. React, a popular JavaScript framework known for its component-based architecture, is widely adopted in building user interfaces. Given its modular nature, React applications are particularly well-suited for integrating feature flags seamlessly. In this guide, we'll explore how to integrate feature flags into your React applications using IBM App Configuration, a robust platform designed to manage application features and configurations. By leveraging feature flags and IBM App Configuration, developers can unlock enhanced flexibility and control in their development process, ultimately delivering tailored user experiences with ease. IBM App Configuration can be integrated with any framework be it React, Angular, Java, Go, etc. React is a popular JavaScript framework that uses a component-based architecture, allowing developers to build reusable and modular UI components. This makes it easier to manage complex user interfaces by breaking them down into smaller, self-contained units. Adding feature flags to React components will make it easier for us to handle the components. Integrating With IBM App Configuration IBM App Configuration provides a comprehensive platform for managing feature flags, environments, collections, segments, and more. Before delving into the tutorial, it's important to understand why integrating your React application with IBM App Configuration is necessary and what benefits it offers. By integrating with IBM App Configuration, developers gain the ability to dynamically toggle features on and off within their applications. This capability is crucial for modern application development, as it allows developers to deliver controlled and personalized user experiences. With feature flags, developers can activate or deactivate specific functionalities based on factors such as user access, geographic location, or user preferences. This not only enhances user experiences but also provides developers with greater flexibility and control over feature deployments. Additionally, IBM App Configuration offers segments for targeted rollouts, enabling developers to gradually release features to specific groups of users. Overall, integrating with IBM App Configuration empowers developers to adapt their applications' behavior in real time, improving agility, and enhancing user satisfaction. To begin integrating your React application with App Configuration, follow these steps: 1. Create an Instance Start by creating an instance of IBM App Configuration on cloud.ibm.com. Within the instance, create an environment, such as Dev, to manage your configurations. Now create a collection. Creating collections comes in handy when there are multiple feature flags created for various projects. Each project can have a collection in the same App Configuration instance and you can tag these feature flags to the collection to which they belong. 2. Generate Credentials Access the service credentials section and generate new credentials. These credentials will be required to authenticate your React application with App Configuration. 3. Install SDK In your React application, install the IBM App Configuration React SDK using npm: CSS npm i ibm-appconfiguration-react-client-sdk 4. Configure Provider In your index.js or App.js, wrap your application component with AppConfigProvider to enable AppConfig within your React app. The Provider must be wrapped at the main level of the application, to ensure the entire application has access. The AppConfigProvider requires various parameters as shown in the screenshot below. All of these values can be found in the credentials created. 5. Access Feature Flags Now, within your App Configuration instance, create feature flags to control specific functionalities. Copy the feature flag ID for further integration into your code. Integrating Feature Flags Into React Components Once you've set up the AppConfig in your React application, you can seamlessly integrate feature flags into your components. Enable Components Dynamically Use the feature flag ID copied from the App Configuration instance to toggle specific components based on the flag's status. This allows you to enable or disable features dynamically without redeploying your application. Utilizing Segments for Targeted Rollouts IBM App Configuration offers segments to target specific groups of users, enabling personalized experiences and controlled rollouts. Here's how to leverage segments effectively: Define Segments Create segments based on user properties, behaviors, or other criteria to target specific user groups. Rollout Percentage Adjust the rollout percentage to control the percentage of users who receive the feature within a targeted segment. This enables gradual rollouts or A/B testing scenarios. Example If the rollout percentage is set to 100% and a particular segment is targeted, then the feature is rolled out to all the users in that particular segment. If the rollout percentage is set between 1% to 99% and the rollout percentage is 60%, for example, and a particular segment is targeted, then the feature is rolled out to randomly 60% of the users in that particular segment. If the rollout percentage is set to 0% and a particular segment is targeted, then the feature is rolled out to none of the users in that particular segment. Conclusion Integrating feature flags with IBM App Configuration empowers React developers to implement dynamic feature toggling and targeted rollouts seamlessly. By leveraging feature flags and segments, developers can deliver personalized user experiences while maintaining control over feature deployments. Start integrating feature flags into your React applications today to unlock enhanced flexibility and control in your development process.

By Pradeep Gopalgowda

Deploying MuleSoft Using Azure DevOps

Continuous Integration and Continuous Delivery (CI/CD) have become common practices in developing high-quality applications because they promote scaling and shorten the time between coding and deployment. Mulesoft is a popular integration platform that streamlines data integration across disparate systems. The predefined APIs have support for all types of applications where data can be unlocked between legacy systems, applications, and cloud devices. This highly networked MuleSoft Anypoint platform enables smarter and faster decision-making abilities in an organization. Azure DevOps, on the other hand, provides version control, reporting, requirements management, project management, automated builds, lab management, testing, and release management capabilities. In short, it covers the entire software development lifecycle and enables organizations to automate the software delivery process. The variety of services leverages the collaboration and development of any project. By combining Azure DevOps with Mulesoft, organizations can leverage the CI/CD pipeline by streamlining the process of development and delivery. This not only increases the efficiency of the process but also considerably reduces risk due to error. Azure CI/CD How to Deploy MuleSoft With Azure DevOps Here we are going to see how to implement CI/CD pipeline for MuleSoft application using Azure DevOps. Step 1: Mule-Maven Plugin Configuration Make sure you have added the Mule-Maven plugin in your pom.xml. XML x 39 1 <plugin> 2 <groupId>org.mule.tools.maven</groupId> 3 <artifactId>mule-maven-plugin</artifactId> 4 <version>3.3.1</version> 5 <extensions>true</extensions> 6 <configuration> 7 <cloudHubDeployment> 8 <uri>https://anypoint.mulesoft.com</uri> 9 <muleVersion>${app.runtime}</muleVersion> 10 <username>${anypoint.username}</username> 11 <password>${anypoint.password}</password> 12 <businessGroup>${businessGroup}</businessGroup> 13 <workers>${workers}</workers> 14 <workerType>${workerType}</workerType> 15 <region>us-west-1</region> 16 <environment>${environment}</environment> 17 <applicationName>${applicationName}</applicationName> 18 <properties> 19 <mule.env>${mule.env}</mule.env> 20 <encrypt.key>${encrypt.key}</encrypt.key> <anypoint.platform.client_id>${anypoint.platform.client_id} </anypoint.platform.client_id> <anypoint.platform.client_secret>${anypoint.platform.client_secret}</anypoint.platform.client_secret> 21 <api.id>${ilg.api.version}</api.id> 22 <anypoint.platform.config.analytics.agent.enabled>true</anypoint.platform.config.analytics.agent.enabled> 23 </properties> 24 </cloudHubDeployment> 25 <executions> 26 <execution> 27 <id>deploy</id> 28 <goals> 29 <goal>deploy</goal> 30 </goals> 31 <configuration> 32 <classifier>mule-application</classifier> 33 </configuration> 34 </execution> 35 </executions> 36 37 <classifier>mule-application</classifier> 38 </configuration> 39 </plugin> The <cloudHubDeployment> tag of the Mule Maven plugin defines the attributes of your deployable. So, let's see what each of these properties tells us about the deployment onto CloudHub. uri: Mulesoft Anypoint platform URL muleVersion: Mulesoft version to be used for deployment(e.g. 4.2.3). username: Username for Anypoint platform password: Password for Anypoint platform businessGroup: Business group of the Anypoint platform under which you want to deploy your application workers: The number of workers to be assigned to the application. This is used for horizontal scaling workerType: The number of vCores to be assigned to the application. This is used for vertical scaling region: Region to be used while deploying the application environment: Environment onto which application needs to be deployed applicationName: Application name to be deployed properties: Provide here all the properties which are required by application at deployment time i.e. encryption key Step 2: Create a New Pipeline Log in to your Azure DevOps account. There are two ways to create a pipeline. First, you can go to the Pipelines tab and click a new pipeline. Creating a new pipeline Or, you can go to your repositories where you have your files and select the Build Pipeline option. Build pipeline from repositories So, let's create it using the Create pipeline option. Go to Pipeline and Create Pipeline. Creating pipeline Step 3: Select Classic Editor Now, select Classic Editor as it gives better customization. Using classic editor Step 4: Select Source You can select your source where your code is stored. In my case, it’s in Azure Repos. Further, select your Team Project, Repository and branch. Then, click continue. Continue Step 5: Select Maven Template Now, Select the Maven template. Maven template This is the default screen you’ll see. Default screen I’ll be removing all the three steps created under the Agent job and customize it according to my requirements. Customization Step 6: Download Secure File Firstly, I am adding “Download secure file.” I’ll be using it to download settings.xml. Download secure file But to download and use secure file in our pipeline, we need to store our file somewhere. In order to add settings.xml, go to Pipelines → Library → Secure files and add (+) Secure files. Then navigate to settings.xml: Settings.xml Make sure your settings.xml has the below server and profile added under the respective tags. XML xxxxxxxxxx 1 27 1 <server> 2 <id>Exchange2</id> 3 <username>guest-mule</username> 4 <password>Mulesoft1</password> 5 </server> 6 7 <profile> 8 9 <id>Mule</id> 10 <activation> 11 <activeByDefault>true</activeByDefault> 12 </activation> 13 <repositories> 14 <repository> 15 <id>Exchange2</id> 16 <name>Exchange2</name> 17 <url>https://repository.mulesoft.org/nexus-ee/content/repositories/releases-ee/</url> 18 <layout>default</layout> 19 <releases> 20 <enabled>true</enabled> 21 </releases> 22 <snapshots> 23 <enabled>true</enabled> 24 </snapshots> 25 </repository> 26 </repositories> 27 </profile> Now the settings that we created need to be selected, as shown below. Selecting settings Step 7: Create Reference Name Now let’s create a reference name “mulesoftSettingsFile” for this settings file, which we will use later in maven goals. Creating reference name Step 8: Add Maven Let’s add the next step in the Agent job, which will be Maven. Adding Maven In Options, we have given the secure settings file a reference name, which we created in the previous step. Providing reference name Step 9: Add Variables We require "username" and "password" for the Anypoint platform to be provided at build time, so lets create two variables for them and pass it into the Maven setup for deployment. Add variable Step 10: Save and Deploy Now let’s add the final Step, which will be performed by our Agent. Package deploy Now let's run the pipeline. We can either save the pipeline using "Save" or save it and run using the "Save and run" option. Saving the pipeline Once the pipeline is executed, you will see the below screen. Finalize job And you will be able to see the application deployed onto CloudHub. We can add more steps into the CI/CD pipeline like code analysis, testing, and report generation. Thus, we have successfully created a CI/CD pipeline using Azure DevOps and deployed an application to CloudHub.

By Gaurav Dhimate

CORE

The DevOps Toolchain

In this article, we’ll be looking at the core technology categories and specific technologies that help you realize your DevOps objectives. What Is a DevOps Toolchain? A DevOps toolchain is a set of integrated tools that facilitate and automate the processes involved in software development, testing, deployment, and operations within a DevOps framework. It spans the entire software development lifecycle, aiming to streamline collaboration, enhance efficiency, and ensure continuous delivery. The toolchain typically includes tools for version control, build automation, continuous integration, continuous deployment, configuration management, monitoring, and collaboration. By connecting these tools in a cohesive workflow, a DevOps toolchain promotes a seamless and automated pipeline, allowing development and operations teams to work collaboratively, iterate rapidly, and deliver high-quality software with speed and reliability. The ultimate goal is to achieve continuous improvement and faster time-to-market while maintaining the stability and reliability of the software. About DevOps and Tools Things to remember about DevOps and its corresponding tools: The goal is continuous improvement. You can’t buy DevOps. Tool adoption happens in stages. DevOps Technology Categories 1. Planning Why Does It Matter for DevOps? Shared purpose: Planning in DevOps is crucial for establishing a shared purpose among cross-functional teams. Through collaborative planning, teams align their goals, priorities, and expectations, fostering a unified understanding of the project's objectives. This shared purpose promotes cohesion and collaboration, ensuring that everyone is working towards common goals, which is essential for the success of a DevOps initiative. Transparency: Planning provides transparency into the entire software development lifecycle, from ideation to deployment. Transparent planning processes enable all team members to have visibility into project timelines, progress, and dependencies. This transparency facilitates communication and helps teams identify potential bottlenecks or challenges early in the development cycle, allowing for proactive problem-solving and mitigating risks. Empowerment: Planning empowers DevOps teams by involving them in decision-making processes and giving them a sense of ownership. When teams actively participate in planning, they have a better understanding of their responsibilities and how their work contributes to the overall project. This empowerment fosters a culture of accountability, innovation, and continuous improvement, as team members are motivated to take ownership of their tasks and contribute to the success of the entire DevOps pipeline. Examples of Planning Tools GitLab GitLab is a web-based DevOps lifecycle tool. It provides a Git repository manager, which provides a wiki, issue tracking, and CI/CD pipeline features, using an open-source license developed by GitLab, Inc. GitLab has some tools for actually planning things out and being able to look at the scope of the work. Tasktop: Tasktop allows all of these tools to be added to the entire range of Agile and application lifecycle management (ALM), PPM, and ITSM tools supported by Tasktop, enabling unprecedented visibility across the entire lifecycle. CollabNet’s VersionOne: VersionOne supports Scrum, Kanban, XP, SAFe, and hybrid development methodologies and makes it easier to plan, track, and report across all teams, programs, software portfolios, and the enterprise. Pivotal Tracker: The Agile project-management tool of choice for developers around the world for real-time collaboration around a shared, prioritized backlog Trello: Trello is a web-based Kanban-style list-making application that’s a subsidiary of Atlassian. It’s used by many teams as they plan out individual Sprints of work. Azure Boards: Track your software projects and plan better with Agile tools, including Scrum boards, Kanban boards, and dashboards for any Agile methodology. And, again, these are ways to visualize work, share plans, track progress, and make sure you’re moving toward the goal. 2. Issue Tracking Why Does It Matter for DevOps? Customer responsiveness: By systematically logging and addressing issues, teams can promptly identify and resolve customer-reported problems. This ensures a rapid response to user feedback and enhances customer satisfaction. Moreover, issue tracking allows DevOps teams to prioritize and address issues efficiently, demonstrating a commitment to delivering a reliable and customer-focused product. Limited knowledge waste: By documenting and categorizing issues, teams create a centralized repository of information that can be referenced to understand past challenges and solutions. This knowledge base reduces redundancy and prevents team members from reinventing the wheel when encountering similar issues. This streamlined approach to issue resolution contributes to improved efficiency and a more informed decision-making process across the entire DevOps lifecycle. Feedback loop: Establishing a feedback loop within DevOps workflows enables teams to continuously monitor and evaluate the performance of their applications, infrastructure, and development processes. Through the analysis of tracked issues, teams can identify patterns, root causes, and areas for improvement. This iterative feedback loop helps DevOps teams refine their practices and deliver higher quality software with each iteration. Examples of Issue-Tracking Tools Atlassian’s Jira: Jira is a proprietary issue-tracking product developed by Atlassian that allows bug tracking and Agile project management. JetBrains’ YouTrack: YouTrack is a proprietary, commercial browser-based bug tracker, issue-tracking system, and project-management software developed by JetBrains. It focuses on query-based issue searching — with auto-completing, manipulating issues in batches, customizing the set of issue attributes, and creating custom workflows. Zendesk: Zendesk makes it simple to track every customer issue from all of your channels. Many companies, including Uber and Airbnb, use Zendesk Support as their issue-tracking software. 3. Source Control Why Does It Matter for DevOps? Asset control: Source control is fundamental to DevOps as it provides a centralized and versioned repository for managing source code and related assets. This ensures precise control over the development artifacts, allowing teams to track changes, rollback to previous states, and maintain a clear audit trail of modifications. Asset control through source control is critical for maintaining the integrity of the codebase, enabling reproducibility, and supporting collaborative development efforts within a DevOps environment. Limit transportation waste: Source control helps limit transportation waste in the software development process. Instead of manually sharing code files or relying on ad-hoc methods, source control systems offer a structured and automated way to manage code changes. Developers can efficiently collaborate by pulling, merging, and pushing changes through a controlled and versioned process. This reduces the risk of errors, enhances code consistency, and minimizes the time and effort spent on coordinating code changes. Empower teams: Source control empowers DevOps teams by providing a platform for collaborative and distributed development. With versioned repositories, teams can work concurrently on different features or fixes without conflicts, fostering parallel development. Additionally, source control platforms often include features like branching and merging, enabling teams to experiment with new ideas, isolate changes, and integrate them seamlessly. This empowerment enhances team agility, allowing them to adapt to changing requirements and deliver high-quality software in a more responsive and collaborative manner. Examples of SCM Tools Git: Git is a distributed version-control system for tracking changes in source code during software development. It’s designed for coordinating work among programmers, but it can be used to track changes in any set of files. Its goals include speed, data integrity, and support for distributed, nonlinear workflows. GitHub: GitHub offers the distributed-version control and source code management functionality of Git, plus its own features. GitLab: Version control in GitLab helps your development team share, collaborate, and maximize their productivity with world-class source code management. Bitbucket: Bitbucket is a web-based version control-repository hosting service owned by Atlassian for source code and development projects that use either Mercurial or Git revision-control systems. Subversion: Apache Subversion is a software-versioning and revision-control system distributed as open source under the Apache License. Software developers use Subversion to maintain current and historical versions of files, such as source code, web pages, and documentation. 4. Build Tools Why Do They Matter for DevOps? Consistent packaging: Build tools are essential in DevOps for ensuring consistent packaging of software artifacts. These tools automate the compilation, packaging, and assembly of code into executable binaries or deployable packages. This automation guarantees that every build follows a standardized process, reducing the likelihood of configuration errors and ensuring that the generated artifacts are consistently structured. Consistent packaging is crucial for seamless deployment and helps avoid compatibility issues across different environments in the development, testing, and production stages. Automates an error-prone activity: Building software manually can be error-prone and time-consuming. Build tools automate the build process, eliminating human errors associated with compiling code, managing dependencies, and packaging applications. By automating these repetitive tasks, build tools reduce the risk of introducing inconsistencies or defects in the final build. This automation not only improves accuracy but also frees up developers' time, allowing them to focus on more complex and creative aspects of software development. Yields early quality signals: Build tools play a pivotal role in providing early quality signals during the development process. Automated builds can incorporate various static code analysis and testing tools to assess code quality, identify potential bugs, and ensure adherence to coding standards. By integrating these checks into the build process, developers receive immediate feedback on the health of their code, enabling them to address issues early in the development cycle. This proactive approach to quality assurance contributes to the overall stability and reliability of the software, supporting the principles of continuous integration and continuous delivery in DevOps. Examples of Build Tools Maven/Gradle: Maven is a build-automation tool used primarily for Java projects. Maven can also be used to build and manage projects written in C#, Ruby, Scala, and other languages. Gradle is an open-source build-automation system that builds upon the concepts of Apache Ant and Apache Maven and introduces a Groovy-based domain-specific language instead of the XML form used by Apache Maven for declaring the project configuration. Related: How to publish Maven artifacts to Nexus OSS using pipelines. MSBuild: Microsoft Build Engine, better known as MSBuild, is a free and open-source build tool set for managed code as well as native C++ code and was part of .NET Framework. Visual Studio depends on MSBuild (but not vice versa). Rake: Rake is a software task-management and build-automation tool. It allows the user to specify tasks and describe dependencies. You can also do this for group tasks in a namespace. JFrog Artifactory: JFrog Artifactory is a tool designed to store the binary output of the build process for use in distribution and deployment. Artifactory provides support for a number of package formats, such as Maven, Debian, npm, Helm, Ruby, Python, and Docker. JFrog offers high availability, replication, disaster recovery, and scalability, and works with many on-prem and cloud storage offerings. Sonatype Nexus: Sonatype Nexus is a repository manager. It allows you to proxy, collect, and manage your dependencies so you’re not constantly juggling a collection of JARs. It makes it easy to distribute your software. Internally, you configure your build to publish artifacts to Nexus, and they then become available to other developers. NuGet: NuGet is the package manager for .NET. The NuGet client tools provide the ability to produce and consume packages. The NuGet Gallery is the central package repository used by all package authors and consumers. 5. Testing Tools Why Do They Matter for DevOps? Puts focus on built-in quality: Testing tools are integral to DevOps as they emphasize the importance of built-in quality throughout the software development lifecycle. By incorporating automated testing tools, teams can establish a comprehensive testing strategy that includes unit tests, integration tests, and end-to-end tests. This approach shifts the focus from identifying and fixing defects later in the development process to preventing them from being introduced in the first place. Creates confidence in deployed artifacts: Testing tools play a critical role in creating confidence in the artifacts deployed in production environments. Automated testing, including regression testing and performance testing, helps ensure that new code changes do not introduce unintended side effects or performance degradation. By thoroughly validating each build, testing tools provide teams with the assurance that the deployed artifacts meet predefined quality criteria and are less likely to cause disruptions in production. Examples of Testing Tools JUnit: JUnit is a unit-testing framework for the Java programming language. JUnit has been important in the development of test-driven development and is one of a family of unit-testing frameworks, collectively known as xUnit, that originated with SUnit. xUnit.net: xUnit.net is an open-source unit-testing tool for the .NET Framework written by the original author of NUnit. Selenium: Selenium is a portable framework for testing web applications. Selenium provides a playback tool for authoring functional tests without the need to learn a test-scripting language. Jasmine: Jasmine is an open-source testing framework for JavaScript. It aims to run on any JavaScript-enabled platform, to not intrude on the application or the IDE, and to have an easy-to-read syntax. It’s heavily influenced by other unit-testing frameworks, such as Screw.Unit, JSSpec, JSpec, and RSpec. Cucumber: Cucumber is a software tool that supports behavior-driven development. Central to the Cucumber BDD approach is its ordinary-language parser called Gherkin. It allows expected software behaviors to be specified in a logical language that customers can understand. 6. Continuous Integration and Deployment (CI/CD) Why Do They Matter for DevOps? Fast feedback: Continuous Integration (CI) and Continuous Deployment (CD) are crucial in DevOps for providing fast and continuous feedback on code changes. Through CI, developers integrate their code changes into a shared repository multiple times a day. Automated build and test processes are triggered with each integration, allowing teams to detect and address issues early in the development cycle. This rapid feedback loop enables developers to identify and resolve problems quickly, resulting in greater agility and responsiveness in the development process. Reduce defect waste and waiting waste: CI/CD pipelines significantly contribute to waste reduction by minimizing both defect waste and waiting waste. Defect waste is reduced as automated testing in CI/CD pipelines helps catch and rectify issues early, preventing the propagation of defects to subsequent stages. Waiting waste is minimized through the automated and continuous deployment process, ensuring that code changes move swiftly from development to production without unnecessary delays. This efficiency in the development pipeline reduces idle time and accelerates the delivery of features, improvements, or bug fixes. Examples of CI Tools Jenkins: Jenkins is a free and open-source automation server. Jenkins helps to automate the nonhuman part of the software development process with CI and facilitates technical aspects of CD. It’s a server-based system that runs in servlet containers, such as Apache Tomcat. DZone’s previously covered Jenkins vs Bamboo CI. CircleCI: CircleCI is the world’s largest shared CI/CD platform and the central hub where code moves from idea to delivery. As one of the most-used DevOps tools, processing more than 1 million builds a day, CircleCI has unique access to data on how engineering teams work and how their code runs. Companies like Spotify, Coinbase, Stitch Fix, and BuzzFeed use CircleCI to improve engineering-team productivity, release better products, and get to it faster. Travis CI: Travis CI is a hosted CI service used to build and test software projects hosted on GitHub. Travis CI provides various paid plans for private projects and a free plan open source. TravisPro provides custom deployments of a proprietary version on the customer’s own hardware. Concourse: Concourse is an automation system written in Go. It’s most commonly used for CI/CD and is built to scale to any kind of automation pipeline, from simple to complex. AWS CodePipeline: AWS CodePipeline is a fully managed CD service that helps you automate your release pipelines for fast and reliable application and infrastructure updates. CodePipeline automates the build, test, and deploy phases of your release process every time there’s a code change, based on the release model you define. This enables you to rapidly and reliably deliver features and updates. You can easily integrate AWS CodePipeline with third-party services, such as GitHub, or with your own custom plugin. With AWS CodePipeline, you only pay for what you use. There are no upfront fees or long-term commitments. Azure Pipelines: Get cloud-hosted pipelines for Linux, macOS, and Windows. Build web, desktop, and mobile applications. Deploy to the cloud or on-premises. Automate your builds and deployments with Pipelines so you spend less time with the nuts and bolts and more time being creative. 7. CD Tools Why Do They Matter for DevOps? Limit your preproduction inventory: Continuous Deployment (CD) tools are essential in DevOps for limiting preproduction inventory, meaning the backlog of completed but not yet deployed features or changes. CD tools automate the process of deploying code changes to production, ensuring that features are released as soon as they pass necessary tests. This reduces the time between development completion and actual deployment, minimizing the backlog of unimplemented features and allowing organizations to deliver value to customers more rapidly. Automate complex pipelines: In a DevOps environment, software delivery often involves intricate workflows, including various testing stages, approvals, and environment deployments. CD tools automate these pipelines, orchestrating the entire process from code integration to production deployment. Automation not only speeds up the delivery process but also ensures consistency and reliability, reducing the likelihood of manual errors and enhancing the overall efficiency of the deployment workflow. Unify team around getting value to customers: CD tools contribute to unifying cross-functional teams around the common goal of delivering value to customers. By automating the deployment process, CD tools facilitate collaboration between development, testing, and operations teams. The automation streamlines communication, removes silos, and encourages shared responsibility for the end-to-end delivery pipeline. Examples of CD Tools Spinnaker: Spinnaker is a free and open-source CD software platform originally developed by Netflix and quickly picked up and extended by Google. Spinnaker is a multi-cloud CD platform for releasing software changes with high velocity and confidence. It combines a powerful and flexible pipeline-management system with integrations to the major cloud providers. Learn more about CI/CD with Jenkins and Spinnaker. Octopus Deploy: Octopus Deploy is an automated deployment and release-management tool used by leading CD teams worldwide. Octopus is a toolset that can drastically streamline any DevOps process for continuous testing and deployment of numerous microservices or applications via cloud or virtual machines. AWS CodeDeploy: AWS CodeDeploy is a fully managed deployment service that automates software deployments to a variety of compute services, such as Amazon EC2, AWS Fargate, AWS Lambda, and your on-premises servers. You can use AWS CodeDeploy to automate software deployments, eliminating the need for error-prone manual operations. 8. Configuration-Management Tools Why Do They Matter for DevOps? Enforce consistency: Configuration-management tools enable the automation of configuration tasks, ensuring that servers, networks, and other components are configured identically and in accordance with specified standards. By enforcing consistency, configuration-management tools help mitigate configuration drift – the unintended variance in configurations that can lead to operational issues and security vulnerabilities. Consistency promotes reliability, reduces the risk of errors, and streamlines the management of diverse infrastructure components within a DevOps environment. Treat Infrastructure as Code: Configuration-management tools encourage the practice of treating Infrastructure as Code (IaC). This involves representing infrastructure configurations in a code format, allowing for version control, collaboration, and automation. By adopting IaC principles, DevOps teams can manage and provision infrastructure in a systematic and repeatable manner. Configuration changes are tracked, and documented, and can be rolled back if needed. This approach aligns infrastructure management with software development practices, enhancing agility, traceability, and the ability to apply DevOps principles to the entire infrastructure lifecycle. Examples of Configuration-Management Tools Terraform: Terraform is an open-source infrastructure-as-code (IaC) software tool created by HashiCorp. It enables users to define and provision a data-center infrastructure using a high-level configuration language known as Hashicorp Configuration Language or, optionally, JSON. BOSH: BOSH is a project that unifies release engineering, deployment, and the lifecycle management of small and large-scale cloud software. BOSH can provision and deploy software over hundreds of VMs. It also performs monitoring, failure recovery, and software updates with zero-to-minimal downtime. While BOSH was developed to deploy Cloud Foundry, it can also be used to deploy almost any other software (Hadoop, for instance). BOSH is particularly well-suited for large distributed systems. In addition, BOSH supports multiple infrastructure-as-a-service (IaaS) providers like VMware vSphere, Google Cloud Platform, Amazon Web Services EC2, Microsoft Azure, OpenStack, and Alibaba Cloud. There’s a Cloud Provider Interface (CPI) that enables users to extend BOSH to support additional IaaS providers, such as Apache CloudStack and VirtualBox. Chef: Chef is a configuration-management tool for dealing with machine setup on physical servers, virtual machines, and in the cloud. Many companies use Chef software to control and manage their infrastructure, including Facebook, Etsy, Cheezburger, and Indiegogo. Chef is the leader in continuous automation software, an innovator in application automation, and one of the founders of the DevOps movement. Chef works with more than 1,000 of the most innovative companies around the world to deliver their vision of digital transformation, providing the practices and platform to deliver software at speed Ansible: Ansible is an open-source software-provisioning, configuration-management, and application-deployment tool. It runs on many Unix-like systems and can configure both Unix-like systems as well as Microsoft Windows. It includes its own declarative language to describe system configuration. Puppet: Puppet provides the ability to define which software and configuration a system requires and then maintains a specified state after an initial setup. You use a declarative domain-specific language (DSL) that’s similar to Ruby to define configuration parameters for a specific environment or infrastructure. Puppet discovers information about a system by using a utility called Facter, which is installed when you install the Puppet software package. The Puppet master is the system that manages important configuration information for all of the nodes that it controls by using manifests. The nodes that the master controls are those that have Puppet installed on them and are running the Puppet agent, which is a daemon. The configuration information that the agent collects about a node is sent to the Puppet master. The Puppet master then compiles a catalog based on how the node should be configured. Each node uses that information to apply any necessary configuration updates to itself. Puppet works by using a pull mode, where agents poll the master at regular intervals to retrieve site-specific and node-specific configurations. In this infrastructure, managed nodes run the Puppet agent application, typically as a background service. Google Cloud Deployment Manager: Google Cloud Deployment Manager is an infrastructure-management service that makes it simple to create, deploy, and manage Google Cloud Platform resources. 9. Cloud Platforms Why Do They Matter for DevOps? Automation-friendly: Cloud platforms provide DevOps teams with automation-friendly environments that support the seamless integration of automated processes throughout the software development lifecycle. These platforms offer a wide range of APIs, services, and tools that facilitate the automation of infrastructure provisioning, configuration management, deployment, scaling, and monitoring tasks. By leveraging cloud-native automation capabilities, DevOps teams can streamline workflows, reduce manual intervention, and achieve greater agility and efficiency in delivering software. Observable runtimes for software: Cloud platforms offer observable runtimes that enable comprehensive monitoring and debugging of software applications in production environments. With built-in monitoring, logging, and tracing capabilities, DevOps teams can gain real-time insights into the performance, health, and behavior of their applications running on cloud infrastructure. This visibility allows teams to quickly detect and diagnose issues, optimize resource utilization, and ensure the reliability and availability of their software. Examples of Cloud Platforms Amazon Web Service: Amazon Web Services (AWS) is a secure cloud-services platform, offering compute power, database storage, content delivery, and other functionality to help businesses scale and grow. In simple words, AWS allows you with the following things: Running web and application servers in the cloud to host dynamic websites Securely store all your files on the cloud so you can access them from anywhere Using managed databases, like MySQL, PostgreSQL, Oracle, or SQL Server, to store information Deliver static and dynamic files quickly around the world using a content delivery network (CDN) Send bulk email to your customers Microsoft Azure: Microsoft Azure is a cloud-computing service created by Microsoft for building, testing, deploying, and managing applications and services through Microsoft-managed data centers. Microsoft Azure, formerly known as Windows Azure, is Microsoft’s public cloud-computing platform. It provides a range of cloud services, including compute, analytics, storage, and networking. Users can pick and choose from these services to develop and scale new applications or run existing applications in the public cloud. Google Cloud Platform: Google Cloud Platform, offered by Google, is a suite of cloud-computing services that run on the same infrastructure that Google uses internally for its end-user products, such as Google Search, Gmail, and YouTube. Google Cloud Platform is a provider of computing resources for deploying and operating applications on the web. Its specialty is providing a place for individuals and enterprises to build and run software, and it uses the web to connect to the users of that software. Meet your business challenges head-on with cloud-computing services from Google, including data management, hybrid and multi-cloud, and AI and ML. Pivotal Cloud Foundry: Pivotal Cloud Foundry (PCF) is an open-source, multi-cloud application platform as a service (PaaS) governed by the Cloud Foundry Foundation, a 501 organization. The software was originally developed by VMware but was transferred to Pivotal Software. It was brought back into VMware at the end of 2019 with VMware’s takeover of Pivotal. PCF is a multi-cloud platform for the deployment, management, and continuous delivery of applications, containers, and functions. PCF allows developers to deploy and deliver software quickly, without needing to manage the underlying infrastructure. Heroku: Heroku is a cloud PaaS supporting several programming languages. One of the first cloud platforms, Heroku has been in development since June 2007, when it supported only the Ruby programming language, but now supports Java, Node.js, Scala, Clojure, Python, PHP, and Go. Heroku is a container-based cloud PaaS. Developers use Heroku to deploy, manage, and scale modern apps. Our platform is elegant, flexible, and easy to use, offering developers the simplest path to getting their apps to et. Container schedulers: The main task of a container scheduler is to start containers on the most appropriate host and connect them together. It has to handle failures by doing automatic failovers, and it needs to be able to scale containers when there’s too much data to process/compute for a single instance. The three most popular container schedulers are Docker Swarm, Apache Mesos, and Google Kubernetes. Docker Swarm: Docker Swarm is a container scheduler developed by Docker. The development of this cluster solution being managed by Docker offers advantages like the use of the standard Docker API. Apache Mesos: The purpose of Mesos is to build a scalable and efficient system that supports a wide array of both current and future frameworks. This is also the main issue: frameworks like Hadoop and MPI are developed independently — thus it’s not possible to do fine-grained sharing across frameworks. The proposition of Mesos is to add a thin resource-sharing layer, giving frameworks a common interface for accessing cluster resources. Mesos is delegating control over scheduling to the frameworks because many frameworks already implement sophisticated scheduling. Four types of frameworks exist depending on the type of jobs you want to run on your cluster, and some of them have native Docker support like Marathon. The support of Docker containers has been added in Mesos 0.20.0. We’ll focus on the use of Mesos with Marathon because it’s a framework actively maintained by Mesosphere that offers many features in terms of scheduling, such as constraints, health checks, service discovery, and load balancing. As we can see in the image, there are four elements in the cluster. ZooKeeper helps Marathon look up the address of the Mesos master — multiple instances are available to handle failure. Marathon starts, monitors, and scales the containers. The Mesos master sends the tasks assigned to a node and makes offers to Marathon when a node has some free CPU/RAM. The Mesos slaves run the container and submit a list of their available resources. Google Kubernetes: Kubernetes is an orchestration system for Docker containers using the concepts of labels and pods to group containers into logical units. Pods are the main difference between Kubernetes and the two other solutions — they’re collections of colocated containers forming a service deployed and scheduled together. This approach simplifies the management of the cluster — compared to an Affinity-based co-scheduling of containers (like Swarm and Mesos). The Kubernetes scheduler’s task is to watch for pods that have an empty PodSpec. NodeName attributes a value to schedule the container somewhere in the cluster. This is a difference compared to Swarm and Mesos, as Kubernetes allows developers to bypass the scheduler by defining a PodSpec.NodeName when running the pod. The scheduler uses predicates and priorities to define which nodes a pod should run. The default values of these parameters can be overridden using a new scheduler policy configuration. By using the command-line flag — policy-config-file — pointing to a JSON file describing the predicates and priorities to use when starting Kubernetes, the scheduler will use the policy defined by the administrator. 10. Monitoring and Logging Tools Why Do They Matter for DevOps? Fast recovery: Monitoring and logging tools allow DevOps teams to swiftly detect and address system failures or performance issues, minimizing downtime and restoring normal operations quickly. Real-time insights provided by these tools enable rapid response, ensuring business continuity and mitigating the impact on end-users. Responsiveness: By collecting and analyzing metrics and logs in real-time, monitoring and logging tools empower DevOps teams to proactively identify and resolve potential issues before they escalate. This proactive approach enhances system reliability and responsiveness, maintaining a positive user experience and instilling confidence in the software's reliability. Transparency: Centralizing logs and metrics from various components provides a holistic view of system health, facilitating collaboration and informed decision-making among team members. Transparent monitoring practices foster accountability and trust within the organization, as stakeholders have access to accurate and up-to-date information about system performance. Limited human involvement during incidents: Automated alerting and remediation workflows reduce the need for manual intervention during incidents, freeing up DevOps teams to focus on strategic tasks. By automating incident response, monitoring and logging tools improve operational efficiency, shorten incident resolution times, and enhance the overall reliability of systems. Examples of Monitoring/Logging Tools ELK Stack: The ELK Stack is a collection of three open-source products — Elasticsearch, Logstash, and Kibana. They’re all developed, managed, and maintained by the company Elastic. E stands for ElasticSearch — used for storing logs L stands for Logstash — used for both shipping as well as processing and storing logs K stands for Kibana — a visualization tool (a web interface) Datadog: Datadog is a monitoring service for cloud-scale applications, providing monitoring of servers, databases, tools, and services through a SaaS-based data analytics platform. Datadog Application Performance Monitoring (APM or tracing) provides you with deep insight into your application’s performance — from automatically generated dashboards for monitoring key metrics, like request volume and latency, to detailed traces of individual requests — side by side with your logs and infrastructure monitoring. When a request is made to an application, Datadog can see the traces across a distributed system and show you systematic data about precisely what is happening to this request. New Relic: New Relic is a San Francisco, California–based technology company that develops cloud-based software to help website and application owners track the performances of their services. New Relic’s software analytics product for application-performance monitoring (APM) delivers real-time and trending data about your web application’s performance and the level of satisfaction your end users experience. Prometheus: Prometheus is a free software application used for event monitoring and alerting. It records real-time metrics in a time-series database, built using an HTTP pull model with flexible queries and real-time alerting. The Prometheus server works on the principle of scraping — i.e., invoking the metrics endpoints of the various nodes that it’s configured to monitor. It collects these metrics at regular intervals and stores them locally. The nodes expose these over the endpoints that the Prometheus server scrapes. Zipkin: Zipkin is a distributed tracing system. It helps gather timing data needed to troubleshoot latency problems in service architectures. Features include both the collection and lookup of this data. If you have a trace ID in a log file, you can jump directly to it. Otherwise, you can query based on attributes such as service, operation name, tags, and duration. Some interesting data will be summarized for you, such as the percentage of time spent in a service and whether or not operations have failed. Azure Monitor: Azure Monitor maximizes the availability and performance of your applications and services by delivering a comprehensive solution for collecting, analyzing, and acting on telemetry from your cloud and on-premises environments. It helps you understand how your applications are performing and proactively identifies issues affecting them and the resources they depend on. Here are just a few examples of what you can do with Azure Monitor include: Detect and diagnose issues across applications and dependencies with Application Insights Correlate infrastructure issues with Azure Monitor for VMs and Azure Monitor for Containers Drill into your monitoring data with Log Analytics for troubleshooting and deep diagnostics Support operations at scale with smart alerts and automated actions Create visualizations with Azure dashboards and workbooks 11. Communication Tools Why Do They Matter for DevOps? Connect teams: Communication tools are vital for DevOps as they facilitate seamless communication and collaboration among distributed teams. These tools provide instant messaging, video conferencing, and collaborative workspace features, enabling team members to connect and communicate in real-time regardless of geographical location. By fostering continuous communication, these tools ensure that all team members stay informed and engaged throughout the development process. Limit waiting waste: Communication tools help reduce waiting waste in DevOps workflows by enabling swift decision-making and issue resolution. With instant messaging and real-time collaboration features, teams can quickly escalate and address issues, preventing delays in the development pipeline. By minimizing the time spent waiting for responses or approvals, communication tools optimize workflow efficiency and accelerate the delivery of software. Improve collaboration: Effective collaboration is essential for successful DevOps practices, and communication tools play a key role in facilitating collaboration among team members with diverse roles and expertise. These tools provide centralized platforms for sharing information, documents, and updates, fostering a collaborative environment where ideas can be exchanged, feedback can be provided, and decisions can be made collectively. Improved collaboration enhances teamwork, innovation, and the overall effectiveness of DevOps initiatives. Examples of Communication Tools Slack: Slack is a proprietary instant messaging platform developed by Slack Technologies. Slack is essentially a chat room for your whole company, designed to replace email as your primary method of communication and sharing. Its workspaces allow you to organize communications by channels for group discussions and allow for private messages to share information, files, and more all in one place. Microsoft Teams: Microsoft Teams is a unified communication and collaboration platform that combines persistent workplace chat, video meetings, file storage, and application integration. Teams is a chat-based collaboration tool that provides global, remote, and dispersed teams with the ability to work together and share information via a common space. You can utilize cool features like document collaboration, one-on-one chat, team chat, and more. Google Hangouts: Google Hangouts is a communication software product developed by Google. Originally a feature of Google+, Hangouts became a stand-alone product in 2013, when Google also began integrating features from Google+ Messenger and Google Talk into Hangouts. Hangouts Chat is a powerful way to communicate with people in your organization. Start a message with one or more coworkers, create a room for ongoing conversations with groups of people, and use bots to help automate your work. Chat works in your computer browser and in mobile apps for Android and iOS. Zoom: Zoom Video Communications is a remote conferencing services company headquartered in San Jose, California. It provides a remote conferencing service that combines video conferencing, online meetings, chat, and mobile collaboration. Skype is better for teams looking for a holistic business solution. Zoom is a better fit for teams that have frequent video chats and meetings. However, neither tool was specifically built for remote work. 12. Knowledge-Sharing Tools Why Do They Matter for DevOps? Reduce knowledge waste: Knowledge-sharing tools are essential for DevOps as they provide a centralized repository for storing and sharing information, thereby reducing knowledge waste within the organization. By documenting best practices, lessons learned, and troubleshooting guides, these tools ensure that valuable insights and expertise are captured and accessible to all team members. This promotes knowledge sharing and collaboration, empowering teams to learn from each other's experiences and avoid reinventing the wheel. Increase new-hire productivity: Knowledge-sharing tools play a crucial role in increasing the productivity of new hires by providing them with access to onboarding materials, training resources, and institutional knowledge. These tools enable new team members to quickly get up to speed on project workflows, coding standards, and infrastructure configurations, accelerating their integration into the team and reducing the time required for onboarding. By facilitating knowledge transfer from experienced team members to newcomers, these tools ensure a smoother transition and faster ramp-up for new hires. Limit repeat mistakes: Knowledge-sharing tools help limit repeat mistakes by capturing and disseminating information about past issues, solutions, and workarounds. By documenting root causes and resolutions for common problems, these tools enable teams to learn from past mistakes and avoid repeating them in the future. This proactive approach to knowledge management promotes continuous improvement and helps teams maintain high standards of quality and reliability in their work. Examples of Knowledge-Sharing Tools GitHub Pages: GitHub Pages is a static site-hosting service that takes HTML, CSS, and JavaScript files straight from a repository on GitHub, optionally runs the files through a build process, and publishes a website. You can see examples of GitHub Pages sites in the GitHub Pages examples collection. Confluence: Confluence is a collaboration software program developed and published by Atlassian. Atlassian wrote Confluence in the Java programming language and first published it in 2004. Confluence is a collaboration wiki tool used to help teams collaborate and share knowledge efficiently. With Confluence, you can capture project requirements, assign tasks to specific users, and manage several calendars at once with the help of the Team Calendars add-on. Jekyll: Jekyll is a simple, blog-aware, static site generator for personal, project, or organizational sites. Written in Ruby by Tom Preston-Werner, GitHub’s co-founder, it’s distributed under the open-source MIT license. Jekyll is a parsing engine, bundled as a Ruby gem, used to build static websites from dynamic components, such as templates, partials, liquid code, down, etc. Jekyll is known as “a simple, blog-aware, static-site generator.” Google Sites: Google Sites is a structured wiki- and webpage-creation tool offered by Google. The declared goal of Google Sites is for anyone to be able to create simple websites that support collaboration between different editors. Collaboratively create engaging, high-quality sites for your team, project, or event. The sites look great on every screen, from desktop to smartphone — all without learning design or programming. Summary I covered all these different categories that come together to collectively help you do DevOps. Any one of these is helpful, but I’d argue all of these are required for you to be in the place where you’re continuously delivering value to the customer in a way that’s sustainable to the team, as well as helping you differentiate. I covered the overall DevOps toolchain — things like planning tools, issue-tracking tools, source-control management, building and testing your code and being successful with that, continuously integrating and deploying your source code, managing configurations, using cloud platforms, and then monitoring and logging before doing some of the softer things like communication and knowledge sharing. DevOps is an exciting way to work. It’s not going to be the only way or even the best way. Something better may come along in the future, but for right now, it seems to represent a good way for teams to collectively work together to ship value that matters in a way that’s sustainable for the people on the team, while also helping them focus on the customer, not on IT efficiency — and not just on internal things but on speed, on learning, and on the things that actually matter as companies are differentiating now based on how good they are at technology. Thank you for reading this. I hope you enjoyed the article.

By Suresh Sekar

Jenkins: Publish Maven Artifacts to Nexus OSS Using Pipelines or Maven Jobs

Jenkins, in conjunction with Nexus OSS, forms a powerful combination for managing the deployment of Maven artifacts. Crucial in the Software Development Life Cycle (SDLC), this process typically falls within the build and release phases of the SDLC, where automating artifact deployment ensures consistency, reliability, and efficiency. For developers and DevOps engineers, understanding how to seamlessly integrate Jenkins pipelines or Maven jobs with Nexus OSS for artifact publication is essential for streamlining software delivery. In this article, we will cover the usage of Jenkins along with Nexus OSS and go through how we can publish Maven artifacts using Jenkins. Leveraging these tools effectively is the key to accelerating your development cycles and ensuring robust artifact management. Moreover, by mastering this integration, you'll gain the ability to automate artifact deployment with confidence, ultimately enhancing productivity and the quality of your software. So let's get our hands dirty. Assumptions Jenkins is running in: http://localhost:8080/ Nexus OSS 3 is running in: http://172.17.0.3:8081/ You can get more details about the Docker images I used in this article from these resources: Jenkins Continuous Integration and Delivery server Sonatype Nexus Repository Docker: sonatype/nexus3 I am using my own fork of the Cargo EE Java project as source code to build the artifact. The main difference is I'm not using it as SNAPSHOT. Step 1: Install "Nexus Artifact Uploader" and "Pipeline Utility Steps" Plugins The Nexus plugin IDs we are going to install are nexus-artifact-uploader and pipeline-utility-steps. For this, go to http://localhost:8080/pluginManager/available, search for "nexus-artifact-uploader," and install the plugin. nexus-artifact-uploader (plugins.jenkins.io) For steps to get help to easily read a pom.xml file, let's do the same for the Jenkins plugin id: pipeline-utility-steps. You can find more details about this plugin which contains a lot of very useful steps in the documentation about Pipeline Utility Steps. This plugin is a must if you are working with pipelines since contains a lot of useful steps. Step 2: Create a Hosted Repository in Nexus If you already have your repository configured in Nexus then you can skip this step; otherwise, you can create a Maven 2-hosted repository in Nexus OSS where we will upload our artifacts. In the image below, we are creating a hosted repository named "repository-example," which we'll use in this article. As for this example, we have chosen these configurations: Deployment policy: allow redeploy If you want to deploy as many times as you want the same version Storage: default Note: you can get deep on this and go for AWS S3 storage, which is a very nice feature enabled in the free version for Nexus. Step 3: Create a Valid User in Nexus OSS If you already have a login, you can skip this step. An example of how a user could look might be found in the below image. For simplicity's sake, this has admin rights, so you can choose and configure a user according to your needs. Step 4: Create Valid Jenkins Credentials To Authenticate To Nexus OSS In this step, we should add a Jenkins credential of the kind "Username with password" with a valid login to our Nexus instance, and let's give it an ID of "nexus-credentials." Go to: http://localhost:8080/credentials/. Note: I am using the default scope and domains in an as-is Jenkins installation. Step 5: Set Up Maven as A Managed Tool If you already have Maven in your classpath whenever your build will run (slave or master) then you should be fine and skip this; otherwise, it's always good to have it as a managed tool since Jenkins has built-in support for Maven. Go to http://localhost:8080/configureTools/ and set a Maven configuration. For this example, I chose Maven 3.6.0, so I'll name this as "Maven 3.6.0". Publishing Artifacts Using Jenkins Pipelines Below is a script I am using to publish an artifact in Nexus OSS. pipeline { agent { label "master" } tools { // Note: this should match with the tool name configured in your jenkins instance (JENKINS_URL/configureTools/) maven "Maven 3.6.0" } environment { // This can be nexus3 or nexus2 NEXUS_VERSION = "nexus3" // This can be http or https NEXUS_PROTOCOL = "http" // Where your Nexus is running NEXUS_URL = "172.17.0.3:8081" // Repository where we will upload the artifact NEXUS_REPOSITORY = "repository-example" // Jenkins credential id to authenticate to Nexus OSS NEXUS_CREDENTIAL_ID = "nexus-credentials" } stages { stage("clone code") { steps { script { // Let's clone the source git 'https://github.com/danielalejandrohc/cargotracker.git'; } } } stage("mvn build") { steps { script { // If you are using Windows then you should use "bat" step // Since unit testing is out of the scope we skip them sh "mvn package -DskipTests=true" } } } stage("publish to nexus") { steps { script { // Read POM xml file using 'readMavenPom' step , this step 'readMavenPom' is included in: https://plugins.jenkins.io/pipeline-utility-steps pom = readMavenPom file: "pom.xml"; // Find built artifact under target folder filesByGlob = findFiles(glob: "target/*.${pom.packaging}"); // Print some info from the artifact found echo "${filesByGlob[0].name} ${filesByGlob[0].path} ${filesByGlob[0].directory} ${filesByGlob[0].length} ${filesByGlob[0].lastModified}" // Extract the path from the File found artifactPath = filesByGlob[0].path; // Assign to a boolean response verifying If the artifact name exists artifactExists = fileExists artifactPath; if(artifactExists) { echo "*** File: ${artifactPath}, group: ${pom.groupId}, packaging: ${pom.packaging}, version ${pom.version}"; nexusArtifactUploader( nexusVersion: NEXUS_VERSION, protocol: NEXUS_PROTOCOL, nexusUrl: NEXUS_URL, groupId: pom.groupId, version: pom.version, repository: NEXUS_REPOSITORY, credentialsId: NEXUS_CREDENTIAL_ID, artifacts: [ // Artifact generated such as .jar, .ear and .war files. [artifactId: pom.artifactId, classifier: '', file: artifactPath, type: pom.packaging], // Lets upload the pom.xml file for additional information for Transitive dependencies [artifactId: pom.artifactId, classifier: '', file: "pom.xml", type: "pom"] ] ); } else { error "*** File: ${artifactPath}, could not be found"; } } } } } } Let's go through some very important details. The current plugin used in this article allows the use of the configuration of Nexus with versions of Nexus 3 and Nexus 2 along with HTTPS and HTTP protocols. The magic here is nexusArtifactUploader, so let's decompose some of the parameters: nexusVersion : It can be nexus3 or nexus2. protocol : It can be https or http. nexusUrl : For this example, it is the address where you have located your Nexus OSS instance without including the protocol "http" or "https." For this example, we use 172.17.0.3:8081, which is the IP address of my Docker container. repository : The name of the repository, in this case, is the one I created in Step 3. credentialsId : This is the ID of the credential stored in Jenkins. Try to specify a mnemonic ID in Jenkins. Otherwise, this will be auto-generated in Jenkins, assigning it a random hash value. artifacts: This is very important. A lot of examples I reviewed only upload the binary artifact, but it's so critical to upload our pom.xml file as type "pom." This is important to allow transitive dependencies in case our artifacts contain more information. Just as another project referencing it might need some "extra" information to figure out how or what dependencies to include, a lot of your builds might be just fine even without uploading this. It could compile or package them fine but you might face hard to track RuntimeExceptions due to a lack of dependencies. You could face issues with this kind of message in your Maven log builds: [WARNING] The POM for groupId:artifactId:version is missing, no dependency information available In order to avoid the messages above, we always should upload our pom.xml to Nexus, so that the parameter accepts an array of artifacts. You can clone the code in this GitHub repository with the script nexus/uploadArtifacts.groovy. The result should look very similar to this in our Nexus instance: Publishing Artifacts Using Maven Job First, I am not a fan of this type of job since it has heavy I/O operations in the folder job definition. In my personal opinion, all of these should happen exclusively in the workspace. So if we are using this type of job, we can perform exactly the same task we did in the previous example using Pipelines. This job has the particularity helping us define these variables which are quite self-explained with the information extracted from the pom.xml file: POM_DISPLAYNAME POM_VERSION POM_GROUPID POM_ARTIFACTID POM_PACKAGING Notice we are dynamically building the artifact name using the expression: target/${POM_ARTIFACTID}-${POM_VERSION}.${POM_PACKAGING}, but you know this can vary if you set a custom artifact name in the pom.xml. Repository URL: https://github.com/danielalejandrohc/cargotracker.git, branch: master Root POM: pom.xml, Goals: package -DskipTests=true In the post steps section, we set the final step to publish the artifact to Nexus OSS. Conclusion Having a way to distribute our artifacts is crucial to performing CI and CD. Having artifacts centralized in our organization can easily reduce the efforts to reproduce builds in different platforms. This means developers can connect their Maven settings.xml and perform a local build or we can migrate our CI platform without having to deal with moving artifacts from one place to another because they were stored locally. Nowadays, the most popular products for artifacts allow us to set our storage in the cloud so we can decouple one more layer providing us more availability and getting rid of worries about limited storage blocking us to delivery software.

By Daniel Hernandez

Setting Up Your Java Pipeline With Azure DevOps and Docker

Understanding how to organize a pipeline from development to operation has, in my experience, proven to be quite the endeavor. This tutorial aims to tackle precisely this challenge by guiding you through the required tools necessary to deploy your code as Docker containers by going through the steps involved in creating a simple "Hello, World!" application (although preexisting projects are also easily applicable for this approach). Whether you're a seasoned developer seeking to optimize your workflow or a newcomer eager to learn best practices, this tutorial will equip you with the knowledge and tools to streamline your development process effectively. Moreover, becoming proficient in this pipeline setup will greatly enhance your workflow, allowing you to deliver high-quality software faster, with fewer errors, and ultimately, better meet the demands of today's agile development environments. If you have come far enough to consider a pipeline for your project, I expect you to be familiar with some of the simpler tools involved in this process (e.g. Git, Java, Maven), and will not cover these in-depth. You may also enjoy: Building CI/CD Pipelines for Java Using Azure DevOps (Formerly VSTS) To go about making a pipeline for our "Hello, World!" application, the following subjects will briefly be covered: Azure DevOps Azure Repos Maven Git Azure Pipelines Docker To make things clear: Our goal is to be able to run docker run <dockerid>/<image>:<tag> while before, only having run git push on master. This is an attempt to create a foundation for future CI/CD implementations ultimately leading to a DevOps environment. Azure DevOps One of the prerequisites for this walk-through is to use the Azure DevOps platform. I can highly encourage the full package, but the modules Repos and Pipelines are the only ones required. So, if you have not already, you should sign yourself up and create a project. After doing so, we can proceed to the Repos module. Azure Repos This module provides some simple tools for maintaining a repository for your code. While a repository could easily be managed by something like GitHub, this module supports solid synergy between repositories and pipelines. After you click on the module, you will be met with the usual Git preface for setting up a repository. I highly recommend using the SSH methods for long-term usage (if this is unknown to you, see Connect to your Git repos with SSH). Now, after setting it up, you will be able to clone the repository onto your computer. Continuing, we will create a Maven project within the repository folder using IntelliJ IDEA (other IDEs can be used, but I will only cover IntelliJ) that ultimately prints the famous sentence, "Hello, World!" (for setting up a project with Maven, see Creating a new Maven project - IntelliJ). This should leave you with a project tree like so: Hello World project tree Finish off by creating a main class in src/main/java: Java x 1 public class Main { 2 public static void main(String[] args) { 3 System.out.println("Hello World!"); 4 } 5 } But before pushing these changes to master, a few things need to be addressed. Maven Maven provides developers with a powerful software management tool configurable from one location, the pom.xml file. Looking at the generated pom file in our project, we will see the following: XML xxxxxxxxxx 1 10 1 <?xml version="1.0" encoding="UTF-8"?> 2 <project xmlns="http://maven.apache.org/POM/4.0.0" 3 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 4 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> 5 <modelVersion>4.0.0</modelVersion> 6 7 <groupId>surgo.testing</groupId> 8 <artifactId>testing-helloworld</artifactId> 9 <version>1.0</version> 10 </project> In our case, the only really interesting part of the pom file is the version tag. The reason is that upon pushing our source code to master, Maven will require a new version each time — enforcing good practice. As an extension, we need to make Maven create an executable .jar file with a manifest of where the main class is to be located. Luckily, we can just use their own Maven plugin: XML xxxxxxxxxx 1 33 1 <?xml version="1.0" encoding="UTF-8"?> 2 <project xmlns="http://maven.apache.org/POM/4.0.0" 3 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 4 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> 5 <modelVersion>4.0.0</modelVersion> 6 7 <groupId>surgo.testing</groupId> 8 <artifactId>testing-helloworld</artifactId> 9 <version>1.0</version> 10 11 <properties> 12 <main.class>Main</main.class> 13 </properties> 14 15 <build> 16 <plugins> 17 <plugin> 18 <groupId>org.apache.maven.plugins</groupId> 19 <artifactId>maven-jar-plugin</artifactId> 20 <version>3.1.2</version> 21 <configuration> 22 <archive> 23 <manifest> 24 <addClasspath>true</addClasspath> 25 <classpathPrefix>lib/</classpathPrefix> 26 <mainClass>${main.class}</mainClass> 27 </manifest> 28 </archive> 29 </configuration> 30 </plugin> 31 </plugins> 32 </build> 33 </project> The only thing you might want to change is the name of the main class (line 12). Remember the package name, if not directly located in src/main/java (I prefer using properties, but you can insert the name directly in line 26 if you like). Lastly, before committing our additions to master, we will need to build the target folder which includes our .jar file. This can be done either directly through IntelliJ or in the terminal (if you have Maven installed). Simply press the lifecycle "package" in the UI, or run mvn package in the terminal. Upon finalization, a .jar file will have appeared in the target folder: This concludes the initial setup necessary for our pipeline and we can now finally push our changes to master. Git Most of you are probably quite familiar with Git, but I will go ahead and cover what needs to be done anyway. The Git tool provides us with a distributed version control system easily accessible from anywhere. Now provided we correctly configured our repository in Azure Repos, cloned it to our local computer, and initialized the IntelliJ project within that folder, it should be straightforward. As all of our added files have yet to be staged, run git add. This will stage every changed or added file. Then run git commit -m "initial commit" to commit the staged files. Lastly, run git push to push the committed files to master. You might now be wondering, "Has all the magic happened?" And the answer would be no. In fact, not much has happened. We have created a repository and filled it with a Maven project that prints "Hello, World!" when invoked, which in all honesty, is not much of an achievement. But, more importantly, we have established a foundation for our pipeline. Azure Pipelines Pipelines, the star of the show, provides us with build and deployment automation. It enables us to customize what should happen whenever a build is triggered (in our case by pushing to master). Let me take you through the process of setting up a simple pipeline. Step 1: First, go to the Azure DevOps Pipeline module. This will present you with a single button "Create Pipeline," press it. Step 2: We will now be prompted for the location of our code, and since we used Azure Repos, press "Azure Repos Git." Step 3: It will now look through your repositories. Press the one you pushed the Maven project onto. Step 4: Since it is a Maven project, select "Maven."You should now be presented with the following azure.pipelines.yml file: HXML xxxxxxxxxx 1 22 1 # Maven 2 # Build your Java project and run tests with Apache Maven. 3 # Add steps that analyze code, save build artifacts, deploy, and more: 4 # https://docs.microsoft.com/azure/devops/pipelines/languages/java 5 6 trigger: 7 - master 8 9 pool: 10 vmImage: 'ubuntu-latest' 11 12 steps: 13 - task: Maven@3 14 inputs: 15 mavenPomFile: 'pom.xml' 16 mavenOptions: '-Xmx3072m' 17 javaHomeOption: 'JDKVersion' 18 jdkVersionOption: '1.8' 19 jdkArchitectureOption: 'x64' 20 publishJUnitResults: true 21 testResultsFiles: '**/surefire-reports/TEST-*.xml' 22 goals: 'package' Do not think too much about the semantics of the file. The important thing to know now is that the trigger is set to master and the steps include a task for Maven. For more information about the Maven inputs, see Maven task. Step 5: If everything looks in order, press "save and run" in the top-right corner to add the azure.pipelines.yml file to the repository. The pipeline will then be activated and run its first job. Docker Docker, the final piece of the puzzle, provides us with an OS-level of virtualization in the shape of containers with lots of versatility and opportunity. We need this tool to deploy our builds on machines and luckily, it is greatly integrated into the Azure DevOps platform. To fully utilize its many capabilities, you will need to register on the DockerHub. Step 1: After registration, create a repository with the name of your application. Then choose whether or not to make it public (you can only have one private repository with the free plan). Step 2: Next, we need to authorize DockerHub into our Azure DevOps project. To do this go back to Azure DevOps and click on 'Project Settings' in the bottom-left corner. Step 3: Choose "Pipelines/Service Connections*". Step 4: Now click on the top-right button "New service connection" and search for Docker registry. Mark it, and hit next. Step 5: Choose "Docker Hub" as the registry type. Step 6: Fill in the remaining fields (the service connection name is up to you). You should now be able to see your entry below "Service Connections." The connection will make itself relevant later, but for now, we need to go back to the project and add a few things. Since we added the azure.pipelines.yml file to the repository, a git pull needs to be called to pull the newest changes. Furthermore, we need to define our Docker image using a Dockerfile. Step 7: Create a new file in the root of the project and name it "Dockerfile." Your project tree should now look something like this: Project Tree with addition of Dockerfile The Dockerfile should be considered a template for containers much like classes are for objects. What needs to be defined in this template is as follows: We need to set a basis for the virtual environment (FROM openjdk:8). We need to copy our .jar file onto the virtual environment (COPY /target/testing-helloworld-?.?*.jar .). We need to run the .jar file upon initialization (CMD java -jar testing-helloworld-?-?*.jar). You should now have a file looking similar to this: Dockerfile x 1 FROM openjdk:8 2 COPY /target/testing-helloworld-?.?*.jar . 3 CMD java -jar testing-helloworld-?.?*.jar The regex simply accounts for different versions being deployed, but the actual name has to match the .jar file from the target folder. Update the azure.pipelines.yml File To sum up our current progress, we have now made a Maven project, linked it to a pipeline, and created a template for the virtual environment. The only thing missing is to connect everything via the azure.pipelines.yml file. Step 1: Add Variables We will need to add some variables for the DockerHub connection, as well as the ever-changing version number to the azure.pipelines.yml file (insert your Service Connection and Docker repository): HXML xxxxxxxxxx 1 1 ... 2 variables: 3 containerRegistryServiceConnection: saban17-testing 4 imageRepository: saban17/testing-helloworld 5 tag: 1.0.0 6 ... These variables are not strictly necessary, but it never hurts to follow the DRY principle. Step 2: Add Tasks to Pipeline Steps Next, we need to add more tasks to our pipeline steps. What needs to happen is to log in to Docker, build the Dockerfile previously defined, and push the image to our DockerHub repository. One at a time, we add the wanted behavior starting with the Docker login: HXML xxxxxxxxxx 1 1 - task: Docker@2 2 displayName: dockerLogin 3 inputs: 4 command: login 5 containerRegistry: $(containerRegistryServiceConnection) Then the Docker build: HXML xxxxxxxxxx 1 1 - task: Docker@2 2 displayName: dockerBuild 3 inputs: 4 repository: $(imageRepository) 5 command: build 6 Dockerfile: Dockerfile 7 tags: | 8 $(tag) And lastly, the Docker push: HXML xxxxxxxxxx 1 1 - task: Docker@2 2 displayName: dockerPush 3 inputs: 4 command: push 5 containerRegistry: $(containerRegistryServiceConnection) 6 repository: $(imageRepository) 7 tags: | 8 $(tag) You should now have an azure.pipelines.yml file looking similar to this (with the addition of mavenAuthenticateFeed:true in Maven@3 inputs): HXML xxxxxxxxxx 1 48 1 trigger: 2 - master 3 4 pool: 5 vmImage: 'ubuntu-latest' 6 7 variables: 8 containerRegistryServiceConnection: saban17-testing 9 imageRepository: saban17/testing-helloworld 10 tag: 1.0.0 11 12 steps: 13 - task: Maven@3 14 inputs: 15 mavenPomFile: 'pom.xml' 16 mavenOptions: '-Xmx3072m' 17 javaHomeOption: 'JDKVersion' 18 jdkVersionOption: '1.8' 19 jdkArchitectureOption: 'x64' 20 publishJUnitResults: true 21 mavenAuthenticateFeed: true 22 testResultsFiles: '**/surefire-reports/TEST-*.xml' 23 goals: 'package' 24 25 - task: Docker@2 26 displayName: dockerLogin 27 inputs: 28 command: login 29 containerRegistry: $(containerRegistryServiceConnection) 30 31 - task: Docker@2 32 displayName: dockerBuild 33 inputs: 34 repository: $(imageRepository) 35 command: build 36 Dockerfile: Dockerfile 37 tags: | 38 $(tag) 39 40 - task: Docker@2 41 displayName: dockerPush 42 inputs: 43 command: push 44 containerRegistry: $(containerRegistryServiceConnection) 45 repository: $(imageRepository) 46 tags: | 47 $(tag) 48 Understandingly, this might be a little overwhelming; but fear not, it looks more complicated than it really is. For more information about these inputs see Docker task. Push to the Pipeline Finally, now we get to see the magic happen. However, before doing so, I need to tell you the routine procedure that is to push to the pipeline: Step 1: Go into the pom.xml and the azure.pipelines.yml files, and increment the version number. Step 2: Run the Maven lifecycle clean to remove earlier .jar files in the target folder. Step 3: Run the Maven lifecycle package to build and package your code (creating the new .jar file). Step 3: Provided you are on the master branch, run the Git commands: git add . git commit -m "commit message" git push Step 4: Check whether or not the job passes in the pipeline. If everything went as it should, you have now uploaded an image with your .jar file to the associated Docker Hub repository. Step 5: Running this image now only requires the host to have Docker installed. Let us try it! Running Docker Hub repository The input (a) initiates a container from the requested repository. The image was then retrieved, instantiated, and processed with the final result (b) displaying "Hello World!" This concludes the guide for setting up your Java Pipeline with Azure DevOps and Docker. Conclusion By now, it should hopefully be clear why this approach has its benefits. It enables the developer to form a run-time environment (Dockerfile) and upload it to operation with simple to no effort (git push). While it has not been covered, this approach also creates artifacts in Azure DevOps, which is very useful when using something like Maven, as it makes dependencies surprisingly easy to manage. Since this approach only recently made it into our team, it is still under development and a lot of additions are still to be made. I highly encourage you to further expand upon your pipeline by making it fit your exact needs. I hope this guide has proven to be useful as well as practical, and should you have any further questions, feel free to comment.

By Samuel Bangslund

Continuous Integration and Continuous Delivery for Database Changes

DevOps proposes Continuous Integration and Continuous Delivery (CI/CD) solutions for software project management. In CI/CD, the process of software development and operations falls into a cyclical feedback loop, which promotes not only innovation and improvements but also makes the product quickly adapt to the changing needs of the market. So, it becomes easy to cater to the needs of the customer and garner their satisfaction. The development team that adopts the culture becomes agile, flexible, and adaptive (called the Agile development team) in building an incremental quality product that focuses on continuous improvement and innovation. One of the key areas of CI/CD is to address changes. The evolution of software also has its effect on the database as well. Database change management primarily focuses on this aspect and can be a real hurdle in collaboration with DevOps practices which is advocating automation for CI/CD pipelines. Automating database change management enables the development team to stay agile by keeping database schema up to date as part of the delivery and deployment process. It helps to keep track of changes critical for debugging production problems. The purpose of this article is to highlight how database change management is an important part of implementing Continuous Delivery and recommends some processes that help streamline application code and database changes into a single delivery pipeline. Continuous Integration One of the core principles of an Agile development process is Continuous Integration. Continuous Integration emphasizes making sure that code developed by multiple members of the team is always integrated. It avoids the “integration hell” that used to be so common during the days when developers worked in their silos and waited until everyone was done with their pieces of work before attempting to integrate them. Continuous Integration involves independent build machines, automated builds, and automated tests. It promotes test-driven development and the practice of making frequent atomic commits to the baseline or master branch or trunk of the version control system. Figure 1: A typical Continuous Integration process The diagram above illustrates a typical Continuous Integration process. As soon as a developer checks in code to the source control system, it will trigger a build job configured in the Continuous Integration (CI) server. The CI Job will check out code from the version control system, execute a build, run a suite of tests, and deploy the generated artifacts (e.g., a JAR file) to an artifact repository. There may be timed CI jobs to deploy the code to the development environment, push details out to a static analysis tool, run system tests on the deployed code, or any automated process that the team feels is useful to ensure that the health of the codebase is always maintained. It is the responsibility of the Agile team to make sure that if there is any failure in any of the above-mentioned automated processes, it is promptly addressed and no further commits are made to the codebase until the automated build is fixed. Continuous Delivery Continuous Delivery takes the concept of Continuous Integration a couple of steps further. In addition to making sure that different modules of a software system are always integrated, it also makes sure that the code is always deployable (to production). This means that in addition to having an automated build and a completely automated test suite, there should be an automated process of delivering the software to production. Using the automated process, it should be possible to deploy software on short notice, typically within minutes, with the click of a button. Continuous Delivery is one of the core principles of DevOps and offers many benefits including predictable deploys, reduced risk while introducing new features, shorter feedback cycles with the customer, and overall higher quality of software. Figure 2: A typical Continuous Delivery process The above diagram shows a typical Continuous Delivery process. Please note that the above-illustrated Continuous Delivery process assumes that a Continuous Integration process is already in place. The above diagram shows 2 environments: e.g., User Acceptance Test (UAT) and production. However, different organizations may have multiple staging environments (Quality Assurance or QA, load testing, pre-production, etc.) before the software makes it to the production environment. However, it is the same codebase, and more precisely, the same version of the codebase that gets deployed to different environments. Deployment to all staging environments and the production environment are performed through the same automated process. There are many tools available to manage configurations (as code) and make sure that deploys are automatic (usually self-service), controlled, repeatable, reliable, auditable, and reversible (can be rolled back). It is beyond the scope of this article to go over those DevOps tools, but the point here is to stress the fact that there must be an automated process to release software to production on demand. Database Change Management Is the Bottleneck Agile practices are pretty much mainstream nowadays when it comes to developing application code. However, we don’t see as much adoption of agile principles and continuous integration in the area of database development. Almost all enterprise applications have a database involved and thus project deliverables would involve some database-related work in addition to application code development. Therefore, slowness in the process of delivering database-related work - for example, a schema change - slows down the delivery of an entire release. In this article, we would assume the database to be a relational database management system. The processes would be very different if the database involved is a non-relational database like a columnar database, document database, or a database storing data in key-value pairs or graphs. Let me illustrate this scenario with a real example: here is this team that practices Agile software development methodologies. They follow a particular type of Agile called Scrum, and they have a 2-week Sprint. One of the stories in the current sprint is the inclusion of a new field in the document that they interchange with a downstream system. The development team estimated that the story is worth only 1 point when it comes to the development of the code. It only involves minor changes in the data access layer to save the additional field and retrieve it later when a business event occurs and causes the application to send out a document to a downstream system. However, it requires the addition of a new column to an existing table. Had there been no database changes involved, the story could have been easily completed in the current sprint, but since there is a database change involved, the development team doesn’t think it is doable in this sprint. Why? Because a schema change request needs to be sent to the Database Administrators (DBA). The DBAs will take some time to prioritize this change request and rank this against other change requests that they received from other application development teams. Once the DBAs make the changes in the development database, they will let the developers know and wait for their feedback before they promote the changes to the QA environment and other staging environments, if applicable. Developers will test changes in their code against the new schema. Finally, the development team will closely coordinate with the DBAs while scheduling delivery of application changes and database changes to production. Figure 3: Manual or semi-automated process in delivering database changes Please note in the diagram above that the process is not triggered by a developer checking in code and constitutes a handoff between two teams. Even if the deployment process on the database side is automated, it is not integrated with the delivery pipeline of application code. The changes in the application code are directly dependent on the database changes, and they together constitute a release that delivers a useful feature to the customer. Without one change, the other change is not only useless but could potentially cause regression. However, the lifecycle of both of these changes is completely independent of each other. The fact that the database and codebase changes follow independent life cycles and the fact that there are handoffs and manual checkpoints involved, the Continuous Delivery process, in this example, is broken. Recommendations To Fix CI/CD for Database Changes In the following sections, we will explain how this can be fixed and how database-related work including data modeling and schema changes, etc., can be brought under the ambit of the Continuous Delivery process. DBAs Should Be a Part of the Cross-Functional Agile Team Many organizations have their DBAs split into broadly two different types of roles based on whether they help to build a database for application development teams or maintain production databases. The primary responsibility of a production DBA is to ensure the availability of production databases. They monitor the database, take care of upgrades and patches, allocate storage, perform backup and recovery, etc. A development DBA, on the other hand, works closely with the application development team and helps them come up with data model design, converts a logical data model into a physical database schema, estimates storage requirements, etc. To bring database work and application development work into one single delivery pipeline, it is almost necessary that the development DBA be a part of the development team. Full-stack developers in the development team with good knowledge of the database may also wear the hat of a development DBA. Database as Code It is not feasible to have database changes and application code integrated into a single delivery pipeline unless database changes are treated the same way as application code. This necessitates scripting every change in the database and having them version-controlled. It should then be possible to stand up a new instance of the database automatically from the scripts on demand. If we had to capture database objects as code, we would first need to classify them and evaluate each one of those types to see if and how they need to be captured as script (code). Following is a broad classification of them: Database Structure This is basically the definition of how stored data will be structured in the database and is also known as a schema. These include table definitions, views, constraints, indexes, and types. The data dictionary may also be considered as a part of the database structure. Stored Code These are very similar to application code, except that they are stored in the database and are executed by the database engine. They include stored procedures, functions, packages, triggers, etc. Reference Data These are usually a set of permissible values that are referenced from other tables that store business data. Ideally, tables representing reference data have very few records and don’t change much over the life cycle of an application. They may change when some business process changes but don’t usually change during the normal course of business. Application Data or Business Data These are the data that the application records during the normal course of business. The main purpose of any database system is to store these data. The other three types of database objects exist only to support these data. Out of the above four types of database objects, the first three can be and should be captured as scripts and stored in a version control system. Type Example Scripted (Stored like code?) Database Structure Schema Objects like Tables, Views, Constraints, Indexes, etc. Yes Stored Code Triggers, Procedures, Functions, Packages, etc. Yes Reference Data Codes, Lookup Tables, Static data, etc. Yes Business/Application Data Data generated from day-to-day business operations No Table 1: Depicts what types of database objects can be scripted and what types can’t be scripted As shown in the table above, business data or application data are the only types that won’t be scripted or stored as code. All rollbacks, revisions, archival, etc., are handled by the database itself; however, there is one exception. When a schema change forces data migration - say, for example, populating a new column or moving data from a base table to a normalized table - that migration script should be treated as code and should follow the same life cycle as the schema change. Let's take an example of a very simple data model to illustrate how scripts may be stored as code. This model is so simple and so often used in examples, that it may be considered the “Hello, World!” of data modeling. Figure 4: Example model with tables containing business data and ones containing reference data In the model above, a customer may be associated with zero or more addresses, like billing address, shipping address, etc. The table AddressType stores the different types of addresses like billing, shipping, residential, work, etc. The data stored in AddressType can be considered reference data as they are not supposed to grow during day-to-day business operations. On the other hand, the other tables contain business data. As the business finds more and more customers, the other tables will continue to grow. Example Scripts: Tables: Constraints: Reference Data: We won’t get into any more details and cover each type of database object. The purpose of the examples is to illustrate that all database objects, except for business data, can be and should be captured in SQL scripts. Version Control Database Artifacts in the Same Repository as Application Code Keeping the database artifacts in the same repository of the version control system as the application code offers a lot of advantages. They can be tagged and released together since, in most cases, a change in database schema also involves a change in application code, and they together constitute a release. Having them together also reduces the possibility of application code and the database getting out of sync. Another advantage is just plain readability. It is easier for a new team member to come up to speed if everything related to a project is in a single place. Figure 5: Example structure of a Java Maven project containing database code The above screenshot shows how database scripts can be stored alongside application code. Our example is a Java application, structured as a Maven project. The concept is however agnostic of what technology is used to build the application. Even if it was a Ruby or a .NET application, we would store the database objects as scripts alongside application code to let CI/CD automation tools find them in one place and perform necessary operations on them like building the schema from scratch or generating migration scripts for a production deployment. Integrate Database Artifacts Into the Build Scripts It is important to include database scripts in the build process to ensure that database changes go hand in hand with application code in the same delivery pipeline. Database artifacts are usually SQL scripts of some form and all major build tools support executing SQL scripts either natively or via plugins. We won’t get into any specific build technology but will list down the tasks that the build would need to perform. Here we are talking about builds in local environments or CI servers. We will talk about builds in staging environments and production at a later stage. The typical tasks involved are: Drop Schema Create Schema Create Database Structure (or schema objects): They include tables, constraints, indexes, sequences, and synonyms. Deploy stored code, like procedures, functions, packages, etc. Load reference data Load TEST data If the build tool in question supports build phases, this will typically be in the phase before integration tests. This ensures that the database will be in a stable state with a known set of data loaded. There should be sufficient integration tests that will cause the build to fail if the application code goes out of sync with the data model. This ensures that the database is always integrated with the application code: the first step in achieving a Continuous Delivery model involving database change management. Figure 6: Screenshot of code snippet showing a Maven build for running database scripts The above screenshot illustrates the usage of a Maven plugin to run SQL scripts. It drops the schema, recreates it, and runs all the DDL scripts to create tables, constraints, indexes, sequences, and synonyms. Then it deploys all the stored code into the database and finally loads all reference data and test data. Refactor Data Model as Needed Agile methodology encourages evolutionary design over upfront design; however, many organizations that claim to be Agile shops, actually perform an upfront design when it comes to data modeling. There is a perception that schema changes are difficult to implement later in the game, and thus it is important to get it right the first time. If the recommendations made in the previous sections are made, like having an integrated team with developers and DBAs, scripting database changes, and version controlling them alongside application code, it won’t be difficult to automate all schema changes. Once the deployment and rollback of database changes are fully automated and there is a suite of automated tests in place, it should be easy to mitigate risks in refactoring schema. Avoid Shared Database Having a database schema shared by more than one application is a bad idea, but they still exist. There is even a mention of a “Shared Database” as an integration pattern in a famous book on enterprise integration patterns, Enterprise Integration Patterns by Gregor Holpe and Bobby Woolf. Any effort to bring application code and database changes under the same delivery pipeline won’t work unless the database truly belongs to the application and is not shared by other applications. However, this is not the only reason why a shared database should be avoided. "Shared Database" also causes tight coupling between applications and a multitude of other problems. Dedicated Schema for Every Committer and CI Server Developers should be able to work on their own sandboxes without the fear of breaking anything in a common environment like the development database instance; similarly, there should be a dedicated sandbox for the CI server as well. This follows the pattern of how application code is developed. A developer makes changes and runs the build locally, and if the build succeeds and all the tests pass, (s)he commits the changes. The sandbox could be either an independent database instance, typically installed locally on the developer’s machine, or it could be a different schema in a shared database instance. Figure 7: Developers make changes in their local environment and commit frequently As shown in the above diagram, each developer has their own copy of the schema. When a full build is performed, in addition to building the application, it also builds the database schema from scratch. It drops the schema, recreates it, and executes DDL scripts to load all schema objects like tables, views, sequences, constraints, and indexes. It creates objects representing stored code, like functions, procedures, packages, and triggers. Finally, it loads all the reference data and test data. Automated tests ensure that the application code and database object are always in sync. It must be noted that data model changes are less frequent than application code, so the build script should have the option to skip the database build for the sake of build performance. The CI build job should also be set up to have its own sandbox of the database. The build script performs a full build that includes building the application as well as building the database schema from scratch. It runs a suite of automated tests to ensure that the application itself and the database that it interacts with, are in sync. Figure 8: Revised CI process with integration of database build with build of application code Please note that the similarity of the process described in the above diagram with the one described in Figure 1. The build machine or the CI server contains a build job that is triggered by any commit to the repository. The build that it performs includes both the application build and the database build. The database scripts are now always integrated, just like application code. Dealing With Migrations The process described above would build the database schema objects, stored code, reference data, and test data from scratch. This is all good for continuous integration and local environments. This process won’t work for the production database and even QA or UAT environments. The real purpose of any database is storing business data, and every other database object exists only to support business data. Dropping schema and recreating it from scripts is not an option for a database currently running business transactions. In this case, there is a need for scripting deltas, i.e., the changes that will transition the database structure from a known state (a particular release of software) to a desired state. The transition will also include any data migration. Schema changes may lead to a requirement to migrate data as well. For example, as a result of normalization, data from one table may need to be migrated to one or more child tables. In such cases, a script that transforms data from the parent table to the children should also be a part of the migration scripts. Schema changes may be scripted and maintained in the source code repository so that they are part of the build. These scripts may be hand-coded during active development, but there are tools available to automate that process as well. One such tool is Flyway, which can generate migration scripts for the transition of one state of schema to another state. Figure 9: Automation of schema migrations and rollback In the above picture, the left-hand side shows the current state of the database which is in sync with the application release 1.0.1 (the previous release). The right-hand side shows the desired state of the database in the next release. We have the state on the left-hand side captured and tagged in the version control system. The right-hand side is also captured in the version control system as the baseline, master branch, or trunk. The difference between the right-hand side and the left-hand side is what needs to be applied to the database in the staging environments and the production environment. The differences may be manually tracked and scripted, which is laborious and error-prone. The above diagram illustrates that tools like Flyway can automate the creation of such differences in the form of migration scripts. The automated process will create the following: Migration script (to transition the database from the prior release to the new release) Rollback script (to transition the database back to the previous release). The generated scripts will be tagged and stored with other deploy artifacts. This automation process may be integrated with the Continuous Delivery process to ensure repeatable, reliable, and reversible (ability to rollback) database changes. Continuous Delivery With Database Changes Incorporated Into It Let us now put the pieces together. There is a Continuous Integration process already in place that rebuilds the database along with the application code. We have a process in place that generates migration scripts for the database. These generated migration scripts are a part of the deployment artifacts. The DevOps tools will use these released artifacts to build any of the staging environments or the production environment. The deployment artifacts will also contain rollback scripts to support self-service rollback. If anything goes wrong, the previous version of the application may then be redeployed and the database rollback script shall be run to transition the database schema to the previous state that is in sync with the previous release of the application code. Figure 10: Continuous Delivery incorporating database changes The above diagram depicts a Continuous Delivery process that has database change management incorporated into it. This assumes that a Continuous Integration process is already there in place. When a UAT (or any other staging environment like TEST, QA, etc.) deployment is initiated, the automated processes take care of creating a tag in the source control repository, building application deployable artifacts from the tagged codebase, generating database migration scripts, assembling the artifacts and deploying. The deployment process includes the deployment of the application as well as applying migration scripts to the database. The same artifacts will be used to deploy the application to the production environment, following the approval process. A rollback would involve redeploying the previous release of the application and running the database rollback script. Tools Available in the Market The previous sections primarily describe how to achieve CI/CD in a project that involves database changes by following some processes but don’t particularly take into consideration any tools that help in achieving them. The above recommendations are independent of any particular tool. A homegrown solution can be developed using common automation tools like Maven or Gradle for build automation, Jenkins or TravisCI for Continuous Integration, and Chef or Puppet for configuration management; however, there are many tools available in the marketplace, that specifically deal with Database DevOps. Those tools may also be taken advantage of. Some examples are: Datical Redgate Liquibase Flyway Conclusion Continuous Integration and Continuous Delivery processes offer tremendous benefits to organizations, like accelerated time to market, reliable releases, and overall higher-quality software. Database change management is traditionally cautious and slow. In many cases, database changes involve manual processes and often cause a bottleneck to the Continuous Delivery process. The processes and best practices mentioned in this article, along with available tools in the market, should hopefully eliminate this bottleneck and help to bring database changes into the same delivery pipeline as application code.

By Ashoke Bhowmick

DevOps and CI/CD

DZone's Featured DevOps and CI/CD Resources

Top DevOps and CI/CD Experts

The Latest DevOps and CI/CD Topics