Friday, November 4, 2022

Git submodules — Merge the content of multiple branches across git repositories at build time.

In some cases, you may have a requirement to deploy the changes to a platform through a single repository/branch, e.g., AEM Cloud manager; AEM cloud manager supports a single repository and a branch to deploy the code changes to a specific environment — the containers are rebuilt on every deployment, so the complete code needs to be deployed on every deployment.

This can create restrictions while multiple teams working on different projects that all need to be deployed into the same platform; one approach is keeping all the project codes in a single repository, and all the teams can work on the same code repository, but sometimes this can lead into conflicts and dependencies between projects team.

Git submodules:

Git submodules can help to handle this scenario; Git submodules allow to merge of the content of multiple branches across repositories at build time. Git submodules help maintain complex projects by allowing independent repositories to exist as subdirectories within a wider project repository.

I have two maven projects in two different repositories; both repositories are enabled with the main branch

https://github.com/techforum-repo/project1.git
https://github.com/techforum-repo/project2.git

I have the main repository — https://github.com/techforum-repo/main-repository.git with the main branch

The below pom.xml enabled in the main-repository

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="
https://maven.apache.org/POM/4.0.0" xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
https://maven.apache.org/POM/4.0.0 https://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>albin</groupId>
<artifactId>main-repository</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>pom</packaging>
<modules>
<module>project1</module>
<module>project2</module>
</modules>
</project>

Now add the submodule to the main repository by executing the below command on the main-repository folder

git submodule add -b main https://github.com/techforum-repo/project1.git project1
git submodule add -b main
https://github.com/techforum-repo/project2.git project2

This will generate the .gitmodules file with the below content

[submodule "project1"]
path = project1
url =
https://github.com/techforum-repo/project1.git
branch = main
[submodule "project2"]
path = project2
url =
https://github.com/techforum-repo/project2.git
branch = main

The submodules are checked out under the main repository; now you can commit the submodule changes to the main-repository

git add .
git commit -m "Added the submodule to the project."
git push

Now the submodules are committed to the main-repository

To build the project.

Clone the Main Repository - git clone https://github.com/techforum-repo/main-repository.gitExecute - git submodule init && git submodule update

or

git clone --recurse-submodules https://github.com/techforum-repo/main-repository.git

Now build the main project (add the required parameters) — this will build the Sub Modules projects internally

mvn clean installmvn clean install -PautoInstallPackage -Daem.port=4503

Some of the Useful commands to manage Submodules

Update the Submodules in the main repository - git submodule update --remote
Update Specific Project - git submodule update --remote project2
Verify the status of the submodules - git submodule status
Show commit summary - git submodule summary
Run an arbitrary shell command in each checked-out submodule - git submodule foreache.g git submodule foreach 'echo $sm_path `git rev-parse HEAD`'

One of the concerns here is whenever the files from the Submodules (Project1 or Project2) are modified, the auto build through CI/CD pipeline will not be triggered as the build pipeline is configured to the main repository. The main repository will not be aware of the changes on the submodules.

This can be addressed by updating the submodules in the main repository with every change in the submodules (Project1 or Project2)

Execute the below commands on the main repository whenever there is a change on individual submodules, e.g., Project1, Project2 through a repository or external pipeline.git submodule update --remote
git add .
git commit -m "Sub Module Updated."
git push

To enable this in AEM Cloud Manager, create project-specific repositories in the Cloud Manager Program and create a main repository with the required submodules configurations. Enable the main repository to the pipelines so that all the required submodules will be deployed. You make the same repository structure in your source code management tool and enable the pipelines to sync the repository between your code repositories and Cloud Manager Repositories. Refer to https://betterprogramming.pub/a-bitbucket-ci-cd-pipeline-to-sync-branches-with-github-1c885cefe202 for details on syncing repositories.