Maven – profile & troubleshoot build failed of multiple modules

1. Maven Profile

This blog mainly used to organize and record some usage experiences of Maven multiple repositories. No obscure knowledge of principles. Firstly, summarize the role of Maven.

• Maven used to manage the structure of project itself, specially for multiple modules project.
• Maven used to download and manage the dependent thirdparty jar file for a project.
• Maven used to upload jar file of a project to Maven repository.
• Maven Repository used to store and manage the uploaded jar file.

If your project(service) is a multiple modules project or a common basic service, you always use multiple repositories management of Maven. It is used for software development, testing and release.

1.1. Multiple repositories architecture

Maven repository can be classified as local repository and remote repository, moreover, remote repository can be also classified as private remote repository, central remote repository and other public remote repository. Look below:

• Local repository : used to store the jar file built in local environment, e.g., a jar file will be stored into local reposiltory after you execute maven command, mvn install or mvn deploy.
• Private remote repository : used to store private jar file of company level, usually, the private repository is in LAN.
• Default central remote repository : it is a default repository of Maven.
• Other public remote repository : used to store thirdparth jar file supplied by other organization or company.

The architecture or relationship of above multiple repositories as following:

Next, I will use a example of multiple modules project to illustrate the usage of multiple repositories, espacially private remote repository.

1.2. Connect to private remote repositories

There is a manager called Nexus for managing various remote repositories. E.g., there are two private repositories, snapshot repository and release repository, one used for storing jar files in stable version, the other used for storing jar files in unstable version.

Then you can configure username, password and url of above repositories in Maven config file – settings.xml. Maven can be responsible for connecting to the private remote repositories by the username, password and repository url. Next you can upload jar files to or download from these repositories within your local project. Below is a Maven config snippet from settings.xml:

<settings> 
  <servers> 
    <server> 
      <id>easyblog-snapshots</id>  
      <username>abei</username>  
      <password>passwordsnapshots</password> 
    </server>  
    <server> 
      <id>easyblog-releases</id>  
      <username>abei</username>  
      <password>passwordreleases</password> 
    </server> 
  </servers>  

  <profiles> 
    <profile> 
      <id>easeblog</id>  
      <repositories> 
        <repository> 
          <id>easyblog-snapshots</id>  
          <releases> 
            <enabled>false</enabled> 
          </releases>  
          <snapshots> 
            <enabled>true</enabled>  
            <updatePolicy>always</updatePolicy> 
          </snapshots>  
          <url>http://easyblog.com/nexus/content/repositories/snapshots</url> 
        </repository>  
        <repository> 
          <id>easyblog-releases</id>  
          <releases> 
            <enabled>true</enabled>  
            <updatePolicy>daily</updatePolicy> 
          </releases>  
          <snapshots> 
            <enabled>false</enabled> 
          </snapshots>  
          <url>http://easyblog.com/nexus/content/repositories/releases</url> 
        </repository> 
      </repositories> 
    </profile> 
  </profiles>  

  <activeProfiles> 
    <activeProfile>easeblog</activeProfile> 
  </activeProfiles>  
  <mirrors> 
    <mirror> 
      <id>local-china</id>  
      <mirrorOf>central</mirrorOf>  
      <url>http://maven.china.com/nexus/content/groups/public/</url> 
    </mirror>  
    <mirror> 
      <id>local-home</id>  
      <mirrorOf>easeblog</mirrorOf>  
      <url>http://maven.home.com/nexus/content/groups/public/</url> 
    </mirror> 
  </mirrors> 
</settings>

NOTE : If you enabled the <mirror> configuration, the <mirror>.<url> will replace repository url defined in settings.xml or pom.xml, E.g., <mirror>.<url> can replace <repositories>.<repository>.<url> in settings.xml with same <profile>.<id> or <repository>.<id>; and <mirror>.<url> can also replace <repositories>.<repository>.<url> in pom.xml with same <repository>.<id>.

2. Upload Jar files

There is a multiple modules project composed by following submodels:

search
|- search-order
|- search-pay
|- search-util

For submodules, search-order and search-pay, both of them dependend on search-util. The search-util submodule need upload to private remote repository, actually, there are two different versions for search-util, one is snapshot version with unstable code that will be uploaded into snapshot repository used to development and testing for other developers to download, the other is release version with stable code that will be uploaded into release repository used to final release(or go live).

2.1. Upload Snapshot Jar

Firstly, you need configure in pom.xml of search module, distributionManagement of Maven will manage your publish operation, as follows:

<distributionManagement> 
  <snapshotRepository> 
    <id>easyblog-snapshots</id>  
    <url>http://easyblog.com/nexus/content/repositories/snapshots</url> 
  </snapshotRepository> 
</distributionManagement>

NOTE : To use the configured username and password in settings.xml of Maven for snapshot deploying, <distributionManagement>.<snapshotRepository>.<id> from search module pom.xml must equal to <servers>.<server>.<id> configured in settings.xml for snapshot deploying.

Secondly, you should make sure the version of the submodule(search-util) you want to upload with suffix “SNAPSHOT”, case sensitive. E.g., search-util-1.0-SNAPSHOT.jar.

Finally, enter into the directory of pom file of parent module(search module), perform Maven command, “mvn clean deploy”. Here, one point we should pay attention is after command executed, search-order and search-pay also are uploaded into private remote repository, this is because when you execute deploy command in parent module, all its submodules will be deployed too. If you want avoid this situation, you should configure <properties>.<maven.deploy.skip>true</maven.deploy.skip>.</properties> into the submodule’s pom file that you don’t want it deployed. Here, the configuration snippet shoud be configured into pom.xml of both search-pay submodule and search-order submodule to avoid them deployed.

After deployment command is executed, the snapshot version of search-util-1.0-SNAPSHOT.jar will be also deployed into local repository.

NOTE : In addition to the above upload(deploy) ways, there are two other ways, one is by Nexus Manager UI, the other is by Maven command:mvn deploy:deploy-file -DgroupId=xxx -DartifactId=yyy -Dversion=1.0-SNAPSHOT -Dpacckaging=jar -Dfile=zzz-1.0-SNAPSHOT.jar -DrepositoryId=maven-snapshots -Durl=http://easyblog.com/nexus/content/repositories/snapshots

The flow of uploading(deploying) jar files ss shown in the figure below:

NOTE : <distributionManagement>.<snapshotRepository>.<url> is necessary to specify private remote repository url for uploading.

2.2. Upload Release Jar

The flow of uploading release jar files is almost same as uploading snapshot jar files. Firstly, configure search.pom.xml as follows:

<distributionManagement> 
  <repository> 
    <id>easyblog-releases</id>  
    <url>http://easyblog.com/nexus/content/repositories/releases</url> 
  </repository> 
</distributionManagement>

NOTE : To use the configured username and password in settings.xml of Maven for release deploying, the <distributionManagement>.<repository>.<id> from search module pom.xml must equal to <servers>.<server>.<id> configured in settings.xml for release deploying.

Secondly, you should make sure the version of the submodule(search-util) you want upload without suffix “SNAPSHOT” E.g., search-util-1.0.jar.

Finally, there is a little change in the deploying command, now it’s “mvn deploy -P release”. Talk with pictures:

NOTE: if the <distributionManagement>.<repository>.<url> is missing, you cannot deploy release jar files with command “mvn deploy -P release”, but you can still specify repository url in command line to upload.

2.3. Debugging Maven command

command	remarks
mvn -Dmaven.test.skip=true clean deploy -X -l build.log	Print log detail into build.log, build.log will be the same directory with pom.xml of a submodule.
mvn -Dmaven.test.skip=true clean deploy -X	Print log detail to console.

3. Download Jar files

3.1. Download Snapshot Jar

Firstly, configure url of private remote repository in search module pom.xml.

<repositories> 
  <repository> 
    <id>easyblog-snapshots</id>  
    <releases> 
      <enabled>false</enabled> 
    </releases>  
    <snapshots> 
      <enabled>true</enabled>  
      <updatePolicy>always</updatePolicy> 
    </snapshots>  
    <url>http://easyblog.com/nexus/content/repositories/snapshots</url> 
  </repository> 
</repositories>

NOTE : The <repositories>.<repository>.<id> must also equal to the <servers>.<server>.<id> in settings.xml of Maven to use the configured username and password for downloading snapshot jar files from private remote repository, if not, “Not authorized” error will be thrown.

Another thing to be clear is that if you configured repository url in profiles of Maven settings.xml, the url will be used first to download jar files, unless it fails the repository url configured in pom.xml will be used. The configuration snippet as below:

<profiles>
  <profile>
    <id>easeblog</id>
    <repositories>
      <repository>
        <id>easyblog-snapshots</id>
        <releases>
          <enabled>false</enabled>
        </releases>
        <snapshots>
          <enabled>true</enabled>
          <updatePolicy>always</updatePolicy>
        </snapshots>
        <url>http://easyblog.com/nexus/content/repositories/snapshots</url>
      </repository>
    </repositories>
  </profile>
</profiles>

Further says, i.e, first, Maven use <profiles>.<profile>.<repositories>.<repository>.<url> with <repository>.<id> “easyblog-snapshots” from settings.xml to download a jar file(search-util-1.0-SNAPSHOT.jar), only when it fails, then will use <repositories>.<repository>.<url> with <repository>.<id> “easyblog-snapshots” from pom.xml to download it again. Of cause, if first time downloading success, the downloading operation for the jar file is over. If <profile> is not configured or disabled in settings.xml, Maven would use repository url configured in pom.xml.

Secondly, Maven will download a jar file with snapshot version(search-util-1.0-SNAPSHOT.jar.) in a specified frequency configured in element <updatePolicy>.

NOTE : Default update policy is “daily”, you can see more about update policy(link to https://maven.apache.org/ref/3.6.3/maven-settings/settings.html)

Thirdly, if no search-util-1.0-SNAPSHOT.jar in local repository, the search-util-1.0-SNAPSHOT.jar will be downloaded normally from snapshot repository, and stored into local repository used for subsequent Maven lifecycle. However, if there is already a search-util-1.0-SNAPSHOT.jar in local repository, for snapshot version jar, Maven will still download it, and keep the latest version of these two jar files by comparing the <lastUpdated> of maven-metadata-local.xml in local repository to the <lastUpdated> of maven-metadata-easyblog-snapshots.xml in remote repository even if the two jar files have the same name.

Finally, your projet will go into the remaining lifecycle of Maven, e.g., compile, package, install or test etc. Here, I attach a sequence diagram to illustrate the flow of the downloading.

3.2. Download Release Jar

Firstly, still is the key configuration of remote repository url. Compared to downloading snapshot jar file, downloading release jar is relative easy. Below is private remote repository url configured in search module pom.xml.

<repositories> 
  <repository> 
    <id>release</id>  
    <url>http://repo.shanyishanmei.com/nexus/content/repositories/renrenche-business-releases</url>  
    <releases> 
      <enabled>true</enabled> 
    </releases>  
    <snapshots> 
      <enabled>false</enabled> 
    </snapshots> 
  </repository> 
</repositories>

As preceding chapter, below is global url configuration of private remote repository.

<profiles> 
  <profile> 
    <id>easeblog</id>  
    <repositories> 
      <repository> 
        <id>easyblog-releases</id>  
        <releases> 
          <enabled>true</enabled>  
          <updatePolicy>daily</updatePolicy> 
        </releases>  
        <snapshots> 
          <enabled>false</enabled> 
        </snapshots>  
        <url>http://easyblog.com/nexus/content/repositories/releases</url> 
      </repository> 
    </repositories> 
  </profile> 
</profiles>

Secondly, if search-util-1.0.jar needed by search-pay submodule is missing from local repository, Maven will download it from private remote repository by configured “updatePolicy”, otherwise, search-pay submodule just import it from local repository, no need downloading.

Finally, carry on next lifecycle of Maven.

3.3. Flow Char of Downloading

4. Troubleshoot dependent submodule is missing

In developing environment, the project is built and deployed by Jenkins, and during the development phase, submodules refer to each other by SNAPSHOT version. Still follow above multiple modules project as example, search-pay depends on search-util.

As a hypothetical function, a new class DateUtil to be used by search-pay created in search-util submodule, and no version updated(still is search-util-1.0-SNAPSHOT.jar). But an error of DateUtil.class is missing catched into building log when building(mvn install) search-pay, then I begin to troubleshoot this issue.

Firstly, I guess the cause is that the building script of Jenkins not get into the directory of parent pom but submodule pom directory before invoking mvn install, lead to Maven not “Reactor Build Order”. I.e., the wrong building order will lead search-pay to refer the changing in search-util not yet installed into local repository. But after investigating the script, this does not happen, the script just enter the parent pom directory, then execute mvn install.

Secondly, I guess the cache of local repository is the culprit. Maybe, the update policy has been configured “never”, but after checking the pom.xml and settings.xml, abandoned this guessing.

From my opinion, except above two cases, the issue should be still in Jenkins script. The process of building search project by Jenkins is : git clone -> docker run auto-build -> script run mvn install -> docker build -> docker push -> k8s deploy, so the problem is most likely happening in “script run mvn install”. Finally, the troubleshooting result is that there is a syntax error in the Jenkins script.