33

Currently, I'm going to need an implementation that must find all files within a directory and start a parallel task for every file found.

Is it possible to achieve this using declarative pipelines?

pipeline {
    agent any
    stages {
        stage("test") {
            steps {
                dir ("file_path") {
                    // find all files with complete path
                    parallel (
                        // execute parallel tasks for each file found.
                        // this must be dynamic
                        }
                    }
                }
            }
        }
    }
}
Pierre.Vriens
  • 7,205
  • 14
  • 37
  • 84
thclpr
  • 831
  • 1
  • 7
  • 9

5 Answers5

40

Managed to solve it with the following code:

pipeline {
    agent { label "master"}
    stages {
        stage('1') {
            steps {
                script {
                    def tests = [:]
                    for (f in findFiles(glob: '**/html/*.html')) {
                        tests["${f}"] = {
                            node {
                                stage("${f}") {
                                    echo '${f}'
                                }
                            }
                        }
                    }
                    parallel tests
                }
            }
        }       
    }
}
Tensibai
  • 11,366
  • 2
  • 35
  • 62
thclpr
  • 831
  • 1
  • 7
  • 9
  • Please also check out the official Pipeline examples - https://jenkins.io/doc/pipeline/examples/#parallel-multiple-nodes – phedoreanu Apr 10 '18 at 16:15
  • @phedoreanu i'm using declarative pipeline... – thclpr Apr 10 '18 at 16:24
  • @phedoreanu I have rejected your edit, editing code should have good reasons, your comment isn't enough for me to allow an edit of this kind on an answer which was a self solution. I think you should have commented to discuss the matter with the answer author before doing this edit. – Tensibai Apr 16 '18 at 09:38
  • 1
    @phedoreanu I you think you have a better derivative work, then please write your own answer and explain why it is better (in error handling, templates, etc.) instead. – Tensibai Apr 16 '18 at 09:40
  • Hi, I figured out the same after a few failing attempts. My only problem right now is that if I put two stage{..} sections inside a node for some reasons the workflow stage chart and Blu Ocean get confused. For instance in the workflow stage chart I get NaNy NaNd and in Blue Ocean I get only the first stage. – Giuseppe Salvatore Apr 11 '19 at 10:13
  • How would you limit this for, let's say 10 concurrents builds running at once ? – shellwhale Oct 06 '22 at 11:49
14

This also works, if you want to stay within the Declarative Pipeline space

// declare our vars outside the pipeline
def tests = [:]
def files

pipeline {
    agent any
    stages {
        stage('1') {
            steps {
                script {
                    // we've declared the variable, now we give it the values
                    files = findFiles(glob: '**/html/*.html')
                    // Loop through them
                    files.each { f ->
                        // add each object from the 'files' loop to the 'tests' array
                        tests[f] = {
                            // we're already in the script{} block, so do our advanced stuff here
                            echo f.toString()
                        }
                    }
                    // Still within the 'Script' block, run the parallel array object
                    parallel tests
                }
            }
        }       
    }
}
primetheus
  • 181
  • 2
  • 8
  • 1
    If you want to allocate each parallel task to different Jenkins nodes then simply wrap the actions in a node {} block, like this:

    tests[f] = { node { echo f.toString() } }

     – primetheus Jan 29 '19 at 19:57
7

Be aware, that dynamic build steps could cause some problems in some build steps, e.g. when you call an other job:

pipeline {
    stages {
        stage('Test') {
            steps {
                script {
                    def tests = [:]
                    for (f in findFiles(glob: '**/html/*.html')) {
                        // Create temp variable, otherwise the name will be the last value of the for loop
                        def name = f
                        tests["${name}"] = {
                            build job: "${name}"
                        }
                    }
                    parallel tests
                }
            }
        }       
    }
}
Christian Gripp
  • 171
  • 1
  • 1
  • I don't know why but i used stage and echo inside tests, but it will giving stage name and echo as null until I used a local variable like you have used name above. Do you know why that is? Thanks for your answer though! – Kaushik Apr 08 '20 at 10:26
2

It's much easier to use scripted Pipelines to do this since you can use arbitrary Groovy, but you should still be able to do this with declarative Pipelines using the findFiles step.

jayhendren
  • 2,952
  • 7
  • 15
0

Here is a solution where you can setup the maximum number of concurrent tasks running at once with collate(n).

pipeline {
  stages {
    stage("Parallel") {
      steps {
        script {
          def list = ["alpha", "beta", "gamma", "delta", "epsilon", "zeta", "eta", "theta", "iota", "kappa"]

      // Split the list into batches of 4
      def parallelBatches = list.collate(4);
      // [["alpha", "beta", "gamma", "delta"], ["epsilon", "zeta", "eta", "theta"], ["iota", "kappa"]]

      // This allows to run a maximum of 4 tasks in parallel (at once)
      for (parallelBatch in parallelBatches) {
        def tasks = [:]

        for (task in parallelBatch) {
          tasks["${task}"] = {
            stage("${task}") {
              echo '${task}'
            }
          }
        }
        parallel tasks

        // Once the batch is over, the outer loop move onto the next batch
      }
    }
  }
}



}
}
shellwhale
  • 101
  • 1