Class: Rudder::DSL::Pipeline

Inherits:
Object
  • Object
show all
Includes:
Util
Defined in:
lib/rudder/dsl/pipeline.rb

Overview

Concourse Pipeline. Main entry of the DSL. Evaluates user defined pipelines.

DSL Usage:

Pipeline's are composed of various components:

  • Resource: basic inputs and output of jobs.

  • Job: basic computation unit of a pipeline

  • ResourceType: custom resource definitions

  • Group: logical grouping of jobs in the UI. Either every job is in a Group or no job is (hard Concourse requirement)

Adding Components

Components are added to the Pipeline by component type, followed by name, optional arguments, then typically a block.

Loading Other Pipelines

Pipeline's can load other pipeline definitions using #load. This is a useful mechanism for abstracting out common subsections of pipelines, then merging them into larger pipelines.

Merging Pipeline Components

Pipeline's can merge loaded or defined Rudder::DSL:Pipelines's, Hash's of components, or Array's of components into the current definition using #merge_components.

Including Other Pipelines

Pipeline's can include other pipeline definitions using #include_pipeline. This is similar to Rudder::DSL:Pipeline#load except that all the components are automatically included into this Pipeline

Including Individual Components

Individual pipeline components can also be defined on a per-file basis and then loaded into a Pipeline using #include_component. This is useful for factoring out common resources for multiple pipeline's to use.

Loading Concourse Variables

Occasionally it is helpful to have access to concourse variable when generating a pipeline, for example, when a Rudder pipeline should be parameterized on some value already stored in a Concourse parameter file. Rudder supports loading a concourse vars file from the Rudder command line. These are exposed to the pipeline dsl through the vars accessor.

Examples:

Adding Components to Pipelines

#
# my_pipeline_definition.rb
#
resource :my_git_repo, :git do
  source[:uri]    = 'https://github.com/my/repo.git'
  source[:branch] = :master
end

resource :daily, :time do
  source[:interval] = '24h'

job :build_project do
  plan << [in_parallel: [{ get: :my_git_repo }, { get: :daily, trigger: true}]]
  build = { task: 'build my project', config: {
    platform: :linux,
    image_resource: { type: 'docker-image', source: { repository: 'busybox' } },
    run: { path: 'my_git_repo/build.sh' }
  }}
  plan << build
end

Loading / Importing Pipelines

#
# load_neighbor.rb
#
neighbor = load 'neighbor_pipeline.rb'

# merge all the neighboring resources and jobs into this pipeline
resources.merge! neighbor.resources
jobs.merge! neighbor.jobs

resource_type :slack_notification, 'docker-image' do
  source[:repository] = 'some/slack-docker-repo'
end

resource :our_slack_channel, :slack_notification do
  source[:url] = '((slack-team-webhook))'
end

# Add a slack notification task to the end
# of every job
jobs.values.each do |job|
  job.plan << {
    put: :our_slack_channel,
    params: { text: "Job #{job.name} complete!" }
  }
end

Loading / Importing Individual Components

#
# operations_scripts_resource.rb
#
type :git
source[:uri]    = 'https://github.com/<our org>/operations_scripts.git'
source[:branch] = 'master'

#
# some_operations_pipeline.rb
#

# load the resource into the pipeline. Automatically includes
# the resource into the resources list with the name :scripts
include_component 'operations_scripts_resource.rb', :resource, :scripts

job :audit do |pipeline|
  plan << {
    task: 'run the audit script', config: {
      platform: :linux,
      image_resource: {
        type: 'docker-image',
        source: { repository: 'alpine/git' }
      },
      run: {
        path: pipeline.scripts.sub_path('audit.rb')
      }
    }
  }
end

Loading Concourse Variables

#
# vars_pipeline.rb
#
# Assuming a variables file of the form:
#
# my_branch: awesome-feature
#

my_branch = vars[:my_branch] # <--- Pulls in the variable here

resource :my_git_repo, :git do
  source[:uri]    = 'https://github.com/my/repo.git'
  source[:branch] = my_branch
end
...

See Also:

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods included from Util

#_convert_h_val, #_deep_to_h

Constructor Details

#initialize(file_path = nil, resources: {}, jobs: {}, groups: {}, resource_types: {}, vars: {}) ⇒ Pipeline

All pipelines require:

  • Jobs

  • Resources

Concourse Pipelines may optionally provide:

  • Resource Types

  • Groups

Rudder Pipelines may optionally include a file_path. This is required when loading resources from neighboring files.

All pipeline requirements are only needed at the Pipeline render time (after evaluation), and need not be specified for initialization.

Parameters:

  • file_path (String) (defaults to: nil) —

    path to this Rudder::DSL::Pipeline definition.

  • resources (Hash<(String, Symbol), Rudder::DSL::Resource] map of Resource names to their definitions.) —

    esources [Hash<(String, Symbol), Rudder::DSL::Resource] map of Resource names to their definitions.

  • jobs (Hash<(String, Symbol), Rudder::DSL::Job] map of Job names to their definitions.) —

    obs [Hash<(String, Symbol), Rudder::DSL::Job] map of Job names to their definitions.

  • groups (Hash<(String, Symbol), Rudder::DSL::Group] map of Group names to their definitions.) —

    roups [Hash<(String, Symbol), Rudder::DSL::Group] map of Group names to their definitions.

  • resources_types (Hash<(String, Symbol), Rudder::DSL::ResourceType] map of Resource Type names to their definitions.) —

    esources_types [Hash<(String, Symbol), Rudder::DSL::ResourceType] map of Resource Type names to their definitions.

  • vars (Hash<(String, Symbol), (String,Float,Hash,Array)] map of var names to corresponding concourse variable value) —

    ars [Hash<(String, Symbol), (String,Float,Hash,Array)] map of var names to corresponding concourse variable value



214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
 
# File 'lib/rudder/dsl/pipeline.rb', line 214

def initialize(file_path = nil, resources: {}, jobs: {},
               groups: {}, resource_types: {}, vars: {})
  @resources      = resources
  @jobs           = jobs
  @groups         = groups
  @resource_types = resource_types
  # rubocop:disable Layout/AlignHash, Layout/SpaceBeforeComma
  @known_classes  = {
    resource:      { clazz: Resource    , pipeline_group: @resources      },
    job:           { clazz: Job         , pipeline_group: @jobs           },
    group:         { clazz: Group       , pipeline_group: @groups         },
    resource_type: { clazz: ResourceType, pipeline_group: @resource_types }
  }
  # rubocop:enable Layout/AlignHash, Layout/SpaceBeforeComma
  @pipelines = {}
  @file_path = file_path
  @vars      = vars.map do |k, v|
    [k.to_sym, v]
  end.to_h
end

Dynamic Method Handling

This class handles dynamic methods through the method_missing method

#method_missing(method, *args, &component_block) ⇒ Rudder::DSL::Component?

Populates this Rudder::DSL::Pipeline with components and optionally fetches defined components.

Fetching

When method is called with no arguments it is treated as a Rudder::DSL::Pipeline getter method. method is translated to the name of a Component and the Component is returned if defined, otherwise nil is returned.

Setting

When method is passed any arguments (positional, placement, or block) then method is treated as a setter.

When setting, method must be the name of a known Component. The first argument is a required name for the component. All arguments and keyword arguments are then delegated to the Component's specific initializer.

Finally, when a block is provided it is evaluated within the context of the newly constructed Component with full priveleges to operate on it.

Returns:

Raises:

  • (RuntimeError) —

    when attempting to define an unknown Component



295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
 
# File 'lib/rudder/dsl/pipeline.rb', line 295

def method_missing(method, *args, &component_block)
  local_component = _get_local_component(method)
  if !@known_classes.include?(method) && !local_component
    return super.send(method, args, component_block)
  end

  # Look up a previously defined component from the pipeline
  return local_component if local_component && args.empty? && !block_given?

  component_group = @known_classes[method][:pipeline_group]
  name = args[0]
  raise "Overlapping component name: #{method}" if component_group.include? name

  component = @known_classes[method][:clazz].new(*args)

  component.instance_exec self, &component_block if block_given?
  component_group[name] = component
end

Instance Attribute Details

#groups ⇒ Hash<(String, Symbol), Rudder::DSL::Group>

Hash of names to Group

Returns:



180
181
182
 
# File 'lib/rudder/dsl/pipeline.rb', line 180

def groups
  @groups
end

#jobs ⇒ Hash<(String, Symbol), Rudder::DSL::Job>

Hash of names to Job

Returns:



174
175
176
 
# File 'lib/rudder/dsl/pipeline.rb', line 174

def jobs
  @jobs
end

#resource_types ⇒ Hash<(String, Symbol), Rudder::DSL::ResourceType>

Hash of names to ResourceType

Returns:



177
178
179
 
# File 'lib/rudder/dsl/pipeline.rb', line 177

def resource_types
  @resource_types
end

#resources ⇒ Hash<(String, Symbol), Rudder::DSL::Resource>

Hash of names to Resource

Returns:



171
172
173
 
# File 'lib/rudder/dsl/pipeline.rb', line 171

def resources
  @resources
end

#vars ⇒ Hash<Symbol, (String,Float,Hash,Array)>

Hash of concourse variables

Returns:

  • (Hash<Symbol, (String,Float,Hash,Array)>) 


183
184
185
 
# File 'lib/rudder/dsl/pipeline.rb', line 183

def vars
  @vars
end

Instance Method Details

#eval(file_path = nil) ⇒ Rudder::DSL::Pipeline

Evaluates the given file path. If file_path nil, defaults to the one provided at construction time If both are nil, raises an exception

Parameters:

  • file_path (String, nil) (defaults to: nil) —

    path to Rudder::DSL::Pipeline definition to evaluate. Uses the current file_path if nil

Returns:

Raises:

  • (RuntimeError) —

    if file_path and #file_path are both nil



337
338
339
340
341
342
343
344
345
346
347
 
# File 'lib/rudder/dsl/pipeline.rb', line 337

def eval(file_path = nil)
  @file_path = file_path || @file_path
  if @file_path.nil?
    raise 'File path must be provided at Pipeline initialization or eval call'
  end

  File.open(@file_path) do |f|
    instance_eval f.read, @file_path
  end
  self
end

#include_component(component_path, class_sym, name, *args) ⇒ Object

Note:

This automatically includes the component into this pipeline

Given a path to a component, its class, and any args required to construct it, creates a new component

Parameters:

  • component_path (String) —

    path, relative to this pipeline, containing a Component to load

  • class_sym (Symbol) —

    symbol of a Component. May be one of: (:job, :resource, :resource_type, :group)

  • name (String, Symbol) —

    name to use for the loaded Component. Must not be nil.

  • *args —

    any additional arguments to pass to the Component constructor.

Raises:

  • RuntimeError if name is nil or an uknown class_sym is provided.



405
406
407
408
409
410
411
412
413
414
 
# File 'lib/rudder/dsl/pipeline.rb', line 405

def include_component(component_path, class_sym, name, *args)
  raise "Unable to load #{class_sym}" unless @known_classes.keys.include? class_sym
  raise 'Name must not be nil' if name.nil?

  full_path = File.join(File.dirname(@file_path), component_path)
  component = @known_classes[class_sym][:clazz].new(name, *args)
  component.instance_eval File.read(full_path), full_path
  @known_classes[class_sym][:pipeline_group][name] = component
  component
end

#include_pipeline(other_pipeline_path) ⇒ nil

Note:

Any component provided that has an overlapping name with a previously defined component in this pipeline will override the previous definition

Includes another Rudder::DSL::Pipeline from a file into this Rudder::DSL::Pipeline.

Parameters:

Returns:

  • (nil) 


427
428
429
430
 
# File 'lib/rudder/dsl/pipeline.rb', line 427

def include_pipeline(other_pipeline_path)
  pipeline = load other_pipeline_path
  merge_components(pipeline)
end

#load(other_pipeline_path, resources: {}, resource_types: {}, jobs: {}, groups: {}) ⇒ Object

Given a path relative to this pipeline, loads another pipeline and returns it

Note that this includes nothing from the relative pipeline in this one, instead just returning the pipeline to be manipulated

May also optionally provides hashes for

  • resources

  • resource_types

  • jobs

  • groups

Parameters:



372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
 
# File 'lib/rudder/dsl/pipeline.rb', line 372

def load(other_pipeline_path, resources: {}, resource_types: {},
         jobs: {}, groups: {})
  if @pipelines.key? other_pipeline_path
    @pipelines[other_pipeline_path]
  else
    dir = File.dirname(@file_path)
    full_path = File.join(dir, other_pipeline_path)
    pipeline = Rudder::DSL::Pipeline.new(
      full_path, resources: resources, resource_types: resource_types,
                 jobs: jobs, groups: groups
    ).eval
    @pipelines[other_pipeline_path] = pipeline
    pipeline
  end
end

#merge_components(components) ⇒ nil

Note:

Any component provided that has an overlapping name with a previously defined component in this pipeline will override the previous definition

Merges the provided Rudder::DSL::Pipeline components into this Rudder::DSL::Pipeline.

Modes of Operation:

When provided a Pipeline, merges all like components into this Pipeline

  • Hash<String, Component>:

When provided a Hash, merges all the components into the like Hash in this Pipeline

  • Array<Component>:

When provided an Array, merges all Components into this Pipeline. Inserts components into their respective Pipeline group by class.

Parameters:

Returns:

  • (nil)

Raises:

  • (RuntimeError) —

    when type provided is unsupported



459
460
461
462
463
464
465
466
 
# File 'lib/rudder/dsl/pipeline.rb', line 459

def merge_components(components)
  case components
  when Pipeline then _merge_pipeline(components)
  when Hash then _merge_hash(components)
  when Array then _merge_array(components)
  else raise "Unable to merge #{components}: unsupported type"
  end
end

#p_convert_h_val(hash) ⇒ Object

Wraps Util#_convert_h_val since it will always set use_name to false



256
257
258
 
# File 'lib/rudder/dsl/pipeline.rb', line 256

def p_convert_h_val(hash)
  _convert_h_val(hash, false)
end

#respond_to?(name, _include_all = true) ⇒ Boolean

Returns:

  • (Boolean) 


323
324
325
 
# File 'lib/rudder/dsl/pipeline.rb', line 323

def respond_to?(name, _include_all = true)
  @known_classes.key? name
end

#respond_to_missing?(*_) ⇒ Boolean

Rudder::DSL::Pipeline's respond to missing

Returns:

  • (Boolean) —

    true



319
320
321
 
# File 'lib/rudder/dsl/pipeline.rb', line 319

def respond_to_missing?(*_)
  true
end

#to_h ⇒ Hash

Renders all of this pipeline's components to their Hash representations.

Returns:

  • (Hash) —

    YAML friendly Hash representation of this Pipeline if either groups or resource_types is empty they will not be included in the rendering at all.



243
244
245
246
247
248
249
250
251
 
# File 'lib/rudder/dsl/pipeline.rb', line 243

def to_h
  h = {
    'resources' => p_convert_h_val(@resources.values),
    'jobs' => p_convert_h_val(@jobs.values)
  }
  h['groups'] = p_convert_h_val(@groups.values) unless @groups.empty?
  h['resource_types'] = p_convert_h_val(@resource_types.values) unless @resource_types.empty?
  h
end