Pipes Segment Classes

Derived Pipe classes

class pipes.Regulator(functor_obj, name='Regulator', upstreams=None, downstreams=None, ignore_exceptions=None, init_kwargs=None)

Regulator pipes are a special type of transformation that changes the data chunk throughput, typically used for batching or accumulating data. Regulators can have both upstreams and downstreams. Functor should be a Python coroutine. The coroutine should not be initialized, instead use init_kwargs to initialize on the local process.

Parameters:

• functor – Python coroutines
• name – String associated with pipe segment
• upstreams – List of Streams that are inputs to functor
• downstreams – List of Streams that are outputs of functor
• init_kwargs – Kwargs to initiate class object on process (no used when func_type = ‘function’)
• ignore_exceptions – List of exceptions to ignore while pipeline is running

class pipes.Sink(functor_obj, name='Sink', upstreams=None, ignore_exceptions=None, init_kwargs=None)

Sink pipes are typically used to save/output data. Sinks have no downstreams, but will have one or more upstreams. Functor should be either a Python function or class with a “run” method and optionally “local_init” and “local_term” methods. Local_init, if supplied will be called once on the local process before run, while local_term will be called once afterwards.

Parameters:

• functor – Python function or class
• name – String associated with pipe segment
• upstreams – List of Streams that are inputs to functor
• init_kwargs – Kwargs to initiate class object on process (no used when func_type = ‘function’)
• ignore_exceptions – List of exceptions to ignore while pipeline is running

class pipes.Source(functor_obj, name='Source', downstreams=None, ignore_exceptions=None)

Source pipes are used to load and/or generate data. Sources have no upstreams, but will have one or more downstreams. Functor must be a valid Python generator. The generator should be initialized before passing as argument.

Parameters:

• functor – Python generator
• name – String associated with pipe segment
• downstreams – List of Streams that are outputs of functor
• ignore_exceptions – List of exceptions to ignore while pipeline is running

class pipes.Transform(functor_obj, name='Transform', upstreams=None, downstreams=None, ignore_exceptions=None, init_kwargs=None)

Transform pipes are used to perform arbitrary transformations on data. Transforms will have one or more upstreams and downstreams. Functor should be either a Python function or class with a “run” method and optionally “local_init” and “local_term” methods. Local_init, if supplied will be called once on the local process before run, while local_term will be called once afterwards.

Parameters:

• functor – Python function or class
• name – String associated with pipe segment
• upstreams – List of Streams that are inputs to functor
• downstreams – List of Streams that are outputs of functor
• init_kwargs – Kwargs to initiate class object on process (no used when func_type = ‘function’)
• ignore_exceptions – List of exceptions to ignore while pipeline is running