BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE BIG DATA ERA Blazing fast and open-source data processor. Baker pipelines can fetch, transform and store records in a flash thanks to a high-performant and fully-parallel implementation. Baker is open-source and written in Go by NextRoll, inc.

DOCUMENTATION

A high performance, composable and extendable data-processing pipeline for the big data era

GETTING STARTED

Looking to use Baker and start building pipelines now? Great, let’s see what you need. Baker is written in Go. To use it you need to import the Baker module into your program.

CORE CONCEPTS

HOW-TOS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR To configure and run a Baker topology, 4 steps are required: use a TOML configuration file; define a baker.Components object; obtain a Baker configuration object calling baker.NewConfigFromToml run baker.Main; The example folder in the Baker repositories contains many examples of implementing a Baker pipeline.. Start with the basic

example.

COMPONENTS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR All Baker components: inputs, filters, outputs and uploads

PERFORMANCE

On an AWS EC2 instance of size c5.2xlarge, Baker can read zstandard records from S3, uncompress them and apply a basic filtering logic, compressing them back on local files using ~90% of capacity of each vCPU (8 in total) and ~3.5GB of RAM. It reads and writes a total of 94 million records in less than 9 minutes, that’s 178k records per

second.

CREATE A CUSTOM INPUT COMPONENT The Run function implements the component logic and receives a channel where it sends the raw data it processes.. FreeMem(data *Data) is called by Baker when data is no longer needed. This is an occasion for the input to recycle memory, for example if the input uses a sync.Pool to create new instances of baker.Data.. InputDesc CREATE A CUSTOM OUTPUT COMPONENT The Run function implements the component logic and gets a channel where it receives OutputRecord objects and a channel to communicate to the Upload components what to upload.. CanShard is the function telling whether the output is able to manage sharding. Read the page dedicated to the sharding to go deeper in the topic. Stats is used to report metrics, see the dedicated page. PIPELINE CONFIGURATION defines a structure of the records with two fields: foo as first element and bar as second. Validation configuration. The section is an optional configuration that contains one or more field names each of which is associated with a regular expression. If the validation section is specified Baker automatically generates a validation function, which checks that each input record BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE BIG DATA ERA Blazing fast and open-source data processor. Baker pipelines can fetch, transform and store records in a flash thanks to a high-performant and fully-parallel implementation. Baker is open-source and written in Go by NextRoll, inc.

DOCUMENTATION

A high performance, composable and extendable data-processing pipeline for the big data era

GETTING STARTED

Looking to use Baker and start building pipelines now? Great, let’s see what you need. Baker is written in Go. To use it you need to import the Baker module into your program.

CORE CONCEPTS

HOW-TOS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR To configure and run a Baker topology, 4 steps are required: use a TOML configuration file; define a baker.Components object; obtain a Baker configuration object calling baker.NewConfigFromToml run baker.Main; The example folder in the Baker repositories contains many examples of implementing a Baker pipeline.. Start with the basic

example.

COMPONENTS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR All Baker components: inputs, filters, outputs and uploads

PERFORMANCE

On an AWS EC2 instance of size c5.2xlarge, Baker can read zstandard records from S3, uncompress them and apply a basic filtering logic, compressing them back on local files using ~90% of capacity of each vCPU (8 in total) and ~3.5GB of RAM. It reads and writes a total of 94 million records in less than 9 minutes, that’s 178k records per

second.

CREATE A CUSTOM INPUT COMPONENT The Run function implements the component logic and receives a channel where it sends the raw data it processes.. FreeMem(data *Data) is called by Baker when data is no longer needed. This is an occasion for the input to recycle memory, for example if the input uses a sync.Pool to create new instances of baker.Data.. InputDesc CREATE A CUSTOM OUTPUT COMPONENT The Run function implements the component logic and gets a channel where it receives OutputRecord objects and a channel to communicate to the Upload components what to upload.. CanShard is the function telling whether the output is able to manage sharding. Read the page dedicated to the sharding to go deeper in the topic. Stats is used to report metrics, see the dedicated page. PIPELINE CONFIGURATION defines a structure of the records with two fields: foo as first element and bar as second. Validation configuration. The section is an optional configuration that contains one or more field names each of which is associated with a regular expression. If the validation section is specified Baker automatically generates a validation function, which checks that each input record

DOCUMENTATION

A high performance, composable and extendable data-processing pipeline for the big data era BASIC: BUILD A SIMPLE PIPELINE A step-by-step tutorial to learn how to build a Baker pipeline using the included components UPLOADS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR A high performance, composable and extendable data-processing pipeline for the big data era CREATE A CUSTOM OUTPUT COMPONENT Output components in Baker receive records at the end of the filter chain and are in charge of storing them, eventually sending the result (like a temporary file in the disk) to an Upload component. To create an output and make it available to Baker, one must: Implement the Output interface Fill-up an OutputDesc struct and register it within Baker via Components The Output interface New Output

BAKER.COMPONENTS

Validate is the function used to validate a record. It is called for each processed record unless nil or when dont_validate_fields is set to true in TOML’s section.. Regardless of the dont_validate_fields value, the Validate function is made accessible to all components so that they can use it at their will.. A simple validation function based on regular expression could be enabled

EXPORT METRICS

Baker can publish various kind of metrics that may be used to monitor a pipeline in execution. The metrics exported range from numbers giving an high-level overview of the ongoing pipeline (total processed records, current speed in records per second, etc.) or per-component metrics such as the number of files read or written, to performance statistics published by the Go runtime in LIST | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE Read the API documentation » Input List Overview This input fetches logs from a predefined list of local or remote sources. The “Files” configuration variable is a list of “file specifiers”. Each “file specifier” can be: A local file path on the filesystem: the log file at that path will be processed A HTTP/HTTPS URL: the log file at that URL will be downloaded and processed A S3

TUNING CONCURRENCY

Baker allows to tune concurrency at various levels of a pipeline: input: Baker configuration doesn’t expose knobs to tune input concurrency as it highly depends on the input source and how the input is implemented filters: Baker runs N concurrent filter chains output: Baker runs M concurrent outputs By default then, Baker processes records concurrently, without any guaranteed order. CREATE A CUSTOM FILTER COMPONENT Create a custom filter component. Creating a custom filter is probably the most common action a Baker user will perform. In fact, filters are the components that apply the business logic to a Baker pipeline, creating or discarding records or modifying fields. A working example of a custom filter can be found in the filtering example To create a filter and make it available to Baker, one must: SQS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE Read the API documentation » Input SQS Overview This input listens on multiple SQS queues for new incoming log files on S3; it is meant to be used with SQS queues popoulated by SNS. It never exits. Configuration Keys available in the section: Name Type Default Required Description AwsRegion string “us-west-2” false AWS region to connect to Bucket string "" false S3 BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE BIG DATA ERA Blazing fast and open-source data processor. Baker pipelines can fetch, transform and store records in a flash thanks to a high-performant and fully-parallel implementation. Baker is open-source and written in Go by NextRoll, inc.

DOCUMENTATION

A high performance, composable and extendable data-processing pipeline for the big data era

GETTING STARTED

Looking to use Baker and start building pipelines now? Great, let’s see what you need. Baker is written in Go. To use it you need to import the Baker module into your program.

CORE CONCEPTS

HOW-TOS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR To configure and run a Baker topology, 4 steps are required: use a TOML configuration file; define a baker.Components object; obtain a Baker configuration object calling baker.NewConfigFromToml run baker.Main; The example folder in the Baker repositories contains many examples of implementing a Baker pipeline.. Start with the basic

example.

PERFORMANCE

On an AWS EC2 instance of size c5.2xlarge, Baker can read zstandard records from S3, uncompress them and apply a basic filtering logic, compressing them back on local files using ~90% of capacity of each vCPU (8 in total) and ~3.5GB of RAM. It reads and writes a total of 94 million records in less than 9 minutes, that’s 178k records per

second.

COMPONENTS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR All Baker components: inputs, filters, outputs and uploads CREATE A CUSTOM OUTPUT COMPONENT The Run function implements the component logic and gets a channel where it receives OutputRecord objects and a channel to communicate to the Upload components what to upload.. CanShard is the function telling whether the output is able to manage sharding. Read the page dedicated to the sharding to go deeper in the topic. Stats is used to report metrics, see the dedicated page. PIPELINE CONFIGURATION defines a structure of the records with two fields: foo as first element and bar as second. Validation configuration. The section is an optional configuration that contains one or more field names each of which is associated with a regular expression. If the validation section is specified Baker automatically generates a validation function, which checks that each input record FILTERS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR A high performance, composable and extendable data-processing pipeline for the big data era BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE BIG DATA ERA Blazing fast and open-source data processor. Baker pipelines can fetch, transform and store records in a flash thanks to a high-performant and fully-parallel implementation. Baker is open-source and written in Go by NextRoll, inc.

DOCUMENTATION

A high performance, composable and extendable data-processing pipeline for the big data era

GETTING STARTED

Looking to use Baker and start building pipelines now? Great, let’s see what you need. Baker is written in Go. To use it you need to import the Baker module into your program.

CORE CONCEPTS

HOW-TOS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR To configure and run a Baker topology, 4 steps are required: use a TOML configuration file; define a baker.Components object; obtain a Baker configuration object calling baker.NewConfigFromToml run baker.Main; The example folder in the Baker repositories contains many examples of implementing a Baker pipeline.. Start with the basic

example.

PERFORMANCE

On an AWS EC2 instance of size c5.2xlarge, Baker can read zstandard records from S3, uncompress them and apply a basic filtering logic, compressing them back on local files using ~90% of capacity of each vCPU (8 in total) and ~3.5GB of RAM. It reads and writes a total of 94 million records in less than 9 minutes, that’s 178k records per

second.

COMPONENTS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR All Baker components: inputs, filters, outputs and uploads CREATE A CUSTOM OUTPUT COMPONENT The Run function implements the component logic and gets a channel where it receives OutputRecord objects and a channel to communicate to the Upload components what to upload.. CanShard is the function telling whether the output is able to manage sharding. Read the page dedicated to the sharding to go deeper in the topic. Stats is used to report metrics, see the dedicated page. PIPELINE CONFIGURATION defines a structure of the records with two fields: foo as first element and bar as second. Validation configuration. The section is an optional configuration that contains one or more field names each of which is associated with a regular expression. If the validation section is specified Baker automatically generates a validation function, which checks that each input record FILTERS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR A high performance, composable and extendable data-processing pipeline for the big data era HOW-TOS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR To configure and run a Baker topology, 4 steps are required: use a TOML configuration file; define a baker.Components object; obtain a Baker configuration object calling baker.NewConfigFromToml run baker.Main; The example folder in the Baker repositories contains many examples of implementing a Baker pipeline.. Start with the basic

example.

COMPONENTS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR All Baker components: inputs, filters, outputs and uploads CREATE A CUSTOM INPUT COMPONENT The Run function implements the component logic and receives a channel where it sends the raw data it processes.. FreeMem(data *Data) is called by Baker when data is no longer needed. This is an occasion for the input to recycle memory, for example if the input uses a sync.Pool to create new instances of baker.Data.. InputDesc CREATE A CUSTOM OUTPUT COMPONENT Output components in Baker receive records at the end of the filter chain and are in charge of storing them, eventually sending the result (like a temporary file in the disk) to an Upload component. To create an output and make it available to Baker, one must: Implement the Output interface Fill-up an OutputDesc struct and register it within Baker via Components The Output interface New Output FILTERS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR A high performance, composable and extendable data-processing pipeline for the big data era LIST | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE Read the API documentation » Input List Overview This input fetches logs from a predefined list of local or remote sources. The “Files” configuration variable is a list of “file specifiers”. Each “file specifier” can be: A local file path on the filesystem: the log file at that path will be processed A HTTP/HTTPS URL: the log file at that URL will be downloaded and processed A S3 TCP | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE Read the API documentation » Input TCP Overview This input relies on a TCP connection to receive records in the usual format Configure it with a host and port that you want to accept connection from. By default it listens on port 6000 for any connection It never exits. Configuration Keys available in the section: Name Type Default Required Description Listener string "" false KINESIS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR Read the API documentation » Input Kinesis Overview This input fetches log lines from Kinesis. It listens on a specified stream, and processes all the shards in that stream. It never exits. Configuration Keys available in the section: Name Type Default Required Description AwsRegion string “us-west-2” false AWS region to connect to Stream string "" true Stream name on SQS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE Read the API documentation » Input SQS Overview This input listens on multiple SQS queues for new incoming log files on S3; it is meant to be used with SQS queues popoulated by SNS. It never exits. Configuration Keys available in the section: Name Type Default Required Description AwsRegion string “us-west-2” false AWS region to connect to Bucket string "" false S3 SLICE | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE Read the API documentation » Filter Slice Overview Slices the source field value using start/end indexes and copies the value to the destination field. If the start index is greater than the field length, Slice sets the destination to an empty string. If the end index is greater than the field length, Slice considers the end index to be equal to the field length. Note: Indexes are 0-based and BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE BIG DATA ERA Blazing fast and open-source data processor. Baker pipelines can fetch, transform and store records in a flash thanks to a high-performant and fully-parallel implementation. Baker is open-source and written in Go by NextRoll, inc.

DOCUMENTATION

A high performance, composable and extendable data-processing pipeline for the big data era

GETTING STARTED

Looking to use Baker and start building pipelines now? Great, let’s see what you need. Baker is written in Go. To use it you need to import the Baker module into your program.

CORE CONCEPTS

HOW-TOS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR To configure and run a Baker topology, 4 steps are required: use a TOML configuration file; define a baker.Components object; obtain a Baker configuration object calling baker.NewConfigFromToml run baker.Main; The example folder in the Baker repositories contains many examples of implementing a Baker pipeline.. Start with the basic

example.

PERFORMANCE

On an AWS EC2 instance of size c5.2xlarge, Baker can read zstandard records from S3, uncompress them and apply a basic filtering logic, compressing them back on local files using ~90% of capacity of each vCPU (8 in total) and ~3.5GB of RAM. It reads and writes a total of 94 million records in less than 9 minutes, that’s 178k records per

second.

COMPONENTS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR All Baker components: inputs, filters, outputs and uploads CREATE A CUSTOM OUTPUT COMPONENT The Run function implements the component logic and gets a channel where it receives OutputRecord objects and a channel to communicate to the Upload components what to upload.. CanShard is the function telling whether the output is able to manage sharding. Read the page dedicated to the sharding to go deeper in the topic. Stats is used to report metrics, see the dedicated page. PIPELINE CONFIGURATION defines a structure of the records with two fields: foo as first element and bar as second. Validation configuration. The section is an optional configuration that contains one or more field names each of which is associated with a regular expression. If the validation section is specified Baker automatically generates a validation function, which checks that each input record FILTERS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR A high performance, composable and extendable data-processing pipeline for the big data era BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE BIG DATA ERA Blazing fast and open-source data processor. Baker pipelines can fetch, transform and store records in a flash thanks to a high-performant and fully-parallel implementation. Baker is open-source and written in Go by NextRoll, inc.

DOCUMENTATION

A high performance, composable and extendable data-processing pipeline for the big data era

GETTING STARTED

Looking to use Baker and start building pipelines now? Great, let’s see what you need. Baker is written in Go. To use it you need to import the Baker module into your program.

CORE CONCEPTS

HOW-TOS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR To configure and run a Baker topology, 4 steps are required: use a TOML configuration file; define a baker.Components object; obtain a Baker configuration object calling baker.NewConfigFromToml run baker.Main; The example folder in the Baker repositories contains many examples of implementing a Baker pipeline.. Start with the basic

example.

PERFORMANCE

On an AWS EC2 instance of size c5.2xlarge, Baker can read zstandard records from S3, uncompress them and apply a basic filtering logic, compressing them back on local files using ~90% of capacity of each vCPU (8 in total) and ~3.5GB of RAM. It reads and writes a total of 94 million records in less than 9 minutes, that’s 178k records per

second.

COMPONENTS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR All Baker components: inputs, filters, outputs and uploads CREATE A CUSTOM OUTPUT COMPONENT The Run function implements the component logic and gets a channel where it receives OutputRecord objects and a channel to communicate to the Upload components what to upload.. CanShard is the function telling whether the output is able to manage sharding. Read the page dedicated to the sharding to go deeper in the topic. Stats is used to report metrics, see the dedicated page. PIPELINE CONFIGURATION defines a structure of the records with two fields: foo as first element and bar as second. Validation configuration. The section is an optional configuration that contains one or more field names each of which is associated with a regular expression. If the validation section is specified Baker automatically generates a validation function, which checks that each input record FILTERS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR A high performance, composable and extendable data-processing pipeline for the big data era HOW-TOS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR To configure and run a Baker topology, 4 steps are required: use a TOML configuration file; define a baker.Components object; obtain a Baker configuration object calling baker.NewConfigFromToml run baker.Main; The example folder in the Baker repositories contains many examples of implementing a Baker pipeline.. Start with the basic

example.

FILTERS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR A high performance, composable and extendable data-processing pipeline for the big data era CREATE A CUSTOM INPUT COMPONENT The Run function implements the component logic and receives a channel where it sends the raw data it processes.. FreeMem(data *Data) is called by Baker when data is no longer needed. This is an occasion for the input to recycle memory, for example if the input uses a sync.Pool to create new instances of baker.Data.. InputDesc CREATE A CUSTOM OUTPUT COMPONENT Output components in Baker receive records at the end of the filter chain and are in charge of storing them, eventually sending the result (like a temporary file in the disk) to an Upload component. To create an output and make it available to Baker, one must: Implement the Output interface Fill-up an OutputDesc struct and register it within Baker via Components The Output interface New Output LIST | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE Read the API documentation » Input List Overview This input fetches logs from a predefined list of local or remote sources. The “Files” configuration variable is a list of “file specifiers”. Each “file specifier” can be: A local file path on the filesystem: the log file at that path will be processed A HTTP/HTTPS URL: the log file at that URL will be downloaded and processed A S3 SLICE | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE Read the API documentation » Filter Slice Overview Slices the source field value using start/end indexes and copies the value to the destination field. If the start index is greater than the field length, Slice sets the destination to an empty string. If the end index is greater than the field length, Slice considers the end index to be equal to the field length. Note: Indexes are 0-based and FORMATTIME | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR Read the API documentation » Filter FormatTime Overview This filter formats and converts date/time strings from one format to another. It requires the source and destination field names along with 2 format strings, the first one indicates how to parse the input field while the second how to format it. The source time parsing can fail if the time value does not match the provided format. SQS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE Read the API documentation » Input SQS Overview This input listens on multiple SQS queues for new incoming log files on S3; it is meant to be used with SQS queues popoulated by SNS. It never exits. Configuration Keys available in the section: Name Type Default Required Description AwsRegion string “us-west-2” false AWS region to connect to Bucket string "" false S3

CLAUSEFILTER

Read the API documentation » Filter ClauseFilter Overview Discard records which do not match a clause given as a boolean S-expression. Check the filter documentation for some examples. ClauseFilter boolean expression format This document describes the s-expression format used in ClauseFilter. The format uses s-expressions. Empty string matches anything (i.e. all records will pass EXPANDLIST | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR Read the API documentation » Filter ExpandList Overview This filter splits a field using a configured separator and writes the resulting values to other fields of the same record. The mapping between the extracted values and the destination fields is configured with a TOML table. The elements of the list are, by default, separated with the ; character, but it is configurable. BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE BIG DATA ERA Blazing fast and open-source data processor. Baker pipelines can fetch, transform and store records in a flash thanks to a high-performant and fully-parallel implementation. Baker is open-source and written in Go by NextRoll, inc.

DOCUMENTATION

A high performance, composable and extendable data-processing pipeline for the big data era

GETTING STARTED

Looking to use Baker and start building pipelines now? Great, let’s see what you need. Baker is written in Go. To use it you need to import the Baker module into your program. HOW-TOS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR To configure and run a Baker topology, 4 steps are required: use a TOML configuration file; define a baker.Components object; obtain a Baker configuration object calling baker.NewConfigFromToml run baker.Main; The example folder in the Baker repositories contains many examples of implementing a Baker pipeline.. Start with the basic

example.

COMPONENTS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR All Baker components: inputs, filters, outputs and uploads

CORE CONCEPTS

PERFORMANCE

On an AWS EC2 instance of size c5.2xlarge, Baker can read zstandard records from S3, uncompress them and apply a basic filtering logic, compressing them back on local files using ~90% of capacity of each vCPU (8 in total) and ~3.5GB of RAM. It reads and writes a total of 94 million records in less than 9 minutes, that’s 178k records per

second.

PIPELINE CONFIGURATION defines a structure of the records with two fields: foo as first element and bar as second. Validation configuration. The section is an optional configuration that contains one or more field names each of which is associated with a regular expression. If the validation section is specified Baker automatically generates a validation function, which checks that each input record CREATE A CUSTOM INPUT COMPONENT The Run function implements the component logic and receives a channel where it sends the raw data it processes.. FreeMem(data *Data) is called by Baker when data is no longer needed. This is an occasion for the input to recycle memory, for example if the input uses a sync.Pool to create new instances of baker.Data.. InputDesc CREATE A CUSTOM OUTPUT COMPONENT The Run function implements the component logic and gets a channel where it receives OutputRecord objects and a channel to communicate to the Upload components what to upload.. CanShard is the function telling whether the output is able to manage sharding. Read the page dedicated to the sharding to go deeper in the topic. Stats is used to report metrics, see the dedicated page. BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE BIG DATA ERA Blazing fast and open-source data processor. Baker pipelines can fetch, transform and store records in a flash thanks to a high-performant and fully-parallel implementation. Baker is open-source and written in Go by NextRoll, inc.

DOCUMENTATION

A high performance, composable and extendable data-processing pipeline for the big data era

GETTING STARTED

Looking to use Baker and start building pipelines now? Great, let’s see what you need. Baker is written in Go. To use it you need to import the Baker module into your program. HOW-TOS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR To configure and run a Baker topology, 4 steps are required: use a TOML configuration file; define a baker.Components object; obtain a Baker configuration object calling baker.NewConfigFromToml run baker.Main; The example folder in the Baker repositories contains many examples of implementing a Baker pipeline.. Start with the basic

example.

COMPONENTS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR All Baker components: inputs, filters, outputs and uploads

CORE CONCEPTS

PERFORMANCE

On an AWS EC2 instance of size c5.2xlarge, Baker can read zstandard records from S3, uncompress them and apply a basic filtering logic, compressing them back on local files using ~90% of capacity of each vCPU (8 in total) and ~3.5GB of RAM. It reads and writes a total of 94 million records in less than 9 minutes, that’s 178k records per

second.

PIPELINE CONFIGURATION defines a structure of the records with two fields: foo as first element and bar as second. Validation configuration. The section is an optional configuration that contains one or more field names each of which is associated with a regular expression. If the validation section is specified Baker automatically generates a validation function, which checks that each input record CREATE A CUSTOM INPUT COMPONENT The Run function implements the component logic and receives a channel where it sends the raw data it processes.. FreeMem(data *Data) is called by Baker when data is no longer needed. This is an occasion for the input to recycle memory, for example if the input uses a sync.Pool to create new instances of baker.Data.. InputDesc CREATE A CUSTOM OUTPUT COMPONENT The Run function implements the component logic and gets a channel where it receives OutputRecord objects and a channel to communicate to the Upload components what to upload.. CanShard is the function telling whether the output is able to manage sharding. Read the page dedicated to the sharding to go deeper in the topic. Stats is used to report metrics, see the dedicated page.

DOCUMENTATION

A high performance, composable and extendable data-processing pipeline for the big data era BASIC: BUILD A SIMPLE PIPELINE A step-by-step tutorial to learn how to build a Baker pipeline using the included components UPLOADS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR A high performance, composable and extendable data-processing pipeline for the big data era CREATE A CUSTOM OUTPUT COMPONENT Output components in Baker receive records at the end of the filter chain and are in charge of storing them, eventually sending the result (like a temporary file in the disk) to an Upload component. To create an output and make it available to Baker, one must: Implement the Output interface Fill-up an OutputDesc struct and register it within Baker via Components The Output interface New Output LIST | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE Read the API documentation » Input List Overview This input fetches logs from a predefined list of local or remote sources. The “Files” configuration variable is a list of “file specifiers”. Each “file specifier” can be: A local file path on the filesystem: the log file at that path will be processed A HTTP/HTTPS URL: the log file at that URL will be downloaded and processed A S3

TUNING CONCURRENCY

Baker allows to tune concurrency at various levels of a pipeline: input: Baker configuration doesn’t expose knobs to tune input concurrency as it highly depends on the input source and how the input is implemented filters: Baker runs N concurrent filter chains output: Baker runs M concurrent outputs By default then, Baker processes records concurrently, without any guaranteed order.

EXPORT METRICS

Baker can publish various kind of metrics that may be used to monitor a pipeline in execution. The metrics exported range from numbers giving an high-level overview of the ongoing pipeline (total processed records, current speed in records per second, etc.) or per-component metrics such as the number of files read or written, to performance statistics published by the Go runtime in DYNAMODB | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR Read the API documentation » Output DynamoDB Overview This is a non-raw output, it doesn’t receive whole records. Instead it receives a list of fields for each record (output.fields in TOML). This output writes the filtered log lines to DynamoDB. It must be configured specifying the region, the table name, and the columns to write. Columns are specified using the syntax “t:name” where CREATE A CUSTOM FILTER COMPONENT Create a custom filter component. Creating a custom filter is probably the most common action a Baker user will perform. In fact, filters are the components that apply the business logic to a Baker pipeline, creating or discarding records or modifying fields. A working example of a custom filter can be found in the filtering example To create a filter and make it available to Baker, one must: SQS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE Read the API documentation » Input SQS Overview This input listens on multiple SQS queues for new incoming log files on S3; it is meant to be used with SQS queues popoulated by SNS. It never exits. Configuration Keys available in the section: Name Type Default Required Description AwsRegion string “us-west-2” false AWS region to connect to Bucket string "" false S3 BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE BIG DATA ERA Blazing fast and open-source data processor. Baker pipelines can fetch, transform and store records in a flash thanks to a high-performant and fully-parallel implementation. Baker is open-source and written in Go by NextRoll, inc.

DOCUMENTATION

A high performance, composable and extendable data-processing pipeline for the big data era

GETTING STARTED

Looking to use Baker and start building pipelines now? Great, let’s see what you need. Baker is written in Go. To use it you need to import the Baker module into your program. HOW-TOS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR To configure and run a Baker topology, 4 steps are required: use a TOML configuration file; define a baker.Components object; obtain a Baker configuration object calling baker.NewConfigFromToml run baker.Main; The example folder in the Baker repositories contains many examples of implementing a Baker pipeline.. Start with the basic

example.

COMPONENTS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR All Baker components: inputs, filters, outputs and uploads

CORE CONCEPTS

PERFORMANCE

On an AWS EC2 instance of size c5.2xlarge, Baker can read zstandard records from S3, uncompress them and apply a basic filtering logic, compressing them back on local files using ~90% of capacity of each vCPU (8 in total) and ~3.5GB of RAM. It reads and writes a total of 94 million records in less than 9 minutes, that’s 178k records per

second.

PIPELINE CONFIGURATION defines a structure of the records with two fields: foo as first element and bar as second. Validation configuration. The section is an optional configuration that contains one or more field names each of which is associated with a regular expression. If the validation section is specified Baker automatically generates a validation function, which checks that each input record CREATE A CUSTOM INPUT COMPONENT The Run function implements the component logic and receives a channel where it sends the raw data it processes.. FreeMem(data *Data) is called by Baker when data is no longer needed. This is an occasion for the input to recycle memory, for example if the input uses a sync.Pool to create new instances of baker.Data.. InputDesc CREATE A CUSTOM OUTPUT COMPONENT The Run function implements the component logic and gets a channel where it receives OutputRecord objects and a channel to communicate to the Upload components what to upload.. CanShard is the function telling whether the output is able to manage sharding. Read the page dedicated to the sharding to go deeper in the topic. Stats is used to report metrics, see the dedicated page. BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE BIG DATA ERA Blazing fast and open-source data processor. Baker pipelines can fetch, transform and store records in a flash thanks to a high-performant and fully-parallel implementation. Baker is open-source and written in Go by NextRoll, inc.

DOCUMENTATION

A high performance, composable and extendable data-processing pipeline for the big data era

GETTING STARTED

Looking to use Baker and start building pipelines now? Great, let’s see what you need. Baker is written in Go. To use it you need to import the Baker module into your program. HOW-TOS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR To configure and run a Baker topology, 4 steps are required: use a TOML configuration file; define a baker.Components object; obtain a Baker configuration object calling baker.NewConfigFromToml run baker.Main; The example folder in the Baker repositories contains many examples of implementing a Baker pipeline.. Start with the basic

example.

COMPONENTS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR All Baker components: inputs, filters, outputs and uploads

CORE CONCEPTS

PERFORMANCE

On an AWS EC2 instance of size c5.2xlarge, Baker can read zstandard records from S3, uncompress them and apply a basic filtering logic, compressing them back on local files using ~90% of capacity of each vCPU (8 in total) and ~3.5GB of RAM. It reads and writes a total of 94 million records in less than 9 minutes, that’s 178k records per

second.

PIPELINE CONFIGURATION defines a structure of the records with two fields: foo as first element and bar as second. Validation configuration. The section is an optional configuration that contains one or more field names each of which is associated with a regular expression. If the validation section is specified Baker automatically generates a validation function, which checks that each input record CREATE A CUSTOM INPUT COMPONENT The Run function implements the component logic and receives a channel where it sends the raw data it processes.. FreeMem(data *Data) is called by Baker when data is no longer needed. This is an occasion for the input to recycle memory, for example if the input uses a sync.Pool to create new instances of baker.Data.. InputDesc CREATE A CUSTOM OUTPUT COMPONENT The Run function implements the component logic and gets a channel where it receives OutputRecord objects and a channel to communicate to the Upload components what to upload.. CanShard is the function telling whether the output is able to manage sharding. Read the page dedicated to the sharding to go deeper in the topic. Stats is used to report metrics, see the dedicated page.

DOCUMENTATION

A high performance, composable and extendable data-processing pipeline for the big data era BASIC: BUILD A SIMPLE PIPELINE A step-by-step tutorial to learn how to build a Baker pipeline using the included components UPLOADS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR A high performance, composable and extendable data-processing pipeline for the big data era CREATE A CUSTOM OUTPUT COMPONENT Output components in Baker receive records at the end of the filter chain and are in charge of storing them, eventually sending the result (like a temporary file in the disk) to an Upload component. To create an output and make it available to Baker, one must: Implement the Output interface Fill-up an OutputDesc struct and register it within Baker via Components The Output interface New Output LIST | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE Read the API documentation » Input List Overview This input fetches logs from a predefined list of local or remote sources. The “Files” configuration variable is a list of “file specifiers”. Each “file specifier” can be: A local file path on the filesystem: the log file at that path will be processed A HTTP/HTTPS URL: the log file at that URL will be downloaded and processed A S3

TUNING CONCURRENCY

Baker allows to tune concurrency at various levels of a pipeline: input: Baker configuration doesn’t expose knobs to tune input concurrency as it highly depends on the input source and how the input is implemented filters: Baker runs N concurrent filter chains output: Baker runs M concurrent outputs By default then, Baker processes records concurrently, without any guaranteed order.

EXPORT METRICS

Baker can publish various kind of metrics that may be used to monitor a pipeline in execution. The metrics exported range from numbers giving an high-level overview of the ongoing pipeline (total processed records, current speed in records per second, etc.) or per-component metrics such as the number of files read or written, to performance statistics published by the Go runtime in DYNAMODB | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR Read the API documentation » Output DynamoDB Overview This is a non-raw output, it doesn’t receive whole records. Instead it receives a list of fields for each record (output.fields in TOML). This output writes the filtered log lines to DynamoDB. It must be configured specifying the region, the table name, and the columns to write. Columns are specified using the syntax “t:name” where CREATE A CUSTOM FILTER COMPONENT Create a custom filter component. Creating a custom filter is probably the most common action a Baker user will perform. In fact, filters are the components that apply the business logic to a Baker pipeline, creating or discarding records or modifying fields. A working example of a custom filter can be found in the filtering example To create a filter and make it available to Baker, one must: SQS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE Read the API documentation » Input SQS Overview This input listens on multiple SQS queues for new incoming log files on S3; it is meant to be used with SQS queues popoulated by SNS. It never exits. Configuration Keys available in the section: Name Type Default Required Description AwsRegion string “us-west-2” false AWS region to connect to Bucket string "" false S3 BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE BIG DATA ERA Blazing fast and open-source data processor. Baker pipelines can fetch, transform and store records in a flash thanks to a high-performant and fully-parallel implementation. Baker is open-source and written in Go by NextRoll, inc.

DOCUMENTATION

A high performance, composable and extendable data-processing pipeline for the big data era

GETTING STARTED

Looking to use Baker and start building pipelines now? Great, let’s see what you need. Baker is written in Go. To use it you need to import the Baker module into your program. HOW-TOS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR To configure and run a Baker topology, 4 steps are required: use a TOML configuration file; define a baker.Components object; obtain a Baker configuration object calling baker.NewConfigFromToml run baker.Main; The example folder in the Baker repositories contains many examples of implementing a Baker pipeline.. Start with the basic

example.

CORE CONCEPTS

COMPONENTS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR All Baker components: inputs, filters, outputs and uploads

PERFORMANCE

On an AWS EC2 instance of size c5.2xlarge, Baker can read zstandard records from S3, uncompress them and apply a basic filtering logic, compressing them back on local files using ~90% of capacity of each vCPU (8 in total) and ~3.5GB of RAM. It reads and writes a total of 94 million records in less than 9 minutes, that’s 178k records per

second.

PIPELINE CONFIGURATION defines a structure of the records with two fields: foo as first element and bar as second. Validation configuration. The section is an optional configuration that contains one or more field names each of which is associated with a regular expression. If the validation section is specified Baker automatically generates a validation function, which checks that each input record CREATE A CUSTOM INPUT COMPONENT The Run function implements the component logic and receives a channel where it sends the raw data it processes.. FreeMem(data *Data) is called by Baker when data is no longer needed. This is an occasion for the input to recycle memory, for example if the input uses a sync.Pool to create new instances of baker.Data.. InputDesc CREATE A CUSTOM OUTPUT COMPONENT The Run function implements the component logic and gets a channel where it receives OutputRecord objects and a channel to communicate to the Upload components what to upload.. CanShard is the function telling whether the output is able to manage sharding. Read the page dedicated to the sharding to go deeper in the topic. Stats is used to report metrics, see the dedicated page. BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE BIG DATA ERA Blazing fast and open-source data processor. Baker pipelines can fetch, transform and store records in a flash thanks to a high-performant and fully-parallel implementation. Baker is open-source and written in Go by NextRoll, inc.

DOCUMENTATION

A high performance, composable and extendable data-processing pipeline for the big data era

GETTING STARTED

Looking to use Baker and start building pipelines now? Great, let’s see what you need. Baker is written in Go. To use it you need to import the Baker module into your program. HOW-TOS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR To configure and run a Baker topology, 4 steps are required: use a TOML configuration file; define a baker.Components object; obtain a Baker configuration object calling baker.NewConfigFromToml run baker.Main; The example folder in the Baker repositories contains many examples of implementing a Baker pipeline.. Start with the basic

example.

CORE CONCEPTS

COMPONENTS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR All Baker components: inputs, filters, outputs and uploads

PERFORMANCE

On an AWS EC2 instance of size c5.2xlarge, Baker can read zstandard records from S3, uncompress them and apply a basic filtering logic, compressing them back on local files using ~90% of capacity of each vCPU (8 in total) and ~3.5GB of RAM. It reads and writes a total of 94 million records in less than 9 minutes, that’s 178k records per

second.

PIPELINE CONFIGURATION defines a structure of the records with two fields: foo as first element and bar as second. Validation configuration. The section is an optional configuration that contains one or more field names each of which is associated with a regular expression. If the validation section is specified Baker automatically generates a validation function, which checks that each input record CREATE A CUSTOM INPUT COMPONENT The Run function implements the component logic and receives a channel where it sends the raw data it processes.. FreeMem(data *Data) is called by Baker when data is no longer needed. This is an occasion for the input to recycle memory, for example if the input uses a sync.Pool to create new instances of baker.Data.. InputDesc CREATE A CUSTOM OUTPUT COMPONENT The Run function implements the component logic and gets a channel where it receives OutputRecord objects and a channel to communicate to the Upload components what to upload.. CanShard is the function telling whether the output is able to manage sharding. Read the page dedicated to the sharding to go deeper in the topic. Stats is used to report metrics, see the dedicated page.

DOCUMENTATION

A high performance, composable and extendable data-processing pipeline for the big data era CREATE A CUSTOM INPUT COMPONENT The Run function implements the component logic and receives a channel where it sends the raw data it processes.. FreeMem(data *Data) is called by Baker when data is no longer needed. This is an occasion for the input to recycle memory, for example if the input uses a sync.Pool to create new instances of baker.Data.. InputDesc FILTERS | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR A high performance, composable and extendable data-processing pipeline for the big data era

EXPORT METRICS

Baker can publish various kind of metrics that may be used to monitor a pipeline in execution. The metrics exported range from numbers giving an high-level overview of the ongoing pipeline (total processed records, current speed in records per second, etc.) or per-component metrics such as the number of files read or written, to performance statistics published by the Go runtime in

TUNING CONCURRENCY

Baker allows to tune concurrency at various levels of a pipeline: input: Baker configuration doesn’t expose knobs to tune input concurrency as it highly depends on the input source and how the input is implemented filters: Baker runs N concurrent filter chains output: Baker runs M concurrent outputs By default then, Baker processes records concurrently, without any guaranteed order. However LIST | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE Read the API documentation » Input List Overview This input fetches logs from a predefined list of local or remote sources. The “Files” configuration variable is a list of “file specifiers”. Each “file specifier” can be: A local file path on the filesystem: the log file at that path will be processed A HTTP/HTTPS URL: the log file at that URL will be downloaded and processed A S3 CREATE A CUSTOM FILTER COMPONENT Create a custom filter component. Creating a custom filter is probably the most common action a Baker user will perform. In fact, filters are the components that apply the business logic to a Baker pipeline, creating or discarding records or modifying fields. A working example of a custom filter can be found in the filtering example To create a filter and make it available to Baker, one must: SLICE | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR THE Read the API documentation » Filter Slice Overview Slices the source field value using start/end indexes and copies the value to the destination field. If the start index is greater than the field length, Slice sets the destination to an empty string. If the end index is greater than the field length, Slice considers the end index to be equal to the field length. Note: Indexes are 0-based and FORMATTIME | BAKER, HIGH PERFORMANCE MODULAR PIPELINES FOR Read the API documentation » Filter FormatTime Overview This filter formats and converts date/time strings from one format to another. It requires the source and destination field names along with 2 format strings, the first one indicates how to parse the input field while the second how to format it. The source time parsing can fail if the time value does not match the provided format.

SETSTRINGFROMURL

Read the API documentation » Filter SetStringFromURL Overview This filter looks for a set of strings in the URL metadata and sets a field with the found string. Discards the log lines if URL metadata doesn’t contain any of the given strings. On Error: the input record is discarded. Configuration Keys available in the section: Name Type Default Required Description Field

BAKER

* Features

* Docs

* Installing Baker

ENVIRONMENT MANAGEMENT MADE EASY Introducing “Baker”. Create, provision, and share your development environment in minutes. Download Get Started

__

CONFIGURATION AS CODE Create a VM with docker, instantly.

name: docker-test

vm:

ip: 192.168.28.28

services:

- docker

FOCUS ON YOUR IDEA

Simply add a baker.yml file in your SCM and Baker will take care of

the rest.

MANAGED DATA SCIENCE ENVIRONMENTS Easily create a VM for data science.

name: python-nb

vm:

ip: 192.168.88.2

tools:

- jupyter

lang:

- python2

BAKER MAKES CONFIGURATION EASY There are several ways to use baker in your development workflow. Our vision is to enable configurationless software.

*

QUICK & EASY

Baker provides easy tooling for creating complex computing infrastructure with just a few lines of configuration.

*

PACKAGE & SHARE

Baker lets you package vms that can be pulled by your team.

*

CUSTOMIZE & REMIX

Create or re-use bakelets for custom environments.

__

* Features

* Docs

* Download

Copyright © 2018 Ottomatica L.L.C. Share Baker with your friends

__ __ __