Processing Pipelines

Processing pipelines control how a Sigma rule is converted into a query for your SIEM. They let you map field names, point rules at the right index or log stream, and apply environment-specific tweaks — all without touching the original rule.

A pipeline is just a YAML file with an ordered list of transformations. You pass it to sigma convert with -p / --pipeline:

sigma convert -t splunk -p ./pipelines/my_pipeline.yml ./rules/

You can pass several pipelines at once; they run in order of their priority.

Quick Start

The same Sigma rule can be converted for very different backends just by swapping the pipeline. Each recipe below does the two things almost every pipeline needs: tell the backend where the data lives and map a rule field onto the field name your data uses. Notice how the mechanism for "where the data lives" differs per backend, while the field mapping stays the same.

We'll convert this rule throughout. It matches on two fields, EventID and CommandLine:

title: Suspicious Command
logsource:
  category: process_creation
  product: windows
detection:
  selection:
    EventID: 4688
    CommandLine|contains: suspicious_command
  condition: selection

Splunk — build a pipeline step by step

Let's build a Splunk pipeline one transformation at a time, so you can see what each piece adds to the query.

1. Prefix the logs. Splunk scopes a search to data by adding index= and source= terms. The add_condition transformation prepends them to the query:

pipelines/splunk_example.yml

name: Example Splunk Pipeline
priority: 100
transformations:
  # Adds the index and source conditions to the query
  - id: set_index_and_source
    type: add_condition
    conditions: 
      index: windows_logs
      source: WinEventLog:Security

Splunk Output

index="windows_logs" source="WinEventLog:Security" EventID=4688 CommandLine="*suspicious_command*"

2. Map a field. Your index probably doesn't store the field as CommandLine. The field_name_mapping transformation renames it to whatever your data uses — here, Process_Command_Line:

pipelines/splunk_example.yml

name: Example Splunk Pipeline
priority: 100
transformations:
  # Adds the index and source conditions to the query
  - id: set_index_and_source
    type: add_condition
    conditions:
      index: windows_logs
      source: WinEventLog:Security
  # Maps the Sigma field name to the one used in the index
  - id: map_commandline
    type: field_name_mapping
    mapping: 
      CommandLine: Process_Command_Line

Splunk Output

index="windows_logs" source="WinEventLog:Security" EventID=4688 Process_Command_Line="*suspicious_command*"

3. Only apply to the right rules. Right now these transformations run against every rule. Add rule_conditions so they only apply to rules with a matching log source — in this case process_creation on windows. Convert a rule from any other log source and the index, source, and mapping are skipped:

pipelines/splunk_example.yml

name: Example Splunk Pipeline
priority: 100
transformations:
  # Adds the index and source conditions to the query
  - id: set_index_and_source
    type: add_condition
    conditions:
      index: windows_logs
      source: WinEventLog:Security
    # Only apply to rules with this log source
    rule_conditions: 
      - type: logsource
        category: process_creation
        product: windows
  # Maps the Sigma field name to the one used in the index
  - id: map_commandline
    type: field_name_mapping
    mapping:
      CommandLine: Process_Command_Line
    # Only apply to rules with this log source
    rule_conditions: 
      - type: logsource
        category: process_creation
        product: windows

Splunk Output

index="windows_logs" source="WinEventLog:Security" EventID=4688 Process_Command_Line="*suspicious_command*"

Learn more about ConditionsScope transformations to specific log sources, fields, or earlier processing steps with rule, detection, and field conditions.

Elasticsearch ES|QL — set the source index

ES|QL queries begin with FROM <index>, which isn't part of the search expression, so it can't be added with add_condition. Instead, the Elasticsearch backend reads the index from pipeline state under the index key, which you set with set_state. The field mapping works exactly as in the Splunk recipe.

pipelines/esql_example.yml

name: Example ESQL Pipeline
priority: 100
transformations:
  - id: set_index
    type: set_state
    key: index
    val: logs-windows-*
  - id: map_commandline
    type: field_name_mapping
    mapping:
      CommandLine: process.command_line

ES|QL Output

from logs-windows-* metadata _id, _index, _version | where EventID==4688 and process.command_line like "*suspicious_command*"

Grafana Loki — select a log stream

Loki selects data with a {label="value"} stream selector. The Loki backend ships its own set_custom_log_source transformation to build that selector from structured YAML — a good example of a backend extending the pipeline system with its own transformation type. The field mapping is unchanged.

pipelines/loki_example.yml

name: Example Loki Pipeline
priority: 100
transformations:
  - id: set_logsource
    type: set_custom_log_source
    selection:
      job: windows
  - id: map_commandline
    type: field_name_mapping
    mapping:
      CommandLine: process_command_line

Loki Output

{job=`windows`} | json | EventID=4688 and process_command_line=~`(?i).*suspicious_command.*`

Use a predefined pipeline first

Most backends and log sources already have a maintained pipeline you can use directly. List what's installed with sigma list pipelines, and only write your own when you need environment-specific behaviour. The sysmon pipeline, for example, turns generic process_creation rules into Sysmon EventID: 1 queries.

How a Pipeline Works

A pipeline file has a handful of top-level keys, the most important being transformations:

Key	Description
name	(Optional) A human-readable name for the pipeline.
priority	(Optional) The execution priority. Defaults to 0. See Priorities.
vars	(Optional) Variables made available to transformations (e.g. for placeholders).
transformations	The list of transformations applied to rules before conversion.
postprocessing	(Optional) A list of postprocessing steps applied to each query.
finalizers	(Optional) A list of finalizers applied to the full set of queries.
allowed_backends	(Optional) A set of backend identifiers this pipeline may be used with.

Unknown keys are rejected

Any top-level key outside this set causes the pipeline to fail to load with an Unknown keys error. For example, there is no top-level id key.

Each entry under transformations selects a transformation by type, takes an optional id, an optional set of conditions controlling when it applies, and the parameters for that transformation. A typical chain looks like:

1. Generic log sources are translated into specific ones (e.g. process_creation → Sysmon EventID: 1).
2. Field names are mapped into the taxonomy used by the backend.
3. Environment-specific tweaks (indexes, stream selectors, dropped fields) are applied.

Pipeline Usage

Invoking a pipeline is done with -p / --pipeline, which accepts either the name of a predefined pipeline or a path to a .yml file. For example, the predefined sysmon pipeline:

Terminal

sigma convert \
  -t splunk \
  -p sysmon rules/windows/process_creation/proc_creation_win_sysinternals_procdump.yml

Splunk Output

EventID=1 Image IN ("*\\procdump.exe", "*\\procdump64.exe")

You can also supply several pipelines at once, in which case they run in order of their priority (lowest first). For example, you might have a mapping pipeline that maps field names, and a splunk_prefix pipeline that adds index= and source= conditions. If both have the default priority of 0, the order they run in is undefined. Setting priority: 100 on the splunk_prefix pipeline ensures it runs after the mapping:

sigma convert \
  -t splunk \
  -p mapping.yml \
  -p splunk_prefix.yml \
  rules/windows/process_creation/proc_creation_win_sysinternals_procdump.yml

Transformations

Transformations are the building blocks of a pipeline. Each entry under transformations: has a type that selects the transformation, an optional id, an optional set of conditions, and the parameters specific to that transformation.

The available transformation types are:

type	Purpose
field_name_mapping	Map a field name to one or more target field names.
field_name_prefix_mapping	Map a field name prefix to another prefix.
field_name_transform	Transform field names using a (Python-defined) function.
field_name_suffix	Append a suffix to field names.
field_name_prefix	Prepend a prefix to field names.
add_field	Add one or more fields to the rule's field list.
remove_field	Remove one or more fields from the rule's field list.
set_field	Set the rule's field list.
drop_detection_item	Drop matching detection items.
add_condition	Add a condition expression to the rule.
change_logsource	Replace the log source of a rule.
replace_string	Replace values matched by a regular expression.
map_string	Map string values to other string values.
regex	Control how regular expressions are emitted (e.g. case handling).
set_value	Set a detection item value.
convert_type	Convert detection item values between string and number.
set_state	Set a pipeline state variable.
set_custom_attribute	Set a custom attribute on the rule.
hashes_fields	Split a Hashes field into individual hash-algorithm fields.
value_placeholders	Replace placeholders with values from vars.
query_expression_placeholders	Replace placeholders with a query expression.
wildcard_placeholders	Replace remaining placeholders with wildcards.
rule_failure	Abort conversion of the rule with an error message.
detection_item_failure	Abort conversion when a matching detection item is present.
strict_field_mapping_failure	Abort conversion when a field is not covered by a field mapping.
nest	Apply a nested list of transformations as a group.
case	Change the case of detection item values.

The most commonly used transformations are documented in detail below.

Field Name Mapping

Map a field name in the sigma rule to a field name used in your logs.

Parameters:

• 'mapping': the fields that will be mapped (required)

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: useragent_mapping
    type: field_name_mapping
    mapping:
      c-useragent: useragent
      cs-host: hostname
      c-ip: ip
    rule_conditions:
      - type: logsource
        category: proxy

Field Name Prefix Mapping

Map a field name prefix to replace it with another prefix.

Parameters:

• 'mapping': the fields that will be mapped (required)

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: integritylevel_prefix_mapping
    type: field_name_prefix_mapping
    mapping:
      win.: proc.
    rule_conditions:
      - type: logsource
        product: windows

Drop Detection Item

Deletes detection items. Some sort of condition is recommended but not required.

Parameters:

• none

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  # Drops the Hashes field which is specific to Sysmon logs
  - id: hashes_drop_sysmon-specific-field
    type: drop_detection_item
    field_name_conditions:
      - type: include_fields
        fields:
          - Hashes
    rule_conditions:
      - type: logsource
        product: windows
        category: process_creation

Field Name Suffix

Add a field name suffix. field_name_conditions are not required, but are recommended.

Parameters:

• 'suffix': the suffix to be added (required)

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: windows_field_suffix
    type: field_name_suffix
    suffix: ".win"
    field_name_conditions:
      - type: include_fields
        fields:
          - Hashes

Field Name Prefix

Add a field name prefix.

Parameters:

• 'prefix': the prefix to be added (required)

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: windows_field_prefix
    type: field_name_prefix
    prefix: "win."

Wildcard Placeholders

Replaces placeholders with wildcards. This transformation is useful if remaining placeholders should be replaced with something meaningful to make conversion of rules possible without defining the placeholders content.

Parameters:

• none

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: wildcard_placeholders_transformation
    type: wildcard_placeholders

Value Placeholders

Replaces placeholders with values contained in variables defined in the configuration.

Parameters:

• include: identify the specific placeholders you'd like to transform

/pipelines/value_placeholders_test.yml

name: value_placeholder_pipeline
vars:
  administrator_name:
    - "Administrator"
    - "Admin"
    - "SysAdmin"
transformations:
  - type: value_placeholders
    include:
      - "administrator_name"

/rules/rule.yml

title: Administrator Usage
logsource:
  product: windows
detection:
  selection:
    user|expand: "%administrator_name%"
  condition: selection

Splunk Output

user IN ("Administrator", "Admin", "SysAdmin")

Query Expression Placeholders

Replaces a placeholder with a plain query containing the placeholder or an identifier mapped from the placeholder name. The main purpose is the generation of arbitrary list lookup expressions which are passed to the resulting query.

Parameters:

• expression: string that contains query expression with {field} and {id} placeholder where placeholder identifier or a mapped identifier is inserted.
• include:identify the specific placeholders you'd like to transform
• mapping: Mapping between placeholders and identifiers that should be used in the expression. If no mapping is provided the placeholder name is used.

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: Admins_Workstations_query_expression_placeholder
    type: query_expression_placeholders
    include:
      - Admins_Workstations
    expression: "[| inputlookup {id} | rename user as {field}]"
#  For the rule: [User with Privileges Logon](https://github.com/SigmaHQ/sigma/blob/e1a713d264ac072bb76b5c4e5f41315a015d3f41/rules-placeholder/windows/builtin/security/win_security_admin_logon.yml#L30)

Splunk Output

EventID IN (4672, 4964) NOT (SubjectUserSid="S-1-5-18" OR [| inputlookup Admins_Workstations | rename user as SubjectUserName])

Add Condition

Adds an AND condition expression to the rule's conditions. This is most often used to add an index, source type, or other environment-specific constraint.

If template is set to true, the condition values are interpreted as string templates and the following placeholders are replaced:

• category, product and service: with the corresponding values of the Sigma rule log source.

Parameters:

• 'conditions': a mapping of field names to values that will be added to the rule's condition.
• 'template': (Optional) when true, interpret the condition values as templates. Defaults to false.
• 'negated': (Optional) when true, the added condition is negated (e.g. NOT index="..."). Defaults to false.

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: index_condition
    type: add_condition
    conditions:
      index: winevent
    rule_conditions:
      - type: logsource
        product: windows

Change Logsource

Replace log source as defined in transformation parameters.

Parameters:

• 'category', 'product', 'service': the log source to be changed (requires at least one)

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: change_logsource
    type: change_logsource
    category: security
    rule_conditions:
      - type: logsource
        category: process_creation

Replace String

Replace string part matched by regular expression with replacement string that can reference capture groups. It operates on the plain string representation of the SigmaString value.

This is basically an interface to re.sub() and can use all features available there.

Parameters:

• regex: The regular expression to find the desired string
• replacement: The replacement string to be added

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: image_file_only
    type: replace_string
    regex: "^\\*\\\\([^\\\\]+)$"
    replacement: "\\1"
    field_name_conditions:
      - type: include_fields
        fields:
          - Image

Set State

Stores a key/value in the pipeline state — a shared bag of values that later transformations, conditions, templates, and the backend itself can read. Unlike most transformations, set_state doesn't change the rule directly; it leaves a value behind for something else to act on.

This is how several backends receive configuration that isn't part of the query expression. For example, the Elasticsearch ES|QL recipe sets the index key so the backend knows which index to put in the FROM clause, and the Splunk backend reads data_model_set to build tstats data-model queries.

Parameters:

• key: the state key to set.
• val: the value to assign.

pipelines/esql_index.yml

name: esql_index
priority: 100
transformations:
  - id: set_index
    type: set_state
    key: index
    val: logs-windows-*
    rule_conditions:
      - type: logsource
        product: windows

Backend-specific example: Splunk data models

Setting data_model_set lets the Splunk backend emit a tstats query when used with the data_model output format (sigma convert -p pipeline.yml -t splunk -f data_model rule.yml):

transformations:
  - id: set_datamodel
    type: set_state
    key: "data_model_set"
    val: "Endpoint.Processes"

| tstats summariesonly=false allow_old_summaries=true fillnull_value="null" count min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes where ... by CommandLine Image OriginalFilename | `drop_dm_object_name(Processes)`

Rule Failure

Raise a SigmaTransformationError with the provided message. This enables transformation pipelines to signalize that a certain situation can't be handled, e.g. only a subset of values is allowed because the target data model doesn't offers all possibilities. A transformation condition is not required, but is recommended.

Parameters:

• message: the message to present when the transformation failure state is met

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: cs_drop_unsupported_logsource_sysmon_status
    type: rule_failure
    message: CrowdStrike logs do not support sysmon_status logs at this time.
    rule_conditions:
      - type: logsource
        product: windows
        category: sysmon_status

Detection Item Failure

Raise a SigmaTransformationError with the provided message. This enables transformation pipelines to signalize that a certain situation can't be handled, e.g. only a subset of values is allowed because the target data model doesn't offers all possibilities. A transformation condition is not required, but is recommended.

Parameters:

• message: the message to present when the transformation failure state is met

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: cs_drop_eventid
    type: detection_item_failure
    message: CrowdStrike logs do not support the field EventID at this time.
    field_name_conditions:
      - type: include_fields
        fields:
          - EventID
    rule_conditions:
      - type: logsource
        product: windows

Map String

Replace whole string values with one or more other values. Unlike replace_string, this matches the entire value rather than a regular expression substring, and a single value can be mapped to a list of values.

Parameters:

• mapping: a mapping of source values to their replacement value(s).

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: severity_map
    type: map_string
    mapping:
      Informational: Info
      Critical:
        - Critical
        - High
    field_name_conditions:
      - type: include_fields
        fields:
          - Severity

Set Value

Set a detection item to a fixed value, regardless of its original value. force_type can be used to coerce the value to a string or number.

Parameters:

• value: the value to set (string, number, or boolean).
• force_type: (Optional) force the value's type, either str or num.

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: set_eventid
    type: set_value
    value: 4625
    field_name_conditions:
      - type: include_fields
        fields:
          - EventID

Convert Type

Convert detection item values between strings and numbers. This is useful when your SIEM expects a specific type for a field.

Parameters:

• target_type: the type to convert to, either str or num.

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: port_to_string
    type: convert_type
    target_type: str
    field_name_conditions:
      - type: include_fields
        fields:
          - DestinationPort

Case

Change the case of detection item values.

Parameters:

• method: one of lower, upper, or snake_case. Defaults to lower.

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: lowercase_user
    type: case
    method: lower
    field_name_conditions:
      - type: include_fields
        fields:
          - User

Regex

Control how regular expressions are emitted in the resulting query. This is mainly useful for handling case-insensitive matching in backends that expect a particular regex dialect.

Parameters:

• method: one of plain, ignore_case_flag, or ignore_case_brackets. Defaults to ignore_case_brackets.

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: regex_ci
    type: regex
    method: ignore_case_flag

Hashes Fields

Split a combined Hashes field (as produced by Sysmon) into separate fields per hash algorithm, so each algorithm can be matched on its own field.

Parameters:

• valid_hash_algos: the list of hash algorithms to extract (e.g. MD5, SHA1, SHA256).
• field_prefix: (Optional) a prefix to prepend to the generated field names. Defaults to "".
• drop_algo_prefix: (Optional) when true, drop the algorithm prefix from the resulting field name. Defaults to false.

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: split_hashes
    type: hashes_fields
    valid_hash_algos:
      - MD5
      - SHA1
      - SHA256
    field_name_conditions:
      - type: include_fields
        fields:
          - Hashes

Add Field / Remove Field / Set Field

These transformations modify the rule's field list (the list of fields a backend may emit alongside the query, e.g. in a table or saved search), rather than the detection logic itself.

• add_field — add one or more fields. Parameter: field (string or list).
• remove_field — remove one or more fields. Parameter: field (string or list).
• set_field — replace the field list entirely. Parameter: fields (list).

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: add_output_fields
    type: add_field
    field:
      - User
      - ComputerName

Set Custom Attribute

Set an arbitrary custom attribute on the rule. This is useful for carrying metadata through to an output format or finalizer.

Parameters:

• attribute: the attribute name to set.
• value: the value to assign.

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: tag_team
    type: set_custom_attribute
    attribute: team
    value: detection-engineering

Strict Field Mapping Failure

Raise a SigmaTransformationError when a field is encountered that is not covered by a field mapping. This is useful to enforce that every field in a rule has an explicit mapping for a given target, catching unmapped fields at conversion time.

Parameters:

• none

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: enforce_mapping
    type: strict_field_mapping_failure
    field_name_conditions:
      - type: processing_item_applied
        processing_item_id: field_mapping
    field_name_cond_not: true

Nest

Apply a list of transformations as a single grouped unit. The nested transformations share the conditions defined on the nest item, which is handy for applying the same set of conditions to several transformations without repeating them.

Parameters:

• items: a list of nested transformation items (each with its own type).

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: windows_group
    type: nest
    items:
      - id: prefix
        type: field_name_prefix
        prefix: "win."
      - id: suffix
        type: field_name_suffix
        suffix: ".keyword"
    rule_conditions:
      - type: logsource
        product: windows

Backend-Specific Transformations

Backends may register their own transformation types on top of the core set above. These only work when converting with that backend, but are documented in their respective projects.

Custom Pipelines: Set Custom Log Source

The Grafana Loki backend adds a set_custom_log_source transformation that builds a Loki stream selector ({label="value"}) from structured YAML. It's a good illustration of how a backend can extend the pipeline system with behaviour the core can't express.

Parameters:

• selection: a mapping of labels to values used to build the stream selector.
• template: (Optional) when true, category/product/service placeholders are substituted from the rule's log source.

pipelines/loki_custom.yml

name: loki_custom
priority: 100
transformations:
  - id: loki_custom_mapping
    type: set_custom_log_source
    selection:
      job: job-name
      app: loki
    rule_conditions:
      - type: logsource
        category: process_creation
        product: windows

Loki Output

{job=`job-name`,app=`loki`} | json | EventID=4688 and CommandLine=~`(?i).*suspicious_command.*`

Priorities

During conversion, after all the backend and user-supplied pipelines have been gathered, priorities are used to sort the pipelines into their order-of-execution.

Some standard conventions used for these priorities are listed below.

Priority	Description
10	Log source pipelines like for Sysmon
20	Pipelines provided by backend packages that should be run before the backend pipeline.
50	Backend pipelines that are integrated in the backend and applied automatically.
60	Backend output format pipelines that are integrated in the backend and applied automatically for the associated output format.

Tip:

Pipelines are executed from the lowest priority to the highest.

Note:

• Pipelines with the same priority are applied in the order they were provided.
• Pipelines without a priority are assumed to have the Priority 0.

Pipeline Variables and State

Pipelines can define variables in the top-level vars section. These are most commonly consumed by the placeholder transformations to inject predefined values into rules at conversion time.

vars:
  admin_users:
    - "Administrator"
    - "Admin"
transformations:
  - type: value_placeholders
    include:
      - "admin_users"

In addition to vars, pipelines maintain a state that can be read and modified during processing through the set_state transformation. The state is used by some backends (for example, the Splunk backend reads data_model_set when generating tstats queries) and is also exposed to templates in postprocessing and finalizers via the pipeline object.

Placeholders

Placeholders let a rule defer part of its value definition to the pipeline. A rule marks a value as a placeholder using the |expand modifier (e.g. user|expand: "%admin_users%"), and a placeholder transformation then resolves it during conversion.

Three placeholder transformations are available, each documented in detail under Transformations:

• value_placeholders — replace placeholders with values supplied in the pipeline vars.
• query_expression_placeholders — replace placeholders with a query expression, typically a list lookup.
• wildcard_placeholders — replace any remaining (undefined) placeholders with wildcards, so rules can convert without defining every placeholder's content.

The most common case — injecting a list of values defined in vars — looks like this:

vars:
  admin_users:
    - "Administrator"
    - "Admin"
transformations:
  - type: value_placeholders
    include:
      - "admin_users"

Postprocessing

While transformations operate on the Sigma rule before conversion, query postprocessing operates on the converted query string for each individual rule. Postprocessing items are defined under the top-level postprocessing: key and, like transformations, support the same conditions (rule_conditions, etc.).

The available postprocessing types are:

type	Purpose
embed	Wrap the query in a prefix and/or suffix.
simple_template	Insert the query into a Python str.format() template.
template	Insert the query into a Jinja2 template.
json	Embed the query into a JSON template by replacing a %QUERY% placeholder.
replace	Replace parts of the query matched by a regular expression.
nest	Apply a nested list of postprocessing items.

Embed Query

Wrap each query in a prefix and/or suffix string.

Parameters:

• prefix: (Optional) text to prepend. Defaults to "".
• suffix: (Optional) text to append. Defaults to "".

/pipelines/postprocessing_demo.yml

name: postprocessing_demo
priority: 100
postprocessing:
  - type: embed
    prefix: "search "
    suffix: " | head 100"

Simple Template

Insert the query into a Python str.format() template. The available placeholders are {query}, {rule}, and {pipeline}.

Parameters:

• template: the template string.

/pipelines/postprocessing_demo.yml

name: postprocessing_demo
priority: 100
postprocessing:
  - type: simple_template
    template: |
      [{rule.title}]
      {query}

Template (Jinja2)

Insert the query into a Jinja2 template. The available context variables are query, rule, and pipeline.

Parameters:

• template: the Jinja2 template string (or, when path is set, a relative path to a template file).

/pipelines/postprocessing_demo.yml

name: postprocessing_demo
priority: 100
postprocessing:
  - type: template
    template: |
      {
        "name": {{ rule.title | tojson }},
        "query": {{ query | tojson }}
      }

JSON

Embed the query into a JSON template by replacing the literal placeholder %QUERY% with the query string.

Parameters:

• json_template: the JSON template containing %QUERY%.

/pipelines/postprocessing_demo.yml

name: postprocessing_demo
priority: 100
postprocessing:
  - type: json
    json_template: '{"query": "%QUERY%"}'

Replace

Replace parts of the query matched by a regular expression.

Parameters:

• pattern: the regular expression to find.
• replacement: the replacement string.

/pipelines/postprocessing_demo.yml

name: postprocessing_demo
priority: 100
postprocessing:
  - type: replace
    pattern: "EventCode="
    replacement: "EventID="

Finalizers

While postprocessing acts on each query individually, finalizers act on the complete list of all generated queries once conversion is finished. They are defined under the top-level finalizers: key.

The available finalizer types are:

type	Purpose
concat	Concatenate all queries into a single string.
json	Serialise the list of queries as JSON.
yaml	Serialise the list of queries as YAML.
template	Render the list of queries through a Jinja2 template.
nested	Apply a list of finalizers in sequence.

Concatenate Queries

Join all queries with a separator and optionally wrap the result in a prefix and suffix.

Parameters:

• separator: (Optional) string placed between queries. Defaults to a newline.
• prefix: (Optional) string placed before the result. Defaults to "".
• suffix: (Optional) string placed after the result. Defaults to "".

finalizers:
  - type: concat
    separator: " OR "
    prefix: "("
    suffix: ")"

JSON Output

Serialise the list of queries as a JSON document.

Parameters:

• indent: (Optional) number of spaces to indent. Defaults to null (compact output).

finalizers:
  - type: json
    indent: 2

YAML Output

Serialise the list of queries as a YAML document.

Parameters:

• indent: (Optional) number of spaces to indent.

finalizers:
  - type: yaml
    indent: 2

Template Output

Apply a Jinja2 template to the list of queries. The following variables are available in the template context:

• queries: the final query output as a list.
• pipeline: all the context provided to the processing pipeline, including pipeline.state.

finalizers:
  - type: template
    template: |
      {
        "query": {{ queries[0] | tojson }}
      }

Nested Finalizer

Apply a list of finalizers in sequence, where the output of one becomes the input of the next.

Parameters:

• finalizers: a list of finalizer items (each with its own type).

finalizers:
  - type: nested
    finalizers:
      - type: concat
        separator: "\n"
      - type: template
        template: "{{ queries[0] }}"

Conditions

Conditions can be attached transformations, so that a transformation may only trigger when a given logsource is present, or only if another transformation was applied previously during the conversion process. Conditions also support a conditional logic system implemented in PySigma version v0.11.19 that perform similar to how the detection logic in a Sigma rule itself works allowing you to assign an identifier to a transformation and perform logical operations against those identifiers.

/pipelines/m365.yml

name: m365
priority: 20
transformations:
  - id: atp_index
    type: add_condition
    conditions:
      index: microsoft_atp
    rule_conditions:
      - type: logsource
        product: m365
        service: threat_detection
  - id: defender_index
    type: add_condition
    conditions:
      index: "microsoft_defender"
    rule_conditions:
      logsource_cond:
        type: logsource
        product: m365
      logsource_cond2:
        type: processing_item_applied
        processing_item_id: atp_index
    rule_cond_expr: logsource_cond and not logsource_cond2

Combining Conditions

Each transformation can carry up to three groups of conditions, evaluated against different parts of the rule:

• rule_conditions — matched against the rule as a whole (rule-based conditions).
• detection_item_conditions — matched against individual detection items (detection-based conditions).
• field_name_conditions — matched against field names (field-based conditions).

By default, all conditions within a group must match. This can be controlled per group:

Key	Description
rule_cond_op	How rule_conditions are combined: and (default) or or.
rule_cond_not	When true, negate the result of the rule_conditions group. Defaults to false.
rule_cond_expr	A boolean expression referencing named conditions (mutually exclusive with rule_cond_op).
detection_item_cond_op	As above, for detection_item_conditions.
detection_item_cond_not	As above, for detection_item_conditions.
detection_item_cond_expr	As above, for detection_item_conditions.
field_name_cond_op	As above, for field_name_conditions.
field_name_cond_not	As above, for field_name_conditions.
field_name_cond_expr	As above, for field_name_conditions.

When using a *_cond_expr expression (as in the m365.yml example above), the conditions in that group must be supplied as a mapping of identifiers to conditions, rather than a list, so each can be referenced by name in the expression.

Rule-based Conditions

Identifier
logsource
contains_detection_item
contains_field
processing_item_applied
processing_state
is_sigma_rule
is_sigma_correlation_rule
rule_attribute
tag

logsource

Matches log source on rule. Not specified log source fields are ignored. For Correlation rules, the condition returns true if any of the associated rules have the required log source fields.

Parameters:

• 'category': the category field in the sigma rule logsource section
• 'product': the product field in the sigma rule logsource section
• 'service': the service field in the sigma rule logsource section

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: atp_index
    type: add_condition
    conditions:
      index: microsoft_atp
    rule_conditions:
      - type: logsource
        product: m365
        service: threat_detection

contains_detection_item

Returns True if rule contains a detection item that matches the given field name and value.

Parameters:

• 'field': The field you'd like to match on.
• 'value': The value you'd like to match on.

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: atp_index
    type: add_condition
    conditions:
      index: microsoft_atp
    rule_conditions:
      - type: contains_detection_item
        field: Severity
        value: Informational

contains_field

Returns True if the rule contains a detection item that uses the given field name, regardless of its value.

Parameters:

• 'field': The field name you'd like to match on.

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: atp_index
    type: add_condition
    conditions:
      index: microsoft_atp
    rule_conditions:
      - type: contains_field
        field: CommandLine

processing_item_applied

Checks whether an earlier processing item (referenced by its id) was applied. Useful for chaining transformations — e.g. only adding a prefix to fields that a previous mapping step did not touch.

Parameters:

• processing_item_id: the identifier of the processing item to match on.

rule_conditions:
  - type: processing_item_applied
    processing_item_id: field_mapping

processing_state

Matches against a value stored in the pipeline state (see set_state).

Parameters:

• key: the state key.
• val: the value to match.

rule_conditions:
  - type: processing_state
    key: backend
    val: splunk

Available in every condition group

processing_item_applied and processing_state work identically as rule_conditions, detection_item_conditions, or field_name_conditions — only the context they evaluate against differs.

is_sigma_rule

Checks if rule is a SigmaRule.

Parameters:

• N/A

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: atp_index
    type: add_condition
    conditions:
      index: microsoft_atp
    rule_conditions:
      - type: is_sigma_rule

is_sigma_correlation_rule

Checks if rule is a SigmaRule.

Parameters:

• N/A

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: atp_index
    type: add_condition
    conditions:
      index: microsoft_atp
    rule_conditions:
      - type: is_sigma_correlation_rule

rule_attribute

Generic match on rule attributes with supported types:

• strings (exact matches)
• UUIDs (exact matches)
• numbers (relations: eq, ne, gte, ge, lte, le)
• dates (relations: eq, ne, gte, ge, lte, le)
• Rule severity levels (relations: eq, ne, gte, ge, lte, le)
• Rule statuses (relations: eq, ne, gte, ge, lte, le)
• Fields that contain lists of values, maps or other complex data structures are not supported and raise a SigmaConfigurationError. If the type of the value doesn’t allows a particular relation, the condition also raises a SigmaConfigurationError on match.

Parameters:

• 'attribute': The attribute to match on.
• 'value': The value to match on.
• 'op': The relational comparison type to match on. One of eq (equals), ne (not equals), gte (greater than or equals), gt (greater than), lte (less than or equals), lt (less than), in (value is in a list) or not_in (value is not in a list). Defaults to eq.

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: atp_index
    type: add_condition
    conditions:
      index: microsoft_atp
    rule_conditions:
      - type: rule_attribute
        attribute: date
        value: "2025-01-01"
        op: gte

tag

Matches if rule is tagged with a specific tag.

Parameters:

• 'tag': The tag to match on.

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: atp_index
    type: add_condition
    conditions:
      index: microsoft_atp
    rule_conditions:
      - type: tag
        tag: attack.discovery

Detection-based Conditions

Identifier
match_string
match_value
contains_wildcard
is_null
processing_item_applied
processing_state

match_string

Match string values with a regular expression ‘pattern’. The parameter ‘cond’ determines for detection items with multiple values if any or all strings must match. Generally, values which aren’t strings are skipped in any mode or result in a false result in all match mode.

Parameters:

• 'cond': 'any' or 'all'
• 'pattern': The pattern to match on
• 'negate': Default to false, but can be changed to True to make a negated condition

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: atp_index
    type: add_condition
    conditions:
      index: microsoft_atp
    detection_item_conditions:
      - type: match_string
        cond: any
        pattern: informational
        negate: False

match_value

Match detection item values against a plain value (no regular expression). The parameter ‘cond’ determines, for detection items with multiple values, whether any or all values must match.

Parameters:

• 'cond': 'any' or 'all'
• 'value': The value to match on (string, number or boolean).

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: atp_index
    type: add_condition
    conditions:
      index: microsoft_atp
    detection_item_conditions:
      - type: match_value
        cond: any
        value: Informational

contains_wildcard

Matches if a detection item value contains a wildcard character. The parameter ‘cond’ determines, for detection items with multiple values, whether any or all values must contain a wildcard.

Parameters:

• 'cond': 'any' or 'all'

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: atp_index
    type: add_condition
    conditions:
      index: microsoft_atp
    detection_item_conditions:
      - type: contains_wildcard
        cond: any

is_null

Match null values. The parameter ‘cond’ determines for detection items with multiple values if any or all strings must match. Generally, values which aren’t strings are skipped in any mode or result in a false result in all match mode.

Parameters:

• 'cond': 'any' or 'all'

/pipelines/transformation_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: atp_index
    type: add_condition
    conditions:
      index: microsoft_atp
    detection_item_conditions:
      - type: is_null
        cond: any

processing_item_applied / processing_state

Also available here as detection-item conditions — see the shared description above. Use them under detection_item_conditions: instead of rule_conditions:.

Field-based Conditions

Identifier
include_fields
exclude_fields
processing_item_applied
processing_state

include_fields

Matches on field name if it is contained in fields list. The parameter ‘mode’ determines if field names are matched as plain string (“plain”) or regular expressions (“re”).

Parameters:

• 'fields': The fields to match on
• 'mode': Plain match or regex match using 'plain' or 're'. Defaults to 'plain'.

/pipelines/include_fields_demo.yml

name: include_fields_demo
priority: 100
transformations:
  - id: atp_index
    type: drop_detection_item
    field_name_conditions:
      - type: include_fields
        fields:
          - name
          - type

exclude_fields

Matches on field name if it is not contained in the fields list.

Parameters:

• 'fields': The fields to match on
• 'mode': Plain match or regex match using 'plain' or 're'. Defaults to 'plain'.

/pipelines/exclude_fields_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: atp_index
    type: drop_detection_item
    field_name_conditions:
      - type: exclude_fields
        fields:
          - name
          - value

processing_item_applied / processing_state

Also available here as field-name conditions — see the shared description above. Use them under field_name_conditions: instead of rule_conditions:.

Field-based processing_item_applied

Checks if processing item was applied to detection item.

Parameters:

• 'processing_item_id': The identifier of the processing item you'd like to match on

/pipelines/processing_item_applied_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: test_id
    type: add_condition
    conditions:
      hello: world
  - id: atp_index
    type: add_condition
    conditions:
      index: microsoft_atp
    field_name_conditions:
      - type: processing_item_applied
        processing_item_id: test_id

Field-based processing_state

Matches on processing pipeline state.

Parameters:

• 'key': The key for the processing state.
• 'val': The value for the processing state key.

/pipelines/processing_state_demo.yml

name: transformation_demo
priority: 100
transformations:
  - id: atp_index
    type: add_condition
    conditions:
      index: microsoft_atp
    field_name_conditions:
      - type: processing_state
        key: key-test
        val: val-test