pSConfig Template Variables¶

Template Variable Basics¶

Syntax¶

Template variables are strings included in the JSON of a template that access properties of task components. They take the following form where VARIABLE is the variable being used:

{% VARIABLE %}

Expansion is the process by which a variable is converted from the form above to the actual value it represents. Once expansion completes a variable is referred to as expanded. The following template sections support the expansion of template variables:

archives
contexts
tests
reference object of tasks

Note

Sections such as addresses, groups, hosts and fields outside the reference section in tasks DO NOT support template variables and values will not be expanded in those sections if used.

Determining JSON Type of Expanded Variables¶

When a template variable is used, it must be enclosed within quotes so that a pre-expansion template is still valid JSON. Once expanded, whether the expanded variable is still a JSON string or some other JSON type, depends on two factors:

The variable used
Whether the variable is used standalone or embedded within a string

A standalone use of the variable is when you use just the variable surrounded by quotes. For example:

{
    "example": "{% flip %}"
}

In this example, since the flip variable is a boolean type and we have used it in standalone mode, the object will be expanded to the following (assuming flip evaluates to true):

{
    "example": true
}

Notice that the surrounding double quotes are removed so it is a JSON boolean instead of a string. If we were to use a different variable, like scheduled_by_address that always evaluates to a string, the double quotes will remain intact. For example, consider the following template:

{
    "example": "{% scheduled_by_address %}"
}

If we expand that variable we get the following (assuming 10.0.0.1 is the address to which it evaluates):

{
    "example": "10.0.0.1"
}

Notice that since the variable is a string, the quotes are preserved.

pSConfig agents can also expand variables embedded within an existing string. For example if we want to build a URL with a dynamic host portion that uses scheduled_by_address we can do the following:

{
    "example": "https://{% scheduled_by_address %}/example/url"
}

This would expand to:

{
    "example": "https://10.0.0.1/example/url"
}

Placement in Objects¶

For those objects that support template variables, they can be nested as deeply within the object as needed. For example, all the uses of template variables below are completely valid:

{
    "example": "{% scheduled_by_address %}",
    "nest1": {
        "example": "{% scheduled_by_address %} is nested one level",
        "nest2": {
            example": "{% scheduled_by_address %} is nested two levels"
        }
    }
}

Assuming scheduled_by_address still evaluates to 10.0.0.1, the expanded object would look as follows:

{
    "example": "10.0.0.1",
    "nest1": {
        "example": "10.0.0.1 is nested one level",
        "nest2": {
            example": "10.0.0.1 is nested two levels"
        }
    }
}

`address` Variable¶

Description¶

This variable accesses the address property of an address object in the list generated by a group for an individual task.

Note

See Introduction to pSConfig Templates for a discussion on how groups work and how they generate address lists.

The address template variable takes the following form:

{% address[INDEX] %}

The INDEX between square brackets indicates which address object in the generated list to access starting at 0. Since groups of type mesh and disjoint always return pairs of two addresses, the INDEX can be value 0 or 1. For groups of type list it must be 0 since that type only generates one address. This variable always returns a string and will take the form of an IP address or hostname.

Examples¶

The address template variable is commonly used for test specifications that have source and/or dest parameters. Below is an example of a throughput test using this variable to set the source to the first address in the pair and the dest to the second address. Assuming that the first address in the input pair is 10.0.0.1 and the second is 10.0.0.2 the template and expansion look as follows:

Template:

"throughput_test": {
    "type": "throughput",
    "spec": {
        "source": "{% address[0] %}",
        "dest": "{% address[1] %}",
    }
}

Post-Expansion:

"throughput_test": {
    "type": "throughput",
    "spec": {
        "source": "10.0.0.1",
        "dest": "10.0.0.2",
    }
}

Likewise, this variable is useful for archive objects such as those of type http where the URL is to an archive running on one of the endpoints. For example, if paired with the test specification above, this would always store the result on the host that is the source of the test. The template and expanded values are shown below:

Template:

"source_archive": {
     "archiver": "http",
     "data": {
         "_url": "https://{% address[0] %}/logstash"
     }
}

Post-Expansion:

"source_archive": {
     "archiver": "http",
     "data": {
         "_url": "https://10.0.0.1/logstash"
     }
}

`flip` Variable¶

Description¶

The flip variable expands a boolean value that is true if the task will be scheduled by an agent other than the one representing either the first address in the list or the host indicated by the scheduled_by parameter of the task object if specified. What this usually means is that if the first address in a pair has no-agent enabled then the second address will have to schedule it, thus setting flip to true. Otherwise flip will be false.

The flip template variable takes the following form:

{% flip %}

The return type of this is a JSON boolean meaning if used as a standalone variable the surrounding double quotes will be removed.

Examples¶

The flip variable is commonly used with the aptly-named flip option of latency and latencybg test specifications. In fact, it is probably a good idea to set this for all tests of this type. What it does is allow these tests to work even if the the address set to the source has no-agent enabled. Assuming that the first address in the input pair is 10.0.0.1 , the second is 10.0.0.2 and the address object of 10.0.0.1 has no-agent enabled, the template and expansion look as follows:

Template:

"latencybg_test": {
    "type": "latencybg",
    "spec": {
        "source": "{% address[0] %}",
        "dest": "{% address[1] %}",
        "flip": "{% flip %}"
    }
}

Post-Expansion:

"latencybg_test": {
    "type": "latencybg",
    "spec": {
        "source": "10.0.0.1",
        "dest": "10.0.0.2",
        "flip": true
    }
}

`jq` Variable¶

Description¶

This variable accepts a jq script that it performs against the JSON of a task and its components. This is by far the most flexible and powerful of the template variables. In fact, almost all the other template variables could be implemented using the jq template variable instead.

The jq template variable takes the following form:

{% jq SCRIPT %}

The SCRIPT is the jq to be executed. The JSON object on which the jq operates is a special form of the template itself, including only those elements relating to the individual task being generated. Specifically, the JSON being queried has the following sections:

addresses which is an array of the address object for this particular task. This means a variable like {% jq .addresses[0].address %} is the exact equivalent of the {% address[0] %} template variable (see address Variable).
archives is an array of the archive objects to be used for this task. A query such as {% jq .archives[0].archiver %} returns the type of the first archiver in the list. The order is not guaranteed, so plan scripts accordingly if using more than one archiver for a task.
contexts is a two-dimensional array of the context objects to be used for this task. The index at the first level of the array corresponds with the index of the address object that is associated with the context objects in the list at the second level. A query such as {% jq .contexts[0] %} returns an array of all the context objects associated with the first address object. The order within the second level of the context list is not guaranteed, so plan scripts accordingly if using more than one context for an address. If an address has no contexts, then the array will be empty.
hosts is an array of the host objects to be used for this task. The index of a host maps to the address to which that host belongs. If an address does not belong to a host, an empty object will be at that position. A script such as {% jq .host[0].tags %} returns the tags of the host associated with the first address.
task is the task object exactly as defined in the template. A script such as {% jq .task.tools %} returns the tools array of the task.
test is the test object exactly as defined in the template. A script such as {% jq .test.type %} returns the type of the task.

The JSON type returned by jq template variables depends on your query. It is important to keep this in mind when using these variables because the cost of the extra flexibility means unexpected things can happen if one is not careful.

Examples¶

Let’s take a look at an example where we have the following template using the jq template variable in the reference section of a task object:

{
    "addresses": {
        "thr1": {
            "address": "thr1.perfsonar.net",
            "_meta": {
                "ifspeed": 10
            }
        },
        "thr2": {
            "address": "thr2.perfsonar.net",
            "_meta": {
                "ifspeed": 1
            }
        }
    },
    "groups": {
        "throughput_group": {
            "type": "mesh",
            "addresses": [
                 {"name": "thr1"},
                 {"name": "thr2"}
             ]
        }
    },
    "tests": {
        "throughput_test": {
            "type": "throughput",
            "spec": {
                "source": "{% address[0] %}",
                "dest": "{% address[1] %}"
            }
        }
    },
    "tasks": {
        "throughput_task": {
            "group": "throughput_group",
            "test": "throughput_test",
            "reference": {
                "source_ifspeed": "{% jq .addresses[0]._meta.ifspeed %}",
                "dest_ifspeed": "{% jq .addresses[1]._meta.ifspeed %}"
            }
        }
    }
}

In the example, each address object has a custom _meta property intended to indicate the speed of the network interface that the address object represents. Those fields will be included in a reference section of the task meaning it will be included in an informational section of the same name in the corresponding pScheduler task. Someone debugging pScheduler will be able to see these fields and may find it of use. Let’s take a closer look at how these will be expanded so we better understand their meaning.

First, we have a group of type mesh containing two members. This will generate the following address pairs:

thr1.perfsonar.net, thr2.perfsonar.net
thr2.perfsonar.net, thr1.perfsonar.net

For the first address pair, the JSON generated against which we can run our jq script is shown below:

{
   "addresses":[
      {
         "address":"thr1.perfsonar.net",
         "_meta":{
            "ifspeed":10
         }
      },
      {
         "address":"thr2.perfsonar.net",
         "_meta":{
            "ifspeed":1
         }
      }
   ],
   "archives": [],
   "contexts": [ [], [] ],
   "hosts": [ {}, {} ],
   "test":{
      "type":"throughput",
      "spec":{
         "source":"{% address[0] %}",
         "dest":"{% address[1] %}"
      }
   },
   "task":{
      "group":"throughput_group",
      "test":"throughput_test",
      "reference":{
         "source_ifspeed":"{% jq .addresses[0]._meta.ifspeed %}",
         "dest_ifspeed":"{% jq .addresses[1]._meta.ifspeed %}"
      }
   }
}

Our variables {% jq .addresses[0]._meta.ifspeed %} and {% jq .addresses[1]._meta.ifspeed %} are standalone queries selecting a JSON integer from each of the address object in the list. That means the quotes will get dropped in the result. This yields the following expanded task:

{
    "group":"throughput_group",
    "test":"throughput_test",
    "reference":{
        "source_ifspeed":10,
        "dest_ifspeed":1
    }
}

For the second address pair, we get the following JSON against which we can run our jq script:

{
   "addresses":[
      {
         "address":"thr2.perfsonar.net",
         "_meta":{
            "ifspeed":1
         }
      },
      {
         "address":"thr1.perfsonar.net",
         "_meta":{
            "ifspeed":10
         }
      }
   ],
   "archives": [],
   "contexts": [ [], [] ],
   "hosts": [ {}, {} ],
   "test":{
      "type":"throughput",
      "spec":{
         "source":"{% address[0] %}",
         "dest":"{% address[1] %}"
      }
   },
   "task":{
      "group":"throughput_group",
      "test":"throughput_test",
      "reference":{
         "source_ifspeed":"{% jq .addresses[0]._meta.ifspeed %}",
         "dest_ifspeed":"{% jq ,addresses[1]._meta.ifspeed %}"
      }
   }
}

This results in the following expanded task:

{
    "group":"throughput_group",
    "test":"throughput_test",
    "reference":{
        "source_ifspeed":1,
        "dest_ifspeed":10
    }
}

Note

For more information on jq, see the official jq documentation or this video tutorial.

`lead_bind_address` Variable¶

Description¶

This variable accesses the lead-bind-address property of an address object in the list generated by a group for an individual task. It will additionally fallback to the value of the address property if lead-bind-address is not set.

Note

See Introduction to pSConfig Templates for a discussion on how groups work and how they generate address lists.

Note

See Controlling pScheduler Server Binding for a discussion of the lead-bind-address property’s meaning

The lead_bind_address template variable takes the following form:

{% lead_bind_address[INDEX] %}

The INDEX between square brackets indicates which address object in the generated list to access starting at 0. Since groups of type mesh and disjoint always return pairs of two addresses, the INDEX can be value 0 or 1. For groups of type list it must be 0 since that type only generates one address. This variable always returns a string and will take the form of an IP address or hostname.

Examples¶

This property is primarily available for completeness and informational purposes. The pScheduler agent reads the address object’s lead-bind-address property automatically, so it is not required to use this variable in order to properly set a pScheduler server’s binding options. The example below shows lead_bind_address being used for informational purposes in a task:

{
    "group":"throughput_group",
    "test":"throughput_test",
    "reference":{
        "source_lead_bind_address": "{% lead_bind_address[0] %}"
    }
}

Assuming the lead-bind-address property is set to 10.0.0.1 OR that it is not set and the address property is 10.0.0.1 we get the following:

{
    "group":"throughput_group",
    "test":"throughput_test",
    "reference":{
        "source_lead_bind_address": "10.0.0.1"
    }
}

`localhost` Variable¶

Description¶

The localhost template variable returns a string value of “localhost” unless the flip variable is true, then it returns the value of scheduled_by_address.

The localhost template variable takes the following form:

{% localhost %}

This variable may be useful in instances where scheduled_by_address is desired, but the local system identifies itself using a private IP address different from that in scheduled_by_address.

Note

If you do not have the private IP limitation, then scheduled_by_address is generally preferred as its clearer as to where results are stored once expanded.

Examples¶

The localhost template variable can be useful when building URLs for archives. The example below embeds the variable in the url field. If the host expected to schedule the test does not have no-agent enabled then it will expand to “localhost” as shown below:

Template:

"local_archive": {
     "archiver": "http",
     "data": {
         "_url": "https://{% localhost %}/logstash"
     }
}

Post-Expansion:

"local_archive": {
     "archiver": "http",
     "data": {
         "_url": "https://localhost/logstash"
     }
}

If that same example is for a task where the host expected to schedule the task has no-agent enabled, thus flip is true, then it will return the following if the address of the no-agent host is 10.0.0.1:

Post-Expansion:

"local_archive": {
     "archiver": "http",
     "data": {
         "_url": "https://10.0.0.1/logstash"
     }
}

`pscheduler_address` Variable¶

Description¶

This variable accesses the pscheduler-address property of an address object in the list generated by a group for an individual task. It will additionally fallback to the value of the address property if pscheduler-address is not set.

Note

See Introduction to pSConfig Templates for a discussion on how groups work and how they generate address lists.

Note

See Using Non-Standard pScheduler Ports and Addresses for a discussion of the pscheduler-address property’s meaning

The pscheduler_address template variable takes the following form:

{% pscheduler_address[INDEX] %}

The INDEX between square brackets indicates which address object in the generated list to access starting at 0. Since groups of type mesh and disjoint always return pairs of two addresses, the INDEX can be value 0 or 1. For groups of type list it must be 0 since that type only generates one address. This variable always returns a string and will take the form of an IP address, hostname with an optional port.

Examples¶

Most of the standard perfSONAR pScheduler test plug-ins support a source-node and/or dest-node field in their specification. These are for defining a pScheduler server on a port other than 443 or at a different address from the source and dest fields. These are exactly the types of fields for which pscheduler_address is intended. Let’s assume we have a task using the following pair of address definitions:

"addresses":[
    {
        "address":"thr1.perfsonar.net",
        "pscheduler-address": "[fd89:b4d9:341a:8465::1]:8080"
    },
    {
        "address":"thr2.perfsonar.net",
    }
]

Below is test object below and its subsequent expansion using the address objects above:

Template:

 {
   "type":"throughput",
   "spec":{
      "source":"{% address[0] %}",
      "dest":"{% address[1] %}",
      "source-node":"{% pscheduler_address[0] %}",
      "dest-node":"{% pscheduler_address[1] %}"
   }
}

Post-Expansion:

 {
   "type":"throughput",
   "spec":{
      "source":"thr1.perfsonar.net",
      "dest":"thr2.perfsonar.net",
      "source-node":"[fd89:b4d9:341a:8465::1]:8080",
      "dest-node":"thr2.perfsonar.net"
   }
}

Notice how {% pscheduler_address[0] %} expands to the pscheduler-address property of the first address, but for {% pscheduler_address[1] %} it falls back to the value of address since pscheduler-address is not set for the second address.

`scheduled_by_address` Variable¶

Description¶

The scheduled_by_address variable expands to the address property of the address object responsible for creating this task. Responsibility is determined according to the following rules:

If scheduled-by is set in the task, then the generated address object at the index specified by that property is used unless it has no-agent enabled
Otherwise, the first address in the generated list where no-agent is disabled will be used.

Note

There will never be a task where all addresses have no-agent enabled. pSConfig skips those tests since by definition they cannot be created.

The scheduled_by_address template variable takes the following form:

{% scheduled_by_address %}

This variable always returns a string and will take the form of an IP address or hostname.

Examples¶

This is often used in both the url and observer or measurment agent fields for archives (i.e. a field indicating what host performed the measurement). In the _url it tells it to register the results to the host requesting the task. In the x-ps-observer header it explicitly defines the host that requested the measurement, which is a special field the perfSONAR Logstash pipeline that writes to OpenSearch understands. Assuming the variable expands to 10.0.0.1, below is an example template and its expansion:

Template:

"sched_by_archive": {
     "archiver": "http",
     "data": {
         "_url": "https://{% scheduled_by_address %}/logstash"
     },
     "_headers": {
         "x-ps-observer": "{% scheduled_by_address %}"
     }
}

Post-Expansion:

"sched_by_archive": {
     "archiver": "http",
     "data": {
         "_url": "https://10.0.0.1/logstash"
     },
     "_headers": {
         "x-ps-observer": "10.0.0.1"
     }
}

pSConfig Template Variables¶

Template Variable Basics¶

Syntax¶

Determining JSON Type of Expanded Variables¶

Placement in Objects¶

address Variable¶

Description¶

Examples¶

flip Variable¶

Description¶

Examples¶

jq Variable¶

Description¶

Examples¶

lead_bind_address Variable¶

Description¶

Examples¶

localhost Variable¶

Description¶

Examples¶

pscheduler_address Variable¶

Description¶

Examples¶

scheduled_by_address Variable¶

Description¶

Examples¶

`address` Variable¶

`flip` Variable¶

`jq` Variable¶

`lead_bind_address` Variable¶

`localhost` Variable¶

`pscheduler_address` Variable¶

`scheduled_by_address` Variable¶