DataZen Functions
DataZen Functions are available throughout DataZen Manager in most fields that display a function icon or show a function replacement window. DataZen Functions fall under the following categories, some of which may not be available depending on the context.
- Enzo Functions: a list of built-in functions that can be used inline
- Dataset Schema: a list of fields available from the data source or after being modified by a data pipeline to choose from
- Job Parameters: a list of job-specific parameters that can be used inline
Uses Enzo Functions can be used in multiple places throughout DataZen, namely:
- Data Pipelines Dynamic Columns
- URL of a Sync Job using an HTTP Connect as the source
- Custom SQL Scripts in a Database Target operation
- Name of a database table a database Target
- URI Endpoint, XML or JSON payloads in an HTTP Target
- Initialization and Finalization database scripts when using a DB Target
Example: DataZen Functions in HTTP Payload
In this example, the HTTP Payload provided to a target system contains a
DataZen Function and a field from the source data:
- a random guid: #rndguid()
- a field name replacement: {{ids}}
To insert a field or function, place the cursor at the desired insertion point, then double-click on the field or function to insert.
Certain fields display a DataZen Function icon
to show a secondary screen with a similar layout.
Click on a DataZen function to see a description and example on how to use it.
You can nest functions by adding them together in a call.
For example, the following operation gets the current time in UTC, returns
an ISO8601 compliant date format, and URL encodes the output so it can be
passed in a URL function:
#urlencode(#toiso8601(#utcnow()))
List of Functions
| Function | Description | Comments |
|
#decrement(a)
#decrement(a, b) |
Decrements a value by one, or by the amount specified. | #decrement({{field1}}, 1) |
| #doublequoted(a) | Ensures the string provided is surrounded by double quotes. NULL values are ignored. |
Both examples below returns "hello": #doublequoted(hello) #doublequoted("hello") |
| #eval(code) | Treats the inner parameter as a C# script and executes its content. Expects the output of the script to be a string or an object that supports the .ToString() method. |
Use this function to execute a more complex C# operation.
See the Microsoft documentation for the
CSharpScript.EvaluateAsync |
| #format(a, b, …) | Uses the C# string.Format syntax to build a string with any number of parameters |
The C# syntax uses {0} for the first parameter, {1} for the second, and so forth.
#format('User {0} lives in {1}', {{userId}}, {{state}}) |
|
#formatdate(date, format)
#utcformatdate(date, format) |
Formats a string value into a date with a specific format (following the C# DateTime.ToString(format) notation) | #formatDate(11/01/2000T08:00:00.0000, 0:MM/dd/yy H:mm:ss) |
| #fromhex64(hex64value) | Converts a Hex64 value into a string. | |
|
#increment(a)
#increment(a, b) |
Increments a value by one, or by the amount specified. | #increment({{field1}}, 25) |
|
#http_get(…)
#http_post(…) #http_put(…) #http_delete(…) |
Performs an HTTP/S call to an internet resource and returns a string as output. If the call to the HTTP resource is an image, the content is returned as an Hexadecimal representation of the bytes of the image. |
See the HTTP Functions section below for further information
#http_get({{field_url}}) |
| #iif(a,b, c) | Returns value ‘b’ if value ‘a’ evaluates to true, or 'c' otherwise |
This example returns 1, because trimming spaces reduces the length of the string to 0.
#iif(#len(#trim( )) == 0, 1, 0) |
| #isnotnull(a,b) | Returns value ‘b’ if value ‘a’ is NOT a NULL value |
You can use this function to replace a NON-NULL value with a constant,
or the content of another field
#isnotnull({{field1}}, 0) |
| #isnotnullorempty(a,b) | Returns value ‘b’ if value ‘a’ is NOT a NULL value or an empty string |
You can use this function to replace a NON-NULL value with a constant,
or the content of another field
#isnotnullorempty({{field1}}, 0) |
| #isnull(a,b) | Returns value ‘b’ if value ‘a’ is a NULL |
You can use this function to replace a NULL value with a constant,
or the content of another field
#isnull({{field1}}, 0) |
| #isnullorempty(a,b) | Returns value ‘b’ if value ‘a’ is a NULL value or an empty string |
You can use this function to replace a NULL value with a constant,
or the content of another field
#isnullorempty({{field1}}, 0) |
| #jsonformat(a) | Formats an input JSON document using the JSON options previously defined using the #setjsonsettings() function |
This example removes NULL fields from a JSON document.
#setjsonsettings(nullvaluehandling:ignore) #jsonformat({{jsonfield}}) |
| #last_http_headers() | Returns the headers of the last HTTP call (either the source HTTP request, or made using the #http() function) as an array of key-value pairs | |
| #last_http_mediatype() | Returns the media type of the last HTTP call (either the source HTTP request, or made using the #http() function) as a string | |
| #last_http_status() | Returns the HTTP Status of the last HTTP call (either the source HTTP request, or made using the #http() function) as an integer | |
| #last_http_bytecount() | Returns the byte count of the last HTTP call (either the source HTTP request, or made using the #http() function) as an integer | |
| #last_http_content() | Returns the HTTP Content of the last HTTP call (either the source HTTP request, or made using the #http() function) as a string. If the content is an image, this function returns the Hexadecimal representation of the image as a string. | |
| #left(a, n) | Returns the left-most n characters of a string | #left(‘this is a test’, 5) |
| #len(a) | Returns the length of the string provided | #len(‘this is a test’) |
| #now() | Returns the current datetime in the local timezone of the server | |
| #parse(a,b,c) #parsexml(a,b,c) #parsejson(a,b,c) | Parses a document (json or xml) and returns a single field (c) based on a path (b).
The parse() function attempts to automatically detect the kind of document being processed. The path provided (parameter b) can be null for flat documents |
This example parses an XML field from the source data called xml, and returns the UserName value in a node
called logonRequest
#parse({{xml}}, //logonRequest, UserName) |
|
#parsedate(a, b) #parsedate(a, b, c) |
Parses a string (a) using the format provided as (b) and returns the date formatted using a roundtrip format (O). If paramter c is provided, it uses it as the output format. |
Parses a string using the format provided and returns the date in a roundtrip format (O):
#parsedate(31 12 2018 23:50, dd MM yyyy HH:mm) Parses a string using the format provided and returns the date in an RFC 1123 format (r): #parsedate(31 12 2018 23:50, dd MM yyyy HH:mm, r) |
| #pick(a,b,c…) | Randomly selects a value from the list provided as parameters | Any number of parameters can be provided |
| #replace(a, b, c) | Replaces all occurrences of b with c, in string a | #replace(‘this is a cat’, ‘cat’, ‘dog’) |
| #right(a, n) | Returns the right-most n characters of a string | #right(‘this is a test’, 5) |
| #rnddouble(a,b) | Generates a random double value between two numbers | Both the lower and upper bound values are required |
| #rndguid() | Returns a new random GUID | |
| #rndint(a,b) | Generates a random integer value (Int32) between two numbers | Both the lower and upper bound values are required |
|
#setjsonsettings(a, b, c...) valid parameters: nullvaluehandling:ignore|include culture:invariantculture|currentculture|currentuiculture|(a specific culture: fr-FR) formatting:none|indented dateformatstring:(a date format) dateformathandling:isodateformat|microsoftdateformat dateparsehandling:datetime|datetimeoffset|none datetimezonehandling:local|roundtripkind|utc|unspecified floatparsehandling:double|decimal stringescapehandling:default|escapehtml|escapenonascii |
Changes the behaviod of the #jsonformat function within the context of the current pipeline, such as null handling and date formatting. |
This setting removes fields that are null from #jsonformat operations within the pipeline:
#setjsonsettings(nullvaluehandling:ignore) This setting removes fields that are null from future JSON operations within the pipeline, prevents JSON pretty-formatting, and forces a specific date/time format: #setjsonsettings(nullvaluehandling:ignore, formatting:none, dateformatstring:MM/dd/yyy hh:mm:ss) |
| #singlequoted(a) | Ensures the string provided is surrounded by single quotes. NULL values are ignored. | Both examples below returns 'hello': #singlequoted(hello) #singlequoted('hello') |
| #skip(a,b) | Skips the first few characters (b) in the string provided. | #skip(this is a test, 5) |
| #tohex64(a) | Transforms a value into an Hex64 representation | #tohex64(‘this is a test’) |
| #toiso8601(x) | Converts a datetime field into a ISO 8601 compliant format | This function converts a datetime field into a string in the ISO 8601 format assuming the date is in the current timezone |
| #torfc1123(x) | Converts a datetime field into a RFC 1123 compliant format | |
| #tolower(a) | Turns the string into a lower-case string | #tolower(‘THIS IS A TEST’) |
| #toupper(a) | Turns the string into an upper-case string | #toupper('this is a test’) |
| #trim(a) | Removes leading and trailing spaces from the string provided. | #trim( this is a test ) |
| #utcnow() | Returns the current datetime in UTC format | |
| #utctoiso8601(x) | Converts a UTC datetime field into a ISO 8601 compliant format | This function converts a datetime field into a string in the ISO 8601 format assuming the date is in the UTC timezone |
| #utctorfc1123(c) | Converts a UTC datetime field into a RFC 1123 compliant format | |
| #urlencode(c) | URL encodes the value provided so it is safe to pass as a URL parameter. |
HTTP Functions
Calling HTTP Endpoints within a Data Pipeline is a feature that allows you to use
the data provided by the job as input records. In other words, for each input record,
an HTTP method will be called. Enzo HTTP Functions can be invoked in multiple ways
to make both authenticated and unauthenticated calls to HTTP REST endpoints.
The parameters used by the HTTP functions depend on whether or not the call is
authenticated and the type of call being made.
The following signatures are allowed:
| Function | Param 1 | Param 2 | Param 3 | Param 4 | Description |
|
http_get
http_delete |
uri | Call to HTTP endpoint without authentication | |||
| headers | uri | Call to HTTP endpoint with custom headers without authentication | |||
| @conn(key) | uri | Call to HTTP endpoint using a pre-defined HTTP connection string | |||
| @conn(key) | headers | uri | Call to HTTP endpoint with custom headers using a pre-defined HTTP connection string | ||
|
http_post
http_put |
body | uri | Call to HTTP endpoint without authentication with the specified body | ||
| body | headers | uri | Call to HTTP endpoint with custom headers without authentication with the specified body | ||
| @conn(key) | body | uri | Call to HTTP endpoint using a pre-defined HTTP connection string with the specified body | ||
| @conn(key) | body | headers | uri | Call to HTTP endpoint with custom headers using a pre-defined HTTP connection string with the specified body |
When specifying a pre-defined connection, use the following format: @conn(key).
The key should be the name of the Connection String to use that represents
the HTTP Endpoint, its authentication mechanism, and Rate Limiting setting. HTTP Enzo Function calls can use any of the supported authentication mechanisms including
API Keys and OAuth 2.0 bearer tokens.
When specifying custom headers, the headers parameter should be comma-separated,
and the header itself should be in the normal HTTP header notation:
header:value.
When specifying multiple headers, the parameter should be single-quoted:
'header1:val1,header2:val2'.
Example 1: Unauthenticated HTTP Get
The following example performs an HTTP GET operation and returns a string from a service endpoint running on the localhost without authentication:
http_get(https://localhost/func1)
Example 2: Unauthenticated HTTP Post with Headers
Sends an HTTP POST operation to a localhost endpoint unauthenticated, passing a body and a custom Content-Type header. Sends the userKey and loginId column values for each row processed by the job.
http_post(‘{ “id”: {{userKey}}, “userId”: “{{loginId}}”}’,‘Content-Type:text/json’, https://localhost/func1)
Example 3: Authenticated HTTP Get
Sends an HTTP GET operation to the “TWITTER” HTTP connection string, using the /users relative URI, and passing the userid value found in the source dataset for each record processed by the job. This call returns a JSON document of the tweets for the given userid; if this function is used in a Dynamic Column (in a Data Pipeline), the new column value will contain the JSON document returned by this HTTP request.
http_get(@conn(TWITTER), /users/{{userid}}/tweets)