Technical Reference
BotiumScript Concepts
Botium supports running scripted, pre-defined or pre-recorded conversations written as BotiumScript. A conversation consists of a collection of
User inputs which are sent to the chatbot
Bot responses which are expected to be received from the chatbot
Asserters and Logic Hooks to add advanced assertion or conversation flow logic
Scripting Memory to set an get variables
In case there are any differences in the bot responses from the pre-recorded scripts, the script returns a failure notification to the caller. (There are multiple checking methods. You can choose with SCRIPTING_MATCHING_MODE capability)
You can compose your scripted conversations using Text, Excel, CSV, YAML, JSON, Markdown files. You can even mix them in a single test. And with precompilers you can use your own custom JSON or Markdown format.
Convos
Convos are the skeleton of the Botium Scripting, they are describing the flow of a conversation.:
#me
hello
#bot
Hi!
#me
bye
#bot
Goodbye!
Partial Convos
With partial convos it is possible to reuse parts of a convo.
For instance, in order to always start and end with a greeting, you can define a partial convo PCONVO_GREETING (in a file greeting.pconvo.txt):
PCONVO_GREETING
#me
hello
#bot
Hi!
And another one PCONVO_BYE (file bye.pconvo.txt):
PCONVO_BYE
#me
bye
#bot
Goodbye!
Those partial convos can now be included in other convo files:
TC_01
#include PCONVO_GREETING
#me
how are you ?
#bot
I am fine
#include PCONVO_BYE
Another possible syntax would be:
TC_01
#include
PCONVO_GREETING
#me
how are you ?
#bot
I am fine
#include
PCONVO_BYE
Or:
TC_01
#begin
INCLUDE PCONVO_GREETING
#me
how are you ?
#bot
I am fine
#end
INCLUDE PCONVO_BYE
Utterances
With Botium you can separate conversation structure from conversation content using Utterances. They can help you to create multilingual conversations, or alternative messages (like ‘bye’, and ‘goodbye’).
For the sample convo script above, the first text sent to the bot is hello - you surly want your chatbot to react on other greetings like hi, good afternoon, … write an additional utterances file:
USER_HELLO_UTT
hi
hello
nice day
To use this utterance named USER_HELLO:
#me
USER_HELLO_UTT
#bot
Hi!
#me
bye
#bot
Goodbye!
To make Botium use the utterances files in your convos:
When using Botium CLI, use the –expandutterances yes command line switch
When using Botium Bindings, use the expandConvos flag in the package.json configuration
Scripting Memory
You can use Scripting Memory to make your test more dynamic. Within a single Botium conversation, it is possible to push variables to a memory and reuse it later. For example:
an eCommerce chatbot tells some kind of “order number” (“Your order number is X-1235123”)
BotiumScript asks the bot later for the order status (“pls tell me the status for X-1235123”)
You can use the predefined functions of Scripting Memory:
#me
My ID is $random10
And you can multiply your convo using Scripting Memory File. You can create two convos from your buy-beer convo to check that 2 beers costs 4$, and 3 beers costs 6$.
The scripting memory is enabled by setting the :ref:`SCRIPTING_ENABLE_MEMORY capability <cap-scripting-enable-memory>`.
Asserters and Logic Hooks
Asserters and Logic Hooks are used to inject advanced assertion or conversation logic into the conversation flow. They can be added at any position inside the convo file.:
#me
hello
PAUSE 5000
PAUSE is one of the integrated logic hooks, which will just wait for a defined amount of time. The text following the asserter/logic hook reference name are the arguments, separated by a pipe sign (“|”).
Some asserters and logic hooks are integrated into Botium, others are available as additional NPM packages (like Hyperlink Asserter), and you can develop them on your own using Sample Code.
Logic Hooks and User Input Methods always have to be placed below all text in the convo files, as they are always executed at the last possible point in the processing pipeline.
User Input Methods
Main communication channel between a user and chatbot is text. Some chatbots provide simple user interface elements such as buttons:
#me
show me some buttons
#bot
BUTTONS Button1|Button2|Button3
#me
BUTTON Button1
BUTTON will make Botium simulate a click on a button. The implementation depends on the connector in use - for example, the Webdriver connector will look for a HTML button and simulate a user click.
You can use Integrated User Inputs, or develop your own.
Supported File Formats
Composing in Text files
It should be so simple that everyone could compose the conversation files manually. Here is an example for a simple test conversation:
Call Me Captain
#me
hello
#bot
Try: `what is my name` or `structured` or `call me captain`
#me
call me captain
#bot
Got it. I will call you captain from now on.
#me
who am i
#bot
Your name is captain
Conversation and Partial Conversation Syntax
The rules are simple and concise:
The first line is the name of the conversation or test case
The second line up to the first line starting with one of the special tags below (#begin, #me, #bot, #include, #end) is an optional description text
A line starting with #me will send the text following on the next line(s) to your chatbot
Anything following the #me in the same line will be the channel to send to - for example: #me #private will send the message to the private channel (Slack only)
In case there is a registered utterance detected with matching reference code (see below), the utterance samples are expanded (one conversation for each utterance) and sent to the chatbot
If the message to send is not specified, then an empty message will be sent to bot
A line starting with #bot will expect your chatbot to answer accordingly
Anything following the #bot in the same line will be the channel to listen to - for example: #bot #general will wait for a message on the #general-channel (Slack only)
In case there is a registered utterance detected with matching reference code (see below), your chatbot is expected to answer with one of the sample utterances
In case the utterance starts with a “?”, the answer is OPTIONAL. Except if it starts with at least two “?”. In this case first “?” will be removed, and the remaining is checked normally (without optional).
In case the utterance starts with a “!”, the answer is checked to NOT match the text or one of the utterances samples. Except if it starts with at least two “!”. In this case first “!” will be removed, and the remaining is checked normally (without negation).
The OPTIONAL and NOT can be combined. The correct order is first optional then negation: “?!”.
If the message to receive is not specified, then the answer wont be checked.
A line starting with #include will insert a named partial convo at this place
A line starting with #begin will be used on conversation begin only (mainly for asserters and logic hooks, see next section)
A line starting with #end will be used on conversation end only (mainly for asserters and logic hooks, see next section)
For partial convos, #begin and #end is ignored
That’s it.
Utterances Syntax
First line contains a “reference code” for the utterances
Following lines contain sample utterances
In order to have a clear distinction between literal text and reference code, the recommendation is to use a naming scheme with a special prefix, for example UTT_utterancename
Example file:
UTT_HELLO
hi
hello
nice day
An example for a convo - saying “hello” to the bot should make the bot anwer “hi” or “hello” or any other of the above utterance samples.:
Reply to hello
#me
Hello, Bot!
#bot
UTT_HELLO
Utterances Args
If an utterance name is followed by additional text, those are used to apply formatting with util.format:
UTT_HELLO
hi, %s
hello, %s
nice day
When using this utterance list in the #me-side of a convo files, you have to add a parameter:
Reply to hello
#me
UTT_HELLO bot
#bot
hello
The texts sent to the bot are:
hi, bot
hello, bot
nice day bot
In case there is no format specifier given, the extra arguments are concatenated to the utterance, separated by spaces - that’s why the third example above is missing the comma
When using this utterance list in the #bot-side of a convo file:
Reply to hello
#me
Hello, Bot!
#bot
UTT_HELLO user
So the texts matched are
hi, user
hello, user
nice day user
Scripting Memory Syntax
It’s a visual table format, columns are separated with the ||-character:
|$productName |$customer
product1|Wiener Schnitzel|Joe
product2|Frankfurter |Joe
File naming convention
a file named “*.convo.txt” will be considered as conversation file
a file named “*.pconvo.txt” will be considered as partial conversation file
a file named “*.utterances.txt” will be considered to contain utterances
while a file named “*.scriptingmemory.txt” will be considered to contain scripting memory data
Composing in Excel files
The structure is simple and visually appealing.
Conversation and Partial Conversation Syntax
First column holds the test case name (optional)
Left column corresponds to the #me tag
Right column corresponds to the #bot tag
An empty row means the convo is over, and the next will start below
Download an example file with explicit test case names
and another one without explicit test case names
If you put the #me and #bot message in the same row, then it is recognized as a simple one question one answer conversation. (You cannot mix this two mode on a single sheet) - download an example file here
.
Test Case Naming
If the first column contains the test case name, it is used as-is
Otherwise the test cases are named after the worksheet and the starting cell of the convo in the Excel file - in the above example, the test case is named Dialogs-A2 (worksheet name + “-” + Excel cell reference)
Partial convos
Partial convos are written same way as test case convos:
They are included by convo name with the INCLUDE statement:
Download an example file here
Utterances Syntax
Left column has the utterance name
Right column holds the list of utterance texts
Download an example file here
Scripting Memory Syntax
First column contains the test case name
Second column contains the variable name as header and the variable value
Specify Excel Worksheets and Regions
You can tell Botium the sheets and the regions to look for convos and utterances using additional capabilities - see below. By default, Botium will identify the content areas in the worksheets automatically by searching for the first filled cell (row by row).
When feeding Botium with Excel files, the worksheet names point to either conversations, partial conversations utterances, or scripting memory entries. By default, Botium assumes:
that all Excel worksheets with name containing “convo” or “dialog” and not “partial” are for convos
that all Excel worksheets with name containing “utter” are for utterances
that all Excel worksheets with name containing “partial” are for partial convos
that all Excel worksheets with name containing “scripting” or “memory” are for scripting memory
You can use these capabilities to tell Botium what worksheets to select for convos, utterances, partial convos and scripting memory:
SCRIPTING_XLSX_SHEETNAMES
SCRIPTING_XLSX_SHEETNAMES_UTTERANCES
SCRIPTING_XLSX_SHEETNAMES_PCONVOS
SCRIPTING_XLSX_SHEETNAMES_SCRIPTING_MEMORY
Excel Parsing Capabilities
SCRIPTING_XLSX_MODE
Default: ROW_PER_MESSAGE
Set it to QUESTION_ANSWER to force simple question-answer conversations. Botium makes a guess, so usually you dont have to use this cap.
SCRIPTING_XLSX_HASHEADERS
Default: true
When identifying content areas in the excel sheet, the first row usually is a header row and skipped.
SCRIPTING_XLSX_STARTROW
Disable the automatic identification of content areas and use this starting row in the excel sheets to look for convos and utterances. Counting from 1.
SCRIPTING_XLSX_STARTCOL
Disable the automatic identification of content areas and use this starting column in the excel sheets to look for convos and utterances. Counting from 1. You can use column letters here as well (“A”, “B”, …).
SCRIPTING_XLSX_SHEETNAMES
Comma separated list for sheetnames to look for convos. By default, all sheets containing the name “convo” (and not “partial”) are used.
SCRIPTING_XLSX_SHEETNAMES_UTTERANCES
Comma separated list for sheetnames to look for utterances. By default, all sheets containing the name “utter” are used.
SCRIPTING_XLSX_SHEETNAMES_PCONVOS
Comma separated list for sheetnames to look for partial convos. By default, all sheets containing the name “partial” are used.
SCRIPTING_XLSX_SHEETNAMES_SCRIPTING_MEMORY
Comma separated list for sheetnames to look for scripting memory. By default, all sheets containing the name “scripting” or “memory” are used.
SCRIPTING_XLSX_EOL_SPLIT
Default: \r
Line ending character in Excel. You shouldn’t change this.
SCRIPTING_XLSX_EOL_WRITE
Default: \rn
Line ending character for Botium assertions. You shouldn’t change this.
Composing in CSV files
You can read convos (.convo.csv), partial convos (.pconvo.csv) and utterances from CSV file.
CSV File Structure
There are several structures possible. The suggestion is to stick with the default structures, but you can tune them with capabilities, see below.
First row is the header row (will be skipped)
Column delimiter is auto-dected (comma, tab, …) (can be fixed)
structure is recognized by number of columns
3 Columns: Multi-Turn Conversations
For multi-turn conversations, there are 3 columns required:
the “conversationId”-column for grouping conversations together (something unique, no restrictions on format - can be something like the test case name)
The “sender”-column for Botium to know if to send to the bot or listen for bot responses (“me” or “bot”)
The “text” column for Botium to send to the bot or listen as response
A simple conversation looks like this:
conversationId,sender,text
first,me,hello
first,bot,Hi!
2 Columns: 1-Turn Conversations (Question/Answer)
There are 2 columns required for question/answer:
first column contains the question (“#me”)
second column contains the expected answer (“#bot”)
A simple conversation looks like this:
question,answer
hello,Hi!
1 Column: Utterances list
Same format as text utterances file
first line (header) is the utterance name (header won’t be skipped here)
other lines are the user examples
UTT_NAME
hello
Hi!
CSV Parsing Capabilities
SCRIPTING_CSV_DELIMITER
Default: auto-detected
Column separator used for CSV format
SCRIPTING_CSV_QUOTE
Default: “
SCRIPTING_CSV_ESCAPE
Default: “
SCRIPTING_CSV_SKIP_HEADER
By default, a header line is expected.
Column Selectors
By default, the column order is according to the structure (see above). If you have a different column order, you can select other columns by specifying the header name (if present), or the column index (starting with 0):
SCRIPTING_CSV_MULTIROW_COLUMN_CONVERSATION_ID
SCRIPTING_CSV_MULTIROW_COLUMN_SENDER
SCRIPTING_CSV_MULTIROW_COLUMN_TEXT
SCRIPTING_CSV_QA_COLUMN_QUESTION
SCRIPTING_CSV_QA_COLUMN_ANSWER
Composing in YAML files
convos:
- name: goodbye
description: desc of convo goodbye
steps:
- begin:
- PAUSE 500
- me:
- bye
- bot:
- goodbye!
- name: convo 1 name
description: desc of convo
steps:
- me:
- GEETING
- PAUSE:
- 500
- bot:
- NOT_TEXT:
- hello
- INTENT:
- intent_greeting
- bot:
- what can i do for you?
- me:
- nothing
- bot:
- thanks
utterances:
GREETING:
- hi
- hello!
scriptingMemory:
- header:
name: scenario1
values:
$var1: var1_1
$var2: var2_1
- header:
name: scenario2
values:
$var1: var1_2
$var2: var2_2
Starting ! is used to denote the YAML, so quote can help to negate assertions (if using flat strings for assertions).:
convos:
- name: quote
steps:
- me:
- Hello!
- bot:
- "!TEXT_CONTAINS_ANY goodbye, bye"
When using nested YAML objects for assertions (see example above), prefix the asserter name with NOT_ (! is not allowed to be used as tag names in YAML).
Composing in JSON files
{
"convos": [
{
"name": "goodbye",
"description": "desc of convo goodbye",
"steps": [
{
"begin": [
{ "logichook": "PAUSE", "args": "500" }
]
},
{
"me": [
"bye"
]
},
{
"bot": [
"goodbye!"
]
}
]
},
{
"name": "convo 1 name",
"description": "desc of convo",
"steps": [
{
"me": [
"hi",
"PAUSE 500"
]
},
{
"bot": [
{ "asserter": "TEXT", "args": "hello", "not": true },
{ "asserter": "INTENT", "args": "intent_greeting" }
]
},
{
"bot": [
"what can i do for you?"
]
},
{
"me": [
"nothing"
]
},
{
"bot": [
"thanks"
]
}
]
}
],
"utterances": {
"GREETING": [
"hi",
"hello!"
]
},
"scriptingMemory": [
{
"header": {
"name": "scenario1"
},
"values": {
"$var1": "var1_1",
"$var2": "var2_1"
}
},
{
"header": {
"name": "scenario2"
},
"values": {
"$var1": "var1_2",
"$var2": "var2_2"
}
}
]
}
Composing in Markdown files
# Convos
## Test Case 1
- me
- hello bot
- bot
- hello humanoid
- BUTTONS checkbutton|checkbutton2
## Test Case 2
- me
- hello bot
- bot
- TEXT
- hello humanoid
- BUTTONS
- checkbutton
- checkbutton2
## Test Case with utterances
- me
- UTT_HELLO
# Utterances
## UTT_HELLO
- hi
- hello
- greeting
Using the Scripting Memory
Convos, and utterances are the static. You can’t send with them a random number to the bot. Assert that the bot answers the current year, or uses your name if you told it before. But you can do it with scripting memory.
You can think of Scripting Memory as collection of system functions like current year, and variables like your name.
The scripting memory is enabled by setting the :ref:`SCRIPTING_ENABLE_MEMORY capability <cap-scripting-enable-memory>`.
Scripting Memory Variables
Within a single Botium conversation, it is possible to push some information items to a memory and reuse it later. For example:
an eCommerce chatbot tells some kind of “order number” (“Your order number is X-1235123”)
BotiumScript asks the bot later for the order status (“pls tell me the status for X-1235123”)
For a conversation step originating from the chatbot, use $varname as a placeholder. The scripting memory is filled from this part of the chatbot output text:
#bot
Your order number is $orderNum
For a conversation step originating from the user, again use $varname as a placeholder. The scripting memory will be used to complete the text sent to the chatbot.:
#me
pls tell me the status for $orderNum
There are some restrictions when choosing a variable name:
They are all starting with a $, followed by a character (lowercase or uppercase)
Followed by an arbitrary number of lowercase/uppercase characters and numbers
Using variables
They have two functions, depending on the usage. Lets say you already set a variable $username to Joe. Then you can send my name is Joe to bot in the #me section:
#me
my name is $username
Or you can use it to assert the response of the bot in #bot section. Did bot answered your name is Joe?:
#bot
your name is $username
As you see variables are starting with ‘$’ to distinguish them from normal text.
Set variables
But how gets scripting memory data? You have to chose depending on your use case. Most basic case if you set them in convo:
#me
what is your name?
#bot
my name is $botname.
#me
Hello $botname!
You can use this way if the variable is coming from the bot.
It is not always obious how long is the variable. For example botium will extract &@#? as password from sentence my password is &@#? but there is a capability to tune this behavior.
Other case is, when you want to multiply your conversations.
Set the variable from file:
|$toEat |$toDrink |$costs
Case1 |Two salami pizza |Two cola |30
Case2 |Cheeseburger |nothing |8
And use it from convo:
#bot
Do you want to eat something?
#me
yes, $toEat please!
#bot
And some drink?
#me
$toDrink
#bot
It's $costs dollar.
This way we defined two conversations.
Third way is to use logic hooks.:
#begin
SET_SCRIPTING_MEMORY name|joe
#bot
what is your name?
#me
$name
#bot
hello $name!
As you see this conversation is still static. But can help you to create better managable conversations.
And if you want to clear a variable, you can use CLEAR_SCRIPTING_MEMORY logichook.
Scripting Memory Functions
They are the pretty functions provided by botium, like current year ($year), or uniqid ($uniqid). Can be send to bot in #me sections, and can be used as asserters in #bot sections same way as variables.
Some of them can even used with parameters - for example $number(5) generates 5 digit long random number.
You can assert the response of the bot with functions:
#me
What is the current year?
#bot
$year
Or you can send them to bot:
#me
Current year is $year.
You can use parameters:
#me
Please call me $random(5).
You can use system environment variables:
#me
Please authenticate my token $env(MY_PERSONAL_TOKEN)
List of Functions
$env(MY_ENV_VAR): Reads sytem environment variables
$cap(MY_CAP): Reads Botium capabilities
$msg(JSONPATH): Reads something from the current Botium message with a JSONPath expression, for example: $msg($.messageText)
$projectname: Test Project Name
$testsessionname: Test Session Name
$testcasename: Test Case Name (Convo Name)
$date(<date pattern like hh:mm:ss or YYYY-MM-DD>): Pattern specific. You can use this to display date, and/or time.
$now: date and time. Local specific.
$now_ISO: date and time in ISO format. Example: “2019-04-13T19:27:31.882Z”
$now_EN: Example: “4/13/2019, 7:24:48 PM”
$now_DE: Example: “03.07.2019, 08:33:06”
$date: Locale specific.
$date_EN: Example: “4/13/2019”
$date_DE: Example: “03.07.2019”
$date_ISO: Example: “2019-4-13”
$time: Local specific.
$time_EN: Example: “7:44:11 PM”
$time_DE: Example: “08:33:06”
$time_ISO: Example: “19:45:12”
$time_HH_MM: Example: “19:45” or “01:01“
$time_HH: Example: “19” or “01“
$time_H_A: Example: “7 PM”
$timestamp: 13 digit long timestamp (in ms) like 1557386297267
$day_of_month: day of month. Example: “26” if the date is 2019-3-26
$day_of_week: day of week. Local specific. Example: “Monday”
$month: current month. Local specific. Example: “March”
$month_MM: current month. Local specific. Example: “03”
$year: Example: “2019”
$tomorrow(<date pattern like YYYY-MM-DD>): next day, formatted as given by the pattern (if omitted then locale specific)
$yesterday(<date pattern like YYYY-MM-DD>): next day, formatted as given by the pattern (if omitted then locale specific)
$date_add(amount, unit, pattern): adding to current date (see moment.js) and formatting. Example: $date_add(1, “day”, YYYY.MM.DD)
$date_subtract(amount, unit, pattern): subtracting from current date (see moment.js) and formatting. Example: $date_subtract(1, “month”, YYYY.MM.DD)
$random10: 10 digit long random number. Example: “6084037818”
$random(<length>): <length> digit long random number.
$uniqid: V1. Example: “2e65c580-4fb4-11e9-b543-bf076857f1d1”
Scripting Memory Files
You can reuse the same convo more times with Scripting Memory. You have to enable this feature, depending on what Botium Flavour you are using:
In Botium CLI, use the –expandscriptingmemory flag
In Botium Bindings, add the expandScriptingMemoryToConvos setting to package.json
In Botium Box, enable it in the Advanced Scripting Settings
If you don’t enable it explicitly, the scripting variables won’t get pre-filled from the scripting memory file.
Example 1, 4 convos expanded, dynamic variations
Scripting memory for product:
|$productName
product1|Bread
product2|Beer
Scripting memory for order number:
|$orderNumber
orderNumber1|1
orderNumber2|100
Convo:
#me
Hi Bot, i want to order $orderNumber $productName
Example 2, 3 convos expanded, scripted variations
Scripting memory for order:
|$productName|$orderNumber
order1 |Bread |1
order1 |Beer |1
order2 |Beer |100
Convo:
#me
Hi Bot, i want to order $orderNumber $productName
Using Asserters
Asserter are additional validators for conversations. For Example if you want to check if the links send by the bot are valid references you can use and asserter called HyperLinkAsserter, which is trying to reach the sent links.
Media Asserter
Some Chatbots are responding not only with text, but with pictures, videos or other media content. Botium can assert the existance of media attachments in the chatbot response.
It is possible to use this asserter without parameter. In this case asserter will fail if there is no media at all.
Processing media responses depends on the Botium connector to support it. For example, it is supported by the Directline and the Dialogflow connector. Please check the connector documentation.
Example
Imagine a chatbot taking orders for pizza delivery. It has a well-defined inventory of possible pizza sizes and toppings. The user interface should visualize the different sizes by using pictures:
#me
please send me two salami pizza
#bot
Please select the size of the pizza
MEDIA kids_pizza.png|normal_pizza.png|family_pizza.png
The MEDIA asserter (arguments: media uri to look out for), used in #bot section, will assert that media files are attached in the response.
MEDIA_COUNT and MEDIA_COUNT_REC
Those asserters will validate the number of media content (MEDIA_COUNT_REC will also count nested content).
You can use number comparision there (or use a number for equality):
MEDIA_COUNT 2
MEDIA_COUNT =2
MEDIA_COUNT >2
MEDIA_COUNT <=3
Forms Asserter
Some Chatbots are responding with form. You can assert the fields of the form using this asserter.
It is possible to use this asserter without parameter. In this case asserter will fail if there are no forms at all.
Processing forms depends on the Botium connector to support it. For example, it is supported by the Directline connector. Please check the connector documentation.
Example
Imagine a chatbot taking orders for pizza delivery using form. Form has two fields, type to set pizza type, and a count field for its count. You can check those fields:
#me
i want to order a pizza
#bot
FORMS type|count
It means that you excpect two fields, “type”, and “count”, and no more.
Fields have a name, or an ID, and most of them a label before. If you expect ‘type’, and there is a field with label ‘Type of the pizza’ then asserter will accept it, even if its name is not ‘type’.
JSONPath Asserter
This is a generic asserter to assert existance or the value of JSONPath expressions within the underlying chatbot response data. The structure of the data depends on the nature of the connector used - for example, with IBM Watson, the underlying response data is the API response from the Watson HTTP/JSON API.
You can use this asserter for adding your custom assertion logic to BotiumScript.
Example
Imagine an eCommerce chatbot - the response contains the shopping cart in session variables. The following BotiumScript asserts that the cart is available in the session, and the ordered item is in the cart:
#me
add to cart 5 bananas
#bot
JSON_PATH $.session.cart
JSON_PATH $.session.cart.item[0].name | banana
The JSON_PATH asserter takes one or two arguments:
First argument is the JSONPath expression to query
If a second argument is given, the value is compared to the outcome of the JSONPath expression (if the expression results in multiple values, then it is compared to all of them). If not given, then only the existance of the element is asserted.
This asserter always works on the sourceData field of the botMsg, not on the botMsg as a whole.
JSON_PATH_COUNT
This asserter will validate the number of JSONPath results.
You can use number comparision there (or use a number for equality):
JSON_PATH_COUNT $.session.cart.item|2
JSON_PATH_COUNT $.session.cart.item|=2
JSON_PATH_COUNT $.session.cart.item|>2
JSON_PATH_COUNT $.session.cart.item|<=3
Extending JSONPath Asserter
JSONPath Asserter can optionally be configured with global args in botium.json. Arguments from convo file are handed over and used as specified.
argCount - Number of arguments to expect in the convo file
path - predefined JSONPath expression
pathTemplate - Mustache template for predefined JSONPath expression (based on args)
assertTemplate - Mustache template for assertion value (based on args)
matchingMode (since 1.11.6) - matching mode to use for assertions (see SCRIPTING_MATCHING_MODE capability) (default is to use the global matching mode)
Example 1 - WATSONV1_HAS_CONTEXT
{
"botium": {
"Capabilities": {
...
"ASSERTERS": [
{
"ref": "WATSONV1_HAS_CONTEXT",
"src": "JsonPathAsserter",
"args": {
"argCount": 1,
"pathTemplate": "$.context['{{args.0}}']",
"matchingMode": "equalsIgnoreCase"
}
}
]
}
}
}
Usage:
#bot
WATSONV1_HAS_CONTEXT my-context-variable
Example 1 - WATSONV1_CONTEXT
{
"botium": {
"Capabilities": {
...
"ASSERTERS": [
{
"ref": "WATSONV1_CONTEXT",
"src": "JsonPathAsserter",
"args": {
"argCount": 2,
"pathTemplate": "$.context['{{args.0}}']",
"assertTemplate": "{{args.1}}"
}
}
]
}
}
}
Usage:
#bot
WATSONV1_CONTEXT my-context-variable|expected-value
Response Length Asserter
This asserter checks the length of the response, and the count of the responses (if there are multiple delivered at once). Typically, a chatbot shouldn’t deliver too much information at once.
Example
Imagine a user asking a chatbot for help:
#me
please help
#bot
RESPONSE_LENGTH 200|5
This asserter takes one or two arguments:
First argument is maximum length of the bot response
Second argument is the maximum count of the bot responses - some bots deliver multiple responses at once.
NLP Asserter (Intents, Entities, Confidence)
Natural language enabled chatbots are using some kind of NLP engine in the background to recognize intents and entities for the user input.
This information is not shown to the user directly. It may make sense to assert for the recognized intents and entities instead of the text response of the chatbot - or you can even use it in parallel (assert text and intent confidence for example).
Some NLP engines are pure stateless NLP engines without conversation flow (Like Microsoft Luis). They just returning this NLP information. For this engines you cant assert the responded message (text, buttons, etc), just this NLP information using NLP Asserters.
It is possible to extract statistics with the help of this asserters, comparing expectation with the responses from the NLP engine. You can do it on your own, or you can use the our Botium Coach to do it. (Botium Coach is not published yet. It wont be a standalone tool, will work just in the top of the Botium Box)
Not all Botium connectors support these asserters. It depends if the use chatbot technology exposes this information to Botium. For example, it is supported by the Dialogflow and IBM Watson connectors. Please check the connector documentation.
INTENT (arguments: intent name to look out for), used in #bot section, will assert that bot answered with the specified intent.
INTENT_CONFIDENCE (arguments: minimal accepted confidence, like “70” for 70%), used in #bot section, will assert that bot answered with at least the specified minimal confidence.
INTENT_UNIQUE (no arguments), used in #bot section, will assert that the recognized intent is unique (not alternate intent with same confidence identified).
ENTITIES (arguments: expected entities like “from|to”, or minimal entities like “from|…” ), used in #bot section, will assert that bot answered with the specified entities.
ENTITY_VALUES (arguments: expected entity values like “2018|2019”, or minimal entity values like “2018|…” ), used in #bot section, will assert that bot answered with the specified entity values.
ENTITY_CONTENT (arguments: entity and expected values like location|Budapest|Vienna)
One ENTITY_CONTENT asserter checks only one entity. Use more asserters to check more.
Does not fail if the response has more values as specified in arguments.
The INTENT_CONFIDENCE asserter can be used as global asserter to make sure the recognized confidence is always higher than a defined threshold.
Example
Imagine a chatbot taking orders for pizza delivery. It has a well-defined inventory of possible pizza sizes and toppings. The recognized intent, entities and the confidence should be asserter:
#me
please send me two salami pizza
#bot
INTENT I_ORDER_PIZZA
INTENT_CONFIDENCE 70
ENTITIES E_PIZZA_TYPE|E_FOOD
ENTITY_VALUES salami|pizza
Please select the size of the pizza
Using ENTITY_VALUES asserter can be confusing sometimes. This assertation will be valid:
#me
I want to travel from Berlin to Vienna.
#bot
Im happy to hear it. And where are you now?
INTENT travel
#me
in Münich.
#bot
So you are in Münich, and want to travel from Berlin to Vienna?
You will travel to Berlin on your own?
INTENT travel
ENTITY_VALUES Berlin|Vienna|Münich
But maybe it is not what you want. You can be more specific using ENTITY_CONTENT asserter:
...
ENTITY_CONTENT FROM|Berlin
ENTITY_CONTENT TO|Vienna
ENTITY_CONTENT LOCATION|Münich
(This example works just on Dialogflow, it aggregates entities)
Using the Intent Confidence Asserter globally
A very common use case is to use the Intent Confidence Asserter as global asserter, to make sure to filter out the weakly resolved intents. To make all conversation steps fail where the intent falls below a confidence of 80, add this section to your botium.json:
{
"botium": {
"Capabilities": {
...
"ASSERTERS": [
{
"ref": "INTENT_CONFIDENCE",
"src": "IntentConfidenceAsserter",
"global": true,
"args": {
"expectedMinimum": 80
}
}
]
}
}
}
Text Asserters
You can set globally how to assert response using SCRIPTING_MATCHING_MODE capability. You can extend/override this behavior using Text Asserters for each response.
Asserter names
There are more text asserters
Asserter names are starting with TEXT
TEXT…
The matching mode can be wildcard, regexp, include, and exact match
TEXT_WILDCARD…,
TEXT_REGEXP…,
TEXT_CONTAINS…,
TEXT_EQUALS… or simple TEXT…
You can decide to use more args. With AND (…_ALL…) or OR (…_ANY…).
Exact match supports just OR, this postfix ist not allowed there
Example names:
TEXT…, (ALL or ANY is not allowed)
TEXT_CONTAINS_ALL…
TEXT_REGEXP_ANY…
Each asserter can work case insensitive (optional _IC prefix)
Example names:
TEXT_IC,
TEXT_CONTAINS_ALL_IC
Features
Utterances as argument:
convos:
- name: example
steps:
- me:
- Hello!
- bot:
- "!TEXT_IC GOODBYE|bye bye"
utterances:
GREETING:
- Goodbye
- Bye
This is conversation is in yaml format, because utterances. It will fail if bot says goodbye (bye bye, goodbye, or bye) for greeting. Check is case insensitive, but exact. Wont fail for byebye, or for bye Joe .
Starting ! is used to denote the YAML, so negation is quoted.
TEXT_IC is an alternative of TEXT_EQUALS_IC
Matching modes
Exact match works on the text part of the response. All other asserters on the whole response object (on response json as string).
Matching using joker
You can expect any text:
TEXT
or no text at all:
!TEXT
using exact match asserter.
Examples
TEXT_WILDCARD_ALL id2_*3|1*4
will not accept “Im Joe, my number is 12345, and my ID is id1_123”, because noting found for regexp id2_*3
TEXT_REGEXP_ALL id1_\d\d\d|[0-9]+
will accept “Im Joe, my number is 12345, and my ID is id1_123”, because booth regexps are found
TEXT_CONTAINS_ANY Joe|Jane|George
will accept “Im Joe, my number is 12345, and my ID is id1_123”, because Joe is there
convos:
- name: example
steps:
- me:
- Hello!
- bot:
- "!TEXT_IC GOODBYE|bye bye"
utterances:
GREETING:
- Goodbye
- Bye
This is conversation is in yaml format, because utterances. It will fail if bot says goodbye (bye bye, goodbye, or bye) for greeting. Check is case insensitive, but exact. Wont fail for byebye, or for bye Joe .
Starting ! is used to denote the YAML, so negation is quoted.
TEXT_IC is an alternative of TEXT_EQUALS_IC
Cards Asserter
Some Chatbots are responding not only with text, but with grouped UI elements. If the grouping is not just visual, but has some extra function like paging, or hiding, then it called Card. Botium can assert the existence of such Cards in the chatbot response.
It is possible to use this asserter without parameter. In this case asserter will fail if there are no cards at all.
Processing card responses depends on the Botium connector to support it. For example, it is supported by the Directline and the Dialogflow connector.
Example
Imagine a chatbot taking food orders. In the response there are cards for paging with titles Soup, Pizza, and Dessert. You can assert them:
#me
What can i order pls?
#bot
Please choose something from our Menu Card!
CARDS Soup|Pizza|Dessert
CARDS_COUNT and CARDS_COUNT_REC
Those asserters will validate the number of cards (CARDS_COUNT_REC will also count nested cards).
You can use number comparision there (or use a number for equality):
CARDS_COUNT 2
CARDS_COUNT =2
CARDS_COUNT >2
CARDS_COUNT <=3
Bot Reply Count Asserters
Those asserters will validate that there are no “unforgotten” (unconsumed) bot replies in the processing queue.
BOT_CONSUMED will make sure that there is no more unconsumed bot reply in the processing queue
BOT_UNCONSUMED_COUNT will make sure that there are unconsumed bot replies in the processing queue
first argument is an expected number - 2, =2, >2 etc
In combination with the SKIP_BOT_UNCONSUMED logic hook there are several common usage scenarios:
Consume All Bot Replies
Want to make sure that all bot replies are consumed by the convo:
#end
BOT_CONSUMED
Ignore Bot Welcome Messages
Some chatbots are sending welcome messages before a real conversation is started. To ignore the first few welcome messages that are sent in the first 5 seconds:
#begin
PAUSE 5000
SKIP_BOT_UNCONSUMED
Expect an Unkown Number of Bot Replies
If a chatbot replies with an unknown number of messages, it is possible to handle this case with something like this:
#me
Hello
PAUSE 3000
#bot
BOT_UNCONSUMED_COUNT >0
SKIP_BOT_UNCONSUMED
#me
...
Or another option (expecting exactly 5 replies finally):
#me
Hello
#end
PAUSE 3000
BOT_UNCONSUMED_COUNT =5
Attention: the currently processing bot reply is already consumed, so you have to deduct 1 from the expected number *
Negation
It is possible to negate asserters. If you dont expect Button1 and Button2 in response:
#bot
!BUTTONS Button1|Button2
Some asserters are working without args (see asserter documentation):
#bot
BUTTONS
Which means, it must be at least one button. It is possible to negate those assertions:
#bot
!BUTTONS
It will throw error if bot responds with any button.
Register Asserter as Global Asserter
A Global Asserter is called at every convo step. This doesn’t make sense for all asserters, but there are some where this makes sense. To use one of the integrated asserters as global asserter, you have to register it as global asserter in botium.json:
"ASSERTERS": [
{
"ref": "RESPONSE_LENGTH",
"src": "ResponseLengthAsserter",
"global": true,
"args": {
"globalArg1": 17
}
}
]
Using Logic Hooks
Logic Hooks are used to inject advanced conversation logic into the conversation flow.
They can be put everywhere into the script, except bot section. There the order is the following: * Botium Logic Hooks * Requesting bot message * Botium Asserters * Asserting bot message * Botium Asserters and Botium Logic Hooks
- Like
#bot PAUSE 100 BUTTON Option1 Hello! Please choose Option1 or Option2! SET_SCRIPTING_MEMORY secondbutton|Option2 BUTTON $secondbutton
- If you dont have a text to assert, but you have logichook to run after text asserting, use an empty row:
#me Whats your name? #bot
PAUSE 1000
As default the asserters/logichooks are executed in order they are defined. But some special technical logichooks are forced to execute before, or after this order. For example with WAITFORBOT is always executed in the beginning.
PAUSE
argument: pause time in milliseconds
when used in a #me section, will pause before/after text is sent to bot depending on its position in script
when used in a #bot section, will pause before/after reply is received from bot depending on its position in script
WAITFORBOT
argument: wait timeout in milliseconds
used in a #bot section, will wait for a bot response for given amount of milliseconds (or forever if nothing is given). See also WAITFORBOTTIMEOUT capability.
INCLUDE
argument: name of a partial conversation
will insert the referenced partial conversation in the current conversation
SET_SCRIPTING_MEMORY
arguments: name of the variable, new value
Sets/overwrites a variable
Can be used in #begin, #me, and #bot sections, and in botium.json as global.
You should start the variable name usually without “$” (Use “$” if you want to use logic hook argument replacement) #me SET_SCRIPTING_MEMORY orderNum|111 pls tell me the status for $orderNum
ASSIGN_SCRIPTING_MEMORY
arguments: name of the variable, JSON-Path expression
Sets/overwrites a variable from message content
Can be used in #bot sections only after the bot message, and in botium,json as global.
You should start the variable name usually without “$” (Use “$” if you want to use logic hook argument replacement)
Use this logichook with care. If this logichook is before the bot message, then you will got an error while running your test.
Extract a variable from a card and use it in the next conversation step:
validate value
extract value from table and form further utterances based on that value
#me
get invoice details for customer number 435643
#bot
CARDS INVOICE NUMBER
ASSIGN_SCRIPTING_MEMORY invoiceNumber|$.cards[0].content
#me
get invoice due date for invoice number $invoiceNumber
#bot
invoice due date for invoice number $invoiceNumber is 02/12/2020
CLEAR_SCRIPTING_MEMORY
arguments: name of the variable
Deletes a variable.
Can be used in #begin, #me, and #bot sections, but not in botium,json as global. (Global clear has no sense. There is nothing to clear there)
You should start the variable name usually without “$” (Use “$” if you want to use logic hook argument replacement)
SKIP_BOT_UNCONSUMED
no arguments
will clear all currently unconsumed bot reply messages from the processing queue
UPDATE_CUSTOM
Add custom data to an outgoing message to trigger custom behaviour in the connector (consult documentation of the Botium Connector).
arguments: custom action, custom field, custom value
This logic hook is used for triggering custom actions in a connector. You have to consult the connector documentation for the supported custom actions.
When used in the #begin section, the custom action is called for all convo steps
Using UPDATE_CUSTOM globally
To attach custom data to each and every outgoing message, you can make the UPDATE_CUSTOM logic hook act globally:
{
"botium": {
"Capabilities": {
...
"LOGIC_HOOKS": [
{
"ref": "UPDATE_CUSTOM",
"src": "UpdateCustomLogicHook",
"global": true,
"args": {
"name": "SET_DIALOGFLOW_QUERYPARAMS",
"arg": "payload",
"value": { "key":"value" }
}
}
]
}
}
}
Using User Inputs
Main communication channel between a user and chatbot is text. Some chatbots provide simple user interface elements such as buttons.
Not all user inputs are supported by all connectors. Some connectors only allow text input, others allow file uploads, and others allow filling out forms - it depends on the used technology.
MEDIA
Will simulate user sending a picture (url resolved relative to the baseUri or to the convo file) if the connector supports it.
arguments: pathes to a media files
Can include wildcards to run same convo multiple times for multiple files (only loading from folders supported, not from HTTP servers)
global argument baseUri: base media location as URI (for HTTP downloads)
global argument baseDir: base media location directory (default: directory where convo file is located)
global argument downloadMedia: flag if media should be downloaded and attached to message (see Connector documentation if this is required or not)
Example (one file):
sending picture file
#me
MEDIA send_this_file.png
Example (wildcards):
sending audio files
#me
MEDIA audiodirectory/*.wav
Most common use case is to send recorded audio files instead of text files for testing voice bots. This is supported by several Botium connectors.
Dialogflow
Lex
Directline (just attachment)
FORM
To simulate a user filling out a form, typically followed by a simulated button click.
first argument: field name
second argument: field value (If second argument is empty, form value will be set to “true”)
Example:
sending form
#me
FORM text1|something entered
FORM text2|something else
BUTTON Submit
Global Arguments
Global arguments can be set in botium.json:
{
"botium": {
"Capabilities": {
...
"USER_INPUTS": [
{
"ref": "MEDIA",
"src": "MediaInput",
"args": {
"downloadMedia": true
}
}
]
}
}
}