Technical Reference

BotiumScript Concepts

Botium supports running scripted, pre-defined or pre-recorded conversations written as BotiumScript. A conversation consists of a collection of

  • User inputs which are sent to the chatbot

  • Bot responses which are expected to be received from the chatbot

  • Asserters and Logic Hooks to add advanced assertion or conversation flow logic

  • Scripting Memory to set an get variables

In case there are any differences in the bot responses from the pre-recorded scripts, the script returns a failure notification to the caller. (There are multiple checking methods. You can choose with SCRIPTING_MATCHING_MODE capability)

You can compose your scripted conversations using Text, Excel, CSV, YAML, JSON, Markdown files. You can even mix them in a single test. And with precompilers you can use your own custom JSON or Markdown format.

Convos

Convos are the skeleton of the Botium Scripting, they are describing the flow of a conversation.:

#me
hello

#bot
Hi!

#me
bye

#bot
Goodbye!

Partial Convos

With partial convos it is possible to reuse parts of a convo.

For instance, in order to always start and end with a greeting, you can define a partial convo PCONVO_GREETING (in a file greeting.pconvo.txt):

PCONVO_GREETING

#me
hello

#bot
Hi!

And another one PCONVO_BYE (file bye.pconvo.txt):

PCONVO_BYE

#me
bye

#bot
Goodbye!

Those partial convos can now be included in other convo files:

TC_01

#include PCONVO_GREETING

#me
how are you ?

#bot
I am fine

#include PCONVO_BYE

Another possible syntax would be:

TC_01

#include
PCONVO_GREETING

#me
how are you ?

#bot
I am fine

#include
PCONVO_BYE

Or:

TC_01

#begin
INCLUDE PCONVO_GREETING

#me
how are you ?

#bot
I am fine

#end
INCLUDE PCONVO_BYE

Utterances

With Botium you can separate conversation structure from conversation content using Utterances. They can help you to create multilingual conversations, or alternative messages (like ‘bye’, and ‘goodbye’).

For the sample convo script above, the first text sent to the bot is hello - you surly want your chatbot to react on other greetings like hi, good afternoon, … write an additional utterances file:

USER_HELLO_UTT
hi
hello
nice day

To use this utterance named USER_HELLO:

#me
USER_HELLO_UTT

#bot
Hi!

#me
bye

#bot
Goodbye!

To make Botium use the utterances files in your convos:

Scripting Memory

You can use Scripting Memory to make your test more dynamic. Within a single Botium conversation, it is possible to push variables to a memory and reuse it later. For example:

  • an eCommerce chatbot tells some kind of “order number” (“Your order number is X-1235123”)

  • BotiumScript asks the bot later for the order status (“pls tell me the status for X-1235123”)

You can use the predefined functions of Scripting Memory:

#me
My ID is $random10

And you can multiply your convo using Scripting Memory File. You can create two convos from your buy-beer convo to check that 2 beers costs 4$, and 3 beers costs 6$.

The scripting memory is enabled by setting the :ref:`SCRIPTING_ENABLE_MEMORY capability <cap-scripting-enable-memory>`.

Asserters and Logic Hooks

Asserters and Logic Hooks are used to inject advanced assertion or conversation logic into the conversation flow. They can be added at any position inside the convo file.:

#me
hello
PAUSE 5000

PAUSE is one of the integrated logic hooks, which will just wait for a defined amount of time. The text following the asserter/logic hook reference name are the arguments, separated by a pipe sign (“|”).

Some asserters and logic hooks are integrated into Botium, others are available as additional NPM packages (like Hyperlink Asserter), and you can develop them on your own using Sample Code.

Logic Hooks and User Input Methods always have to be placed below all text in the convo files, as they are always executed at the last possible point in the processing pipeline.

User Input Methods

Main communication channel between a user and chatbot is text. Some chatbots provide simple user interface elements such as buttons:

#me
show me some buttons

#bot
BUTTONS Button1|Button2|Button3

#me
BUTTON Button1

BUTTON will make Botium simulate a click on a button. The implementation depends on the connector in use - for example, the Webdriver connector will look for a HTML button and simulate a user click.

You can use Integrated User Inputs, or develop your own.

Supported File Formats

Composing in Text files

It should be so simple that everyone could compose the conversation files manually. Here is an example for a simple test conversation:

Call Me Captain

#me
hello

#bot
Try: `what is my name` or `structured` or `call me captain`

#me
call me captain

#bot
Got it. I will call you captain from now on.

#me
who am i

#bot
Your name is captain

Conversation and Partial Conversation Syntax

The rules are simple and concise:

  • The first line is the name of the conversation or test case

  • The second line up to the first line starting with one of the special tags below (#begin, #me, #bot, #include, #end) is an optional description text

  • A line starting with #me will send the text following on the next line(s) to your chatbot

    • Anything following the #me in the same line will be the channel to send to - for example: #me #private will send the message to the private channel (Slack only)

    • In case there is a registered utterance detected with matching reference code (see below), the utterance samples are expanded (one conversation for each utterance) and sent to the chatbot

    • If the message to send is not specified, then an empty message will be sent to bot

  • A line starting with #bot will expect your chatbot to answer accordingly

    • Anything following the #bot in the same line will be the channel to listen to - for example: #bot #general will wait for a message on the #general-channel (Slack only)

    • In case there is a registered utterance detected with matching reference code (see below), your chatbot is expected to answer with one of the sample utterances

    • In case the utterance starts with a “?”, the answer is OPTIONAL. Except if it starts with at least two “?”. In this case first “?” will be removed, and the remaining is checked normally (without optional).

    • In case the utterance starts with a “!”, the answer is checked to NOT match the text or one of the utterances samples. Except if it starts with at least two “!”. In this case first “!” will be removed, and the remaining is checked normally (without negation).

    • The OPTIONAL and NOT can be combined. The correct order is first optional then negation: “?!”.

    • If the message to receive is not specified, then the answer wont be checked.

  • A line starting with #include will insert a named partial convo at this place

  • A line starting with #begin will be used on conversation begin only (mainly for asserters and logic hooks, see next section)

  • A line starting with #end will be used on conversation end only (mainly for asserters and logic hooks, see next section)

  • For partial convos, #begin and #end is ignored

That’s it.

Utterances Syntax

  • First line contains a “reference code” for the utterances

  • Following lines contain sample utterances

In order to have a clear distinction between literal text and reference code, the recommendation is to use a naming scheme with a special prefix, for example UTT_utterancename

Example file:

UTT_HELLO
hi
hello
nice day

An example for a convo - saying “hello” to the bot should make the bot anwer “hi” or “hello” or any other of the above utterance samples.:

Reply to hello

#me
Hello, Bot!

#bot
UTT_HELLO

Utterances Args

If an utterance name is followed by additional text, those are used to apply formatting with util.format:

UTT_HELLO
hi, %s
hello, %s
nice day

When using this utterance list in the #me-side of a convo files, you have to add a parameter:

Reply to hello

#me
UTT_HELLO bot

#bot
hello

The texts sent to the bot are:

  • hi, bot

  • hello, bot

  • nice day bot

In case there is no format specifier given, the extra arguments are concatenated to the utterance, separated by spaces - that’s why the third example above is missing the comma

When using this utterance list in the #bot-side of a convo file:

Reply to hello

#me
Hello, Bot!

#bot
UTT_HELLO user

So the texts matched are

  • hi, user

  • hello, user

  • nice day user

Scripting Memory Syntax

It’s a visual table format, columns are separated with the ||-character:

        |$productName    |$customer
product1|Wiener Schnitzel|Joe
product2|Frankfurter     |Joe

File naming convention

  • a file named “*.convo.txt” will be considered as conversation file

  • a file named “*.pconvo.txt” will be considered as partial conversation file

  • a file named “*.utterances.txt” will be considered to contain utterances

  • while a file named “*.scriptingmemory.txt” will be considered to contain scripting memory data

Composing in Excel files

The structure is simple and visually appealing.

Conversation and Partial Conversation Syntax

  • First column holds the test case name (optional)

  • Left column corresponds to the #me tag

  • Right column corresponds to the #bot tag

  • An empty row means the convo is over, and the next will start below

Download an example file with explicit test case names and another one without explicit test case names

If you put the #me and #bot message in the same row, then it is recognized as a simple one question one answer conversation. (You cannot mix this two mode on a single sheet) - download an example file here.

../_images/image31.png

Test Case Naming

  • If the first column contains the test case name, it is used as-is

  • Otherwise the test cases are named after the worksheet and the starting cell of the convo in the Excel file - in the above example, the test case is named Dialogs-A2 (worksheet name + “-” + Excel cell reference)

Partial convos

Partial convos are written same way as test case convos:

../_images/image51.png

They are included by convo name with the INCLUDE statement:

../_images/image61.png

Download an example file here

Utterances Syntax

  • Left column has the utterance name

  • Right column holds the list of utterance texts

../_images/image81.png

Download an example file here

Scripting Memory Syntax

  • First column contains the test case name

  • Second column contains the variable name as header and the variable value

../_images/image91.png

Download example files Products / Customers / Convo

Specify Excel Worksheets and Regions

You can tell Botium the sheets and the regions to look for convos and utterances using additional capabilities - see below. By default, Botium will identify the content areas in the worksheets automatically by searching for the first filled cell (row by row).

When feeding Botium with Excel files, the worksheet names point to either conversations, partial conversations utterances, or scripting memory entries. By default, Botium assumes:

  • that all Excel worksheets with name containing “convo” or “dialog” and not “partial” are for convos

  • that all Excel worksheets with name containing “utter” are for utterances

  • that all Excel worksheets with name containing “partial” are for partial convos

  • that all Excel worksheets with name containing “scripting” or “memory” are for scripting memory

You can use these capabilities to tell Botium what worksheets to select for convos, utterances, partial convos and scripting memory:

  • SCRIPTING_XLSX_SHEETNAMES

  • SCRIPTING_XLSX_SHEETNAMES_UTTERANCES

  • SCRIPTING_XLSX_SHEETNAMES_PCONVOS

  • SCRIPTING_XLSX_SHEETNAMES_SCRIPTING_MEMORY

Excel Parsing Capabilities

SCRIPTING_XLSX_MODE

Default: ROW_PER_MESSAGE

Set it to QUESTION_ANSWER to force simple question-answer conversations. Botium makes a guess, so usually you dont have to use this cap.

SCRIPTING_XLSX_HASHEADERS

Default: true

When identifying content areas in the excel sheet, the first row usually is a header row and skipped.

SCRIPTING_XLSX_STARTROW

Disable the automatic identification of content areas and use this starting row in the excel sheets to look for convos and utterances. Counting from 1.

SCRIPTING_XLSX_STARTCOL

Disable the automatic identification of content areas and use this starting column in the excel sheets to look for convos and utterances. Counting from 1. You can use column letters here as well (“A”, “B”, …).

SCRIPTING_XLSX_SHEETNAMES

Comma separated list for sheetnames to look for convos. By default, all sheets containing the name “convo” (and not “partial”) are used.

SCRIPTING_XLSX_SHEETNAMES_UTTERANCES

Comma separated list for sheetnames to look for utterances. By default, all sheets containing the name “utter” are used.

SCRIPTING_XLSX_SHEETNAMES_PCONVOS

Comma separated list for sheetnames to look for partial convos. By default, all sheets containing the name “partial” are used.

SCRIPTING_XLSX_SHEETNAMES_SCRIPTING_MEMORY

Comma separated list for sheetnames to look for scripting memory. By default, all sheets containing the name “scripting” or “memory” are used.

SCRIPTING_XLSX_EOL_SPLIT

Default: \r

Line ending character in Excel. You shouldn’t change this.

SCRIPTING_XLSX_EOL_WRITE

Default: \rn

Line ending character for Botium assertions. You shouldn’t change this.

Composing in CSV files

You can read convos (.convo.csv), partial convos (.pconvo.csv) and utterances from CSV file.

CSV File Structure

There are several structures possible. The suggestion is to stick with the default structures, but you can tune them with capabilities, see below.

  • First row is the header row (will be skipped)

  • Column delimiter is auto-dected (comma, tab, …) (can be fixed)

  • structure is recognized by number of columns

3 Columns: Multi-Turn Conversations

For multi-turn conversations, there are 3 columns required:

  • the “conversationId”-column for grouping conversations together (something unique, no restrictions on format - can be something like the test case name)

  • The “sender”-column for Botium to know if to send to the bot or listen for bot responses (“me” or “bot”)

  • The “text” column for Botium to send to the bot or listen as response

A simple conversation looks like this:

conversationId,sender,text
first,me,hello
first,bot,Hi!

2 Columns: 1-Turn Conversations (Question/Answer)

There are 2 columns required for question/answer:

  • first column contains the question (“#me”)

  • second column contains the expected answer (“#bot”)

A simple conversation looks like this:

question,answer
hello,Hi!

1 Column: Utterances list

Same format as text utterances file

  • first line (header) is the utterance name (header won’t be skipped here)

  • other lines are the user examples

UTT_NAME
hello
Hi!

CSV Parsing Capabilities

SCRIPTING_CSV_DELIMITER

Default: auto-detected

Column separator used for CSV format

SCRIPTING_CSV_QUOTE

Default: “

SCRIPTING_CSV_ESCAPE

Default: “

SCRIPTING_CSV_SKIP_HEADER

By default, a header line is expected.

Column Selectors

By default, the column order is according to the structure (see above). If you have a different column order, you can select other columns by specifying the header name (if present), or the column index (starting with 0):

  • SCRIPTING_CSV_MULTIROW_COLUMN_CONVERSATION_ID

  • SCRIPTING_CSV_MULTIROW_COLUMN_SENDER

  • SCRIPTING_CSV_MULTIROW_COLUMN_TEXT

  • SCRIPTING_CSV_QA_COLUMN_QUESTION

  • SCRIPTING_CSV_QA_COLUMN_ANSWER

Composing in YAML files

convos:
  - name: goodbye
    description: desc of convo goodbye
    steps:
      - begin:
          - PAUSE 500
      - me:
          - bye
      - bot:
          - goodbye!
  - name: convo 1 name
    description: desc of convo
    steps:
      - me:
          - GEETING
          - PAUSE:
            - 500
      - bot:
          - NOT_TEXT:
            - hello
          - INTENT:
            - intent_greeting
      - bot:
          - what can i do for you?
      - me:
          - nothing
      - bot:
          - thanks
utterances:
  GREETING:
    - hi
    - hello!
scriptingMemory:
  - header:
      name: scenario1
    values:
      $var1: var1_1
      $var2: var2_1
  - header:
      name: scenario2
    values:
      $var1: var1_2
      $var2: var2_2

Starting ! is used to denote the YAML, so quote can help to negate assertions (if using flat strings for assertions).:

convos:
  - name: quote
    steps:
      - me:
          - Hello!
      - bot:
          - "!TEXT_CONTAINS_ANY goodbye, bye"

When using nested YAML objects for assertions (see example above), prefix the asserter name with NOT_ (! is not allowed to be used as tag names in YAML).

Composing in JSON files

{
  "convos": [
    {
      "name": "goodbye",
      "description": "desc of convo goodbye",
      "steps": [
        {
          "begin": [
            { "logichook": "PAUSE", "args": "500" }
          ]
        },
        {
          "me": [
            "bye"
          ]
        },
        {
          "bot": [
            "goodbye!"
          ]
        }
      ]
    },
    {
      "name": "convo 1 name",
      "description": "desc of convo",
      "steps": [
        {
          "me": [
            "hi",
            "PAUSE 500"
          ]
        },
        {
          "bot": [
            { "asserter": "TEXT", "args": "hello", "not": true },
            { "asserter": "INTENT", "args": "intent_greeting" }
          ]
        },
        {
          "bot": [
            "what can i do for you?"
          ]
        },
        {
          "me": [
            "nothing"
          ]
        },
        {
          "bot": [
            "thanks"
          ]
        }
      ]
    }
  ],
  "utterances": {
    "GREETING": [
      "hi",
      "hello!"
    ]
  },
  "scriptingMemory": [
    {
      "header": {
        "name": "scenario1"
      },
      "values": {
        "$var1": "var1_1",
        "$var2": "var2_1"
      }
    },
    {
      "header": {
        "name": "scenario2"
      },
      "values": {
        "$var1": "var1_2",
        "$var2": "var2_2"
      }
    }
  ]
}

Composing in Markdown files

# Convos
## Test Case 1
- me
  - hello bot
- bot
  - hello meat bag
  - BUTTONS checkbutton|checkbutton2
## Test Case 2
- me
  - hello bot
- bot
  - TEXT
    - hello meat bag
  - BUTTONS
    - checkbutton
    - checkbutton2
## Test Case with utterances
- me
  - UTT_HELLO
# Utterances
## UTT_HELLO
- hi
- hello
- greeting

Using the Scripting Memory

Convos, and utterances are the static. You can’t send with them a random number to the bot. Assert that the bot answers the current year, or uses your name if you told it before. But you can do it with scripting memory.

You can think of Scripting Memory as collection of system functions like current year, and variables like your name.

The scripting memory is enabled by setting the :ref:`SCRIPTING_ENABLE_MEMORY capability <cap-scripting-enable-memory>`.

Scripting Memory Variables

Within a single Botium conversation, it is possible to push some information items to a memory and reuse it later. For example:

  • an eCommerce chatbot tells some kind of “order number” (“Your order number is X-1235123”)

  • BotiumScript asks the bot later for the order status (“pls tell me the status for X-1235123”)

For a conversation step originating from the chatbot, use $varname as a placeholder. The scripting memory is filled from this part of the chatbot output text:

#bot
Your order number is $orderNum

For a conversation step originating from the user, again use $varname as a placeholder. The scripting memory will be used to complete the text sent to the chatbot.:

#me
pls tell me the status for $orderNum

There are some restrictions when choosing a variable name:

  • They are all starting with a $, followed by a character (lowercase or uppercase)

  • Followed by an arbitrary number of lowercase/uppercase characters and numbers

Using variables

They have two functions, depending on the usage. Lets say you already set a variable $username to Joe. Then you can send my name is Joe to bot in the #me section:

#me
my name is $username

Or you can use it to assert the response of the bot in #bot section. Did bot answered your name is Joe?:

#bot
your name is $username

As you see variables are starting with ‘$’ to distinguish them from normal text.

Set variables

But how gets scripting memory data? You have to chose depending on your use case. Most basic case if you set them in convo:

#me
what is your name?

#bot
my name is $botname.

#me
Hello $botname!

You can use this way if the variable is coming from the bot.

It is not always obious how long is the variable. For example botium will extract &@#? as password from sentence my password is &@#? but there is a capability to tune this behavior.

Other case is, when you want to multiply your conversations.

Set the variable from file:

      |$toEat           |$toDrink |$costs
Case1 |Two salami pizza |Two cola |30
Case2 |Cheeseburger     |nothing  |8

And use it from convo:

#bot
Do you want to eat something?

#me
yes, $toEat please!

#bot
And some drink?

#me
$toDrink

#bot
It's $costs dollar.

This way we defined two conversations.

Third way is to use logic hooks.:

#begin
SET_SCRIPTING_MEMORY name|joe

#bot
what is your name?

#me
$name

#bot
hello $name!

As you see this conversation is still static. But can help you to create better managable conversations.

And if you want to clear a variable, you can use CLEAR_SCRIPTING_MEMORY logichook.

Scripting Memory Functions

They are the pretty functions provided by botium, like current year ($year), or uniqid ($uniqid). Can be send to bot in #me sections, and can be used as asserters in #bot sections same way as variables.

Some of them can even used with parameters - for example $number(5) generates 5 digit long random number.

You can assert the response of the bot with functions:

#me
What is the current year?

#bot
$year

Or you can send them to bot:

#me
Current year is $year.

You can use parameters:

#me
Please call me $random(5).

You can use system environment variables:

#me
Please authenticate my token $env(MY_PERSONAL_TOKEN)

And can execute javascript code with func:

#me
What costs 5 beer?

#bot
They costs $func(5*2) dollar

List of Functions

  • $func(<some javascript code>): Executes JavaScript code. It can has multiple lines. The result of the last row will be injected.

  • $env(MY_ENV_VAR): Reads sytem environment variables

  • $cap(MY_CAP): Reads Botium capabilities

  • $msg(JSONPATH): Reads something from the current Botium message with a JSONPath expression, for example: $msg($.messageText)

  • $projectname: Test Project Name

  • $testsessionname: Test Session Name

  • $testcasename: Test Case Name (Convo Name)

  • $date(<date pattern like hh:mm:ss or YYYY-MM-DD>): Pattern specific. You can use this to display date, and/or time.

  • $now: date and time. Local specific.

  • $now_ISO: date and time in ISO format. Example: “2019-04-13T19:27:31.882Z”

  • $now_EN: Example: “4/13/2019, 7:24:48 PM”

  • $now_DE: Example: “03.07.2019, 08:33:06”

  • $date: Locale specific.

  • $date_EN: Example: “4/13/2019”

  • $date_DE: Example: “03.07.2019”

  • $date_ISO: Example: “2019-4-13”

  • $time: Local specific.

  • $time_EN: Example: “7:44:11 PM”

  • $time_DE: Example: “08:33:06”

  • $time_ISO: Example: “19:45:12”

  • $time_HH_MM: Example: “19:45” or “01:01“

  • $time_HH: Example: “19” or “01“

  • $time_H_A: Example: “7 PM”

  • $timestamp: 13 digit long timestamp (in ms) like 1557386297267

  • $day_of_month: day of month. Example: “26” if the date is 2019-3-26

  • $day_of_week: day of week. Local specific. Example: “Monday”

  • $month: current month. Local specific. Example: “March”

  • $month_MM: current month. Local specific. Example: “03”

  • $year: Example: “2019”

  • $random10: 10 digit long random number. Example: “6084037818”

  • $random(<length>): <length> digit long random number.

  • $uniqid: V1. Example: “2e65c580-4fb4-11e9-b543-bf076857f1d1”

Scripting Memory Files

You can reuse the same convo more times with Scripting Memory. You have to enable this feature, depending on what Botium Flavour you are using:

  • In Botium CLI, use the –expandscriptingmemory flag

  • In Botium Bindings, add the expandScriptingMemoryToConvos setting to package.json

  • In Botium Box, enable it in the Advanced Scripting Settings

If you don’t enable it explicitly, the scripting variables won’t get pre-filled from the scripting memory file.

Example 1, 4 convos expanded, dynamic variations

Scripting memory for product:

        |$productName
product1|Bread
product2|Beer

Scripting memory for order number:

            |$orderNumber
orderNumber1|1
orderNumber2|100

Convo:

#me
Hi Bot, i want to order $orderNumber $productName

Example 2, 3 convos expanded, scripted variations

Scripting memory for order:

        |$productName|$orderNumber
order1  |Bread       |1
order1  |Beer        |1
order2  |Beer        |100

Convo:

#me
Hi Bot, i want to order $orderNumber $productName

Using Asserters

Asserter are additional validators for conversations. For Example if you want to check if the links send by the bot are valid references you can use and asserter called HyperLinkAsserter, which is trying to reach the sent links.

Buttons Asserter

Some Chatbots are responding not only with text, but provide simple interaction elements such as buttons, to trigger special or predefined functionality. Botium can assert the existance of such buttons in the chatbot response.

It is possible to use this asserter without parameter. In this case asserter will fail if there are no buttons at all.

Processing button responses depends on the Botium connector to support it. For example, it is supported by the Directline and the Dialogflow connector. Please check the connector documentation.

Example

Imagine a chatbot taking orders for pizza delivery. It has a well-defined inventory of possible pizza sizes and toppings. The user interface should present those options to the user as simple buttons:

#me
please send me two salami pizza

#bot
Please select the size of the pizza
BUTTONS Kids|Normal|Family

The BUTTONS asserter (arguments: button texts to look out for), used in #bot section, will assert that buttons with text are present in response.

BUTTONS_COUNT and BUTTONS_COUNT_REC

Those asserters will validate the number of buttons (BUTTONS_COUNT_REC will also count nested buttons).

You can use number comparision there (or use a number for equality):

BUTTONS_COUNT 2
BUTTONS_COUNT =2
BUTTONS_COUNT >2
BUTTONS_COUNT <=3

Media Asserter

Some Chatbots are responding not only with text, but with pictures, videos or other media content. Botium can assert the existance of media attachments in the chatbot response.

It is possible to use this asserter without parameter. In this case asserter will fail if there is no media at all.

Processing media responses depends on the Botium connector to support it. For example, it is supported by the Directline and the Dialogflow connector. Please check the connector documentation.

Example

Imagine a chatbot taking orders for pizza delivery. It has a well-defined inventory of possible pizza sizes and toppings. The user interface should visualize the different sizes by using pictures:

#me
please send me two salami pizza

#bot
Please select the size of the pizza
MEDIA kids_pizza.png|normal_pizza.png|family_pizza.png

The MEDIA asserter (arguments: media uri to look out for), used in #bot section, will assert that media files are attached in the response.

MEDIA_COUNT and MEDIA_COUNT_REC

Those asserters will validate the number of media content (MEDIA_COUNT_REC will also count nested content).

You can use number comparision there (or use a number for equality):

MEDIA_COUNT 2
MEDIA_COUNT =2
MEDIA_COUNT >2
MEDIA_COUNT <=3

Forms Asserter

Some Chatbots are responding with form. You can assert the fields of the form using this asserter.

It is possible to use this asserter without parameter. In this case asserter will fail if there are no forms at all.

Processing forms depends on the Botium connector to support it. For example, it is supported by the Directline connector. Please check the connector documentation.

Example

Imagine a chatbot taking orders for pizza delivery using form. Form has two fields, type to set pizza type, and a count field for its count. You can check those fields:

#me
i want to order a pizza

#bot
FORMS type|count

It means that you excpect two fields, “type”, and “count”, and no more.

Fields have a name, or an ID, and most of them a label before. If you expect ‘type’, and there is a field with label ‘Type of the pizza’ then asserter will accept it, even if its name is not ‘type’.

JSONPath Asserter

This is a generic asserter to assert existance or the value of JSONPath expressions within the underlying chatbot response data. The structure of the data depends on the nature of the connector used - for example, with IBM Watson, the underlying response data is the API response from the Watson HTTP/JSON API.

You can use this asserter for adding your custom assertion logic to BotiumScript.

Example

Imagine an eCommerce chatbot - the response contains the shopping cart in session variables. The following BotiumScript asserts that the cart is available in the session, and the ordered item is in the cart:

#me
add to cart 5 bananas

#bot
JSON_PATH $.session.cart
JSON_PATH $.session.cart.item[0].name | banana

The JSON_PATH asserter takes one or two arguments:

  • First argument is the JSONPath expression to query

  • If a second argument is given, the value is compared to the outcome of the JSONPath expression (if the expression results in multiple values, then it is compared to all of them). If not given, then only the existance of the element is asserted.

This asserter always works on the sourceData field of the botMsg, not on the botMsg as a whole.

JSON_PATH_COUNT

This asserter will validate the number of JSONPath results.

You can use number comparision there (or use a number for equality):

JSON_PATH_COUNT $.session.cart.item|2
JSON_PATH_COUNT $.session.cart.item|=2
JSON_PATH_COUNT $.session.cart.item|>2
JSON_PATH_COUNT $.session.cart.item|<=3

Extending JSONPath Asserter

JSONPath Asserter can optionally be configured with global args in botium.json. Arguments from convo file are handed over and used as specified.

  • argCount - Number of arguments to expect in the convo file

  • path - predefined JSONPath expression

  • pathTemplate - Mustache template for predefined JSONPath expression (based on args)

  • assertTemplate - Mustache template for assertion value (based on args)

  • matchingMode (since 1.11.6) - matching mode to use for assertions (see SCRIPTING_MATCHING_MODE capability) (default is to use the global matching mode)

Example 1 - WATSONV1_HAS_CONTEXT

{
  "botium": {
    "Capabilities": {
      ...
      "ASSERTERS": [
        {
          "ref": "WATSONV1_HAS_CONTEXT",
          "src": "JsonPathAsserter",
          "args": {
            "argCount": 1,
            "pathTemplate": "$.context['{{args.0}}']",
            "matchingMode": "equalsIgnoreCase"
          }
        }
      ]
    }
  }
}

Usage:

#bot
WATSONV1_HAS_CONTEXT my-context-variable

Example 1 - WATSONV1_CONTEXT

{
  "botium": {
    "Capabilities": {
      ...
      "ASSERTERS": [
        {
          "ref": "WATSONV1_CONTEXT",
          "src": "JsonPathAsserter",
          "args": {
            "argCount": 2,
            "pathTemplate": "$.context['{{args.0}}']",
            "assertTemplate": "{{args.1}}"
          }
        }
      ]
    }
  }
}

Usage:

#bot
WATSONV1_CONTEXT my-context-variable|expected-value

Response Length Asserter

This asserter checks the length of the response, and the count of the responses (if there are multiple delivered at once). Typically, a chatbot shouldn’t deliver too much information at once.

Example

Imagine a user asking a chatbot for help:

#me
please help

#bot
RESPONSE_LENGTH 200|5

This asserter takes one or two arguments:

  • First argument is maximum length of the bot response

  • Second argument is the maximum count of the bot responses - some bots deliver multiple responses at once.

NLP Asserter (Intents, Entities, Confidence)

Natural language enabled chatbots are using some kind of NLP engine in the background to recognize intents and entities for the user input.

This information is not shown to the user directly. It may make sense to assert for the recognized intents and entities instead of the text response of the chatbot - or you can even use it in parallel (assert text and intent confidence for example).

Some NLP engines are pure stateless NLP engines without conversation flow (Like Microsoft Luis). They just returning this NLP information. For this engines you cant assert the responded message (text, buttons, etc), just this NLP information using NLP Asserters.

It is possible to extract statistics with the help of this asserters, comparing expectation with the responses from the NLP engine. You can do it on your own, or you can use the our Botium Coach to do it. (Botium Coach is not published yet. It wont be a standalone tool, will work just in the top of the Botium Box)

Not all Botium connectors support these asserters. It depends if the use chatbot technology exposes this information to Botium. For example, it is supported by the Dialogflow and IBM Watson connectors. Please check the connector documentation.

  • INTENT (arguments: intent name to look out for), used in #bot section, will assert that bot answered with the specified intent.

  • INTENT_CONFIDENCE (arguments: minimal accepted confidence, like “70” for 70%), used in #bot section, will assert that bot answered with at least the specified minimal confidence.

  • INTENT_UNIQUE (no arguments), used in #bot section, will assert that the recognized intent is unique (not alternate intent with same confidence identified).

  • ENTITIES (arguments: expected entities like “from|to”, or minimal entities like “from|…” ), used in #bot section, will assert that bot answered with the specified entities.

  • ENTITY_VALUES (arguments: expected entity values like “2018|2019”, or minimal entity values like “2018|…” ), used in #bot section, will assert that bot answered with the specified entity values.

  • ENTITY_CONTENT (arguments: entity and expected values like location|Budapest|Vienna)

    • One ENTITY_CONTENT asserter checks only one entity. Use more asserters to check more.

    • Does not fail if the response has more values as specified in arguments.

The INTENT_CONFIDENCE asserter can be used as global asserter to make sure the recognized confidence is always higher than a defined threshold.

Example

Imagine a chatbot taking orders for pizza delivery. It has a well-defined inventory of possible pizza sizes and toppings. The recognized intent, entities and the confidence should be asserter:

#me
please send me two salami pizza

#bot
INTENT I_ORDER_PIZZA
INTENT_CONFIDENCE 70
ENTITIES E_PIZZA_TYPE|E_FOOD
ENTITY_VALUES salami|pizza
Please select the size of the pizza

Using ENTITY_VALUES asserter can be confusing sometimes. This assertation will be valid:

#me
I want to travel from Berlin to Vienna.

#bot
Im happy to hear it. And where are you now?
INTENT travel

#me
in Münich.

#bot
So you are in Münich, and want to travel from Berlin to Vienna?
You will travel to Berlin on your own?
INTENT travel
ENTITY_VALUES Berlin|Vienna|Münich

But maybe it is not what you want. You can be more specific using ENTITY_CONTENT asserter:

...
ENTITY_CONTENT FROM|Berlin
ENTITY_CONTENT TO|Vienna
ENTITY_CONTENT LOCATION|Münich

(This example works just on Dialogflow, it aggregates entities)

Using the Intent Confidence Asserter globally

A very common use case is to use the Intent Confidence Asserter as global asserter, to make sure to filter out the weakly resolved intents. To make all conversation steps fail where the intent falls below a confidence of 80, add this section to your botium.json:

{
  "botium": {
    "Capabilities": {
      ...
      "ASSERTERS": [
        {
          "ref": "INTENT_CONFIDENCE",
          "src": "IntentConfidenceAsserter",
          "global": true,
          "args": {
            "expectedMinimum": 80
          }
        }
      ]
    }
  }
}

Text Asserters

You can set globally how to assert response using SCRIPTING_MATCHING_MODE capability. You can extend/override this behavior using Text Asserters for each response.

Asserter names

There are more text asserters

  • Asserter names are starting with TEXT

    • TEXT…

  • The matching mode can be wildcard, regexp, include, and exact match

    • TEXT_WILDCARD…,

    • TEXT_REGEXP…,

    • TEXT_CONTAINS…,

    • TEXT_EQUALS… or simple TEXT…

  • You can decide to use more args. With AND (…_ALL…) or OR (…_ANY…).

    • Exact match supports just OR, this postfix ist not allowed there

    • Example names:

      • TEXT…, (ALL or ANY is not allowed)

      • TEXT_CONTAINS_ALL…

      • TEXT_REGEXP_ANY…

  • Each asserter can work case insensitive (optional _IC prefix)

    • Example names:

      • TEXT_IC,

      • TEXT_CONTAINS_ALL_IC

Features

Utterances as argument:

convos:
  - name: example
    steps:
      - me:
          - Hello!
      - bot:
          - "!TEXT_IC GOODBYE|bye bye"
utterances:
  GREETING:
    - Goodbye
    - Bye

This is conversation is in yaml format, because utterances. It will fail if bot says goodbye (bye bye, goodbye, or bye) for greeting. Check is case insensitive, but exact. Wont fail for byebye, or for bye Joe .

Starting ! is used to denote the YAML, so negation is quoted.

TEXT_IC is an alternative of TEXT_EQUALS_IC

Matching modes

Exact match works on the text part of the response. All other asserters on the whole response object (on response json as string).

Matching using joker

You can expect any text:

TEXT

or no text at all:

!TEXT

using exact match asserter.

Examples

TEXT_WILDCARD_ALL id2_*3|1*4

will not accept “Im Joe, my number is 12345, and my ID is id1_123”, because noting found for regexp id2_*3

TEXT_REGEXP_ALL id1_\d\d\d|[0-9]+

will accept “Im Joe, my number is 12345, and my ID is id1_123”, because booth regexps are found

TEXT_CONTAINS_ANY Joe|Jane|George

will accept “Im Joe, my number is 12345, and my ID is id1_123”, because Joe is there

convos:
  - name: example
    steps:
      - me:
          - Hello!
      - bot:
          - "!TEXT_IC GOODBYE|bye bye"
utterances:
  GREETING:
    - Goodbye
    - Bye

This is conversation is in yaml format, because utterances. It will fail if bot says goodbye (bye bye, goodbye, or bye) for greeting. Check is case insensitive, but exact. Wont fail for byebye, or for bye Joe .

Starting ! is used to denote the YAML, so negation is quoted.

TEXT_IC is an alternative of TEXT_EQUALS_IC

Cards Asserter

Some Chatbots are responding not only with text, but with grouped UI elements. If the grouping is not just visual, but has some extra function like paging, or hiding, then it called Card. Botium can assert the existence of such Cards in the chatbot response.

It is possible to use this asserter without parameter. In this case asserter will fail if there are no cards at all.

Processing card responses depends on the Botium connector to support it. For example, it is supported by the Directline and the Dialogflow connector.

Example

Imagine a chatbot taking food orders. In the response there are cards for paging with titles Soup, Pizza, and Dessert. You can assert them:

#me
What can i order pls?

#bot
Please choose something from our Menu Card!
CARDS Soup|Pizza|Dessert

CARDS_COUNT and CARDS_COUNT_REC

Those asserters will validate the number of cards (CARDS_COUNT_REC will also count nested cards).

You can use number comparision there (or use a number for equality):

CARDS_COUNT 2
CARDS_COUNT =2
CARDS_COUNT >2
CARDS_COUNT <=3

Bot Reply Count Asserters

Those asserters will validate that there are no “unforgotten” (unconsumed) bot replies in the processing queue.

  • BOT_CONSUMED will make sure that there is no more unconsumed bot reply in the processing queue

  • BOT_UNCONSUMED_COUNT will make sure that there are unconsumed bot replies in the processing queue

    • first argument is an expected number - 2, =2, >2 etc

In combination with the SKIP_BOT_UNCONSUMED logic hook there are several common usage scenarios:

Consume All Bot Replies

Want to make sure that all bot replies are consumed by the convo:

#end
BOT_CONSUMED

Ignore Bot Welcome Messages

Some chatbots are sending welcome messages before a real conversation is started. To ignore the first few welcome messages that are sent in the first 5 seconds:

#begin
PAUSE 5000
SKIP_BOT_UNCONSUMED

Expect an Unkown Number of Bot Replies

If a chatbot replies with an unknown number of messages, it is possible to handle this case with something like this:

#me
Hello
PAUSE 3000

#bot
BOT_UNCONSUMED_COUNT >0
SKIP_BOT_UNCONSUMED

#me
...

Or another option (expecting exactly 5 replies finally):

#me
Hello

#end
PAUSE 3000
BOT_UNCONSUMED_COUNT =5
  • Attention: the currently processing bot reply is already consumed, so you have to deduct 1 from the expected number *

Negation

It is possible to negate asserters. If you dont expect Button1 and Button2 in response:

#bot
!BUTTONS Button1|Button2

Some asserters are working without args (see asserter documentation):

#bot
BUTTONS

Which means, it must be at least one button. It is possible to negate those assertions:

#bot
!BUTTONS

It will throw error if bot responds with any button.

Register Asserter as Global Asserter

A Global Asserter is called at every convo step. This doesn’t make sense for all asserters, but there are some where this makes sense. To use one of the integrated asserters as global asserter, you have to register it as global asserter in botium.json:

"ASSERTERS": [
  {
    "ref": "RESPONSE_LENGTH",
    "src": "ResponseLengthAsserter",
    "global": true,
    "args": {
        "globalArg1": 17
    }
  }
]

Using Logic Hooks

Logic Hooks are used to inject advanced conversation logic into the conversation flow.

They can be put everywhere into the script, except bot section. There the order is the following: * Botium Logic Hooks * Requesting bot message * Botium Asserters * Asserting bot message * Botium Asserters and Botium Logic Hooks

Like

#bot PAUSE 100 BUTTON Option1 Hello! Please choose Option1 or Option2! SET_SCRIPTING_MEMORY secondbutton|Option2 BUTTON $secondbutton

If you dont have a text to assert, but you have logichook to run after text asserting, use an empty row:

#me Whats your name? #bot

PAUSE 1000

As default the asserters/logichooks are executed in order they are defined. But some special technical logichooks are forced to execute before, or after this order. For example with WAITFORBOT is always executed in the beginning.

PAUSE

  • argument: pause time in milliseconds

  • when used in a #me section, will pause before/after text is sent to bot depending on its position in script

  • when used in a #bot section, will pause before/after reply is received from bot depending on its position in script

WAITFORBOT

  • argument: wait timeout in milliseconds

  • used in a #bot section, will wait for a bot response for given amount of milliseconds (or forever if nothing is given). See also WAITFORBOTTIMEOUT capability.

INCLUDE

  • argument: name of a partial conversation

  • will insert the referenced partial conversation in the current conversation

SET_SCRIPTING_MEMORY

  • arguments: name of the variable, new value

  • Sets/overwrites a variable

  • Can be used in #begin, #me, and #bot sections, and in botium.json as global.

  • You should start the variable name usually without “$” (Use “$” if you want to use logic hook argument replacement) #me SET_SCRIPTING_MEMORY orderNum|111 pls tell me the status for $orderNum

ASSIGN_SCRIPTING_MEMORY

  • arguments: name of the variable, JSON-Path expression

  • Sets/overwrites a variable from message content

  • Can be used in #bot sections only after the bot message, and in botium,json as global.

  • You should start the variable name usually without “$” (Use “$” if you want to use logic hook argument replacement)

  • Use this logichook with care. If this logichook is before the bot message, then you will got an error while running your test.

Extract a variable from a card and use it in the next conversation step:

validate value
extract value from table and form further utterances based on that value

#me
get invoice details for customer number 435643

#bot

CARDS INVOICE NUMBER
ASSIGN_SCRIPTING_MEMORY invoiceNumber|$.cards[0].content

#me
get invoice due date for invoice number $invoiceNumber

#bot
invoice due date for invoice number $invoiceNumber is 02/12/2020

CLEAR_SCRIPTING_MEMORY

  • arguments: name of the variable

  • Deletes a variable.

  • Can be used in #begin, #me, and #bot sections, but not in botium,json as global. (Global clear has no sense. There is nothing to clear there)

  • You should start the variable name usually without “$” (Use “$” if you want to use logic hook argument replacement)

SKIP_BOT_UNCONSUMED

  • no arguments

  • will clear all currently unconsumed bot reply messages from the processing queue

UPDATE_CUSTOM

Add custom data to an outgoing message to trigger custom behaviour in the connector (consult documentation of the Botium Connector).

  • arguments: custom action, custom field, custom value

  • This logic hook is used for triggering custom actions in a connector. You have to consult the connector documentation for the supported custom actions.

  • When used in the #begin section, the custom action is called for all convo steps

Using UPDATE_CUSTOM globally

To attach custom data to each and every outgoing message, you can make the UPDATE_CUSTOM logic hook act globally:

{
  "botium": {
    "Capabilities": {
      ...
      "LOGIC_HOOKS": [
        {
          "ref": "UPDATE_CUSTOM",
          "src": "UpdateCustomLogicHook",
          "global": true,
          "args": {
            "name": "SET_DIALOGFLOW_QUERYPARAMS",
            "arg": "payload",
            "value": { "key":"value" }
          }
        }
      ]
    }
  }
}

Using User Inputs

Main communication channel between a user and chatbot is text. Some chatbots provide simple user interface elements such as buttons.

Not all user inputs are supported by all connectors. Some connectors only allow text input, others allow file uploads, and others allow filling out forms - it depends on the used technology.

BUTTON

  • Will simulate a button click if the connector supports it.

  • first argument: button payload

  • second argument (optional): button text

Example:

sending button

#me
BUTTON Help|Displays Help

MEDIA

  • Will simulate user sending a picture (url resolved relative to the baseUri or to the convo file) if the connector supports it.

  • arguments: pathes to a media files

    • Can include wildcards to run same convo multiple times for multiple files (only loading from folders supported, not from HTTP servers)

  • global argument baseUri: base media location as URI (for HTTP downloads)

  • global argument baseDir: base media location directory (default: directory where convo file is located)

  • global argument downloadMedia: flag if media should be downloaded and attached to message (see Connector documentation if this is required or not)

Example (one file):

sending picture file

#me
MEDIA send_this_file.png

Example (wildcards):

sending audio files

#me
MEDIA audiodirectory/*.wav

Most common use case is to send recorded audio files instead of text files for testing voice bots. This is supported by several Botium connectors.

  • Dialogflow

  • Lex

  • Directline (just attachment)

FORM

  • To simulate a user filling out a form, typically followed by a simulated button click.

  • first argument: field name

  • second argument: field value (If second argument is empty, form value will be set to “true”)

Example:

sending form

#me
FORM text1|something entered
FORM text2|something else
BUTTON Submit

Global Arguments

Global arguments can be set in botium.json:

{
  "botium": {
    "Capabilities": {
      ...
      "USER_INPUTS": [
        {
          "ref": "MEDIA",
          "src": "MediaInput",
          "args": {
            "downloadMedia": true
          }
        }
      ]
    }
  }
}