User Manual

Writing Chatbot Test Cases

The previous section demonstrated how to write Botium Test Cases in BotiumScript. Botium test projects are organized in a directory structure which is

  1. extendable

  2. overviewable

  3. and git-friendly

Anatomy of a Botium Project

The recommended directory structure is like this:

  • botium.json

  • package.json (Botium Bindings only)

  • spec/

  • botium.spec.js (Botium Bindings only)

  • convo/

  • some.convo.txt

  • some.utterances.txt

  • subfolder1/

  • another.convo.txt

  • another.utterances.txt

  • subfolder2/

  • onemore.convo.txt

Starting from the base directory (in the example above spec/convo), Botium will recursivly traverse the directory tree. You can choose any directory structure you think is useful in your project.

Skipping/Ignoring Files and Test Cases

Test Cases, files and folders are ignored if:

  • the file or folder name starts with skip*, skip- or skip_ (case is ignored)

  • there is a .gitignore and the file or folder is matched by one of the rules

Utterance Expansion

Utterance and scripting memory expansion is done dynamically when running test cases.

Running Chatbot Test Cases

Botium CLI vs Botium Bindings

You should use Botium Bindings if:

  • you are familiar with Node.js and test runners like Mocha, Jasmine or Jest

  • you already have some unit tests in your Node.js project and want to add Botium tests to it

You should use Botium CLI if:

  • you want to use more Botium Core functionality than just test automation

  • you do not want to deploy a new technology (Node.js) to your workstations (Botium CLI is available as Docker)

Using Botium CLI

botium-cli run

See Botium CLI Documentation

Using Botium Bindings

npm test

See Botium Bindings Documentation

Configuration with Capabilities

This section describes how to configure Botium. Capabilities are similar to the “DesiredCapabilities” as used in Selenium and Appium: they describe in what context a Chatbot runs (or should run) and how Botium can connect to it.

Configuration Source

Botium reads configuration from several configuration sources:

  • botium.json in the current directory

  • In case NODE_ENV environment variable is set botium.<node-env>.json in the current directory

  • botium.local.json in the current directory - can be used to extract sensitive information out of the botium.json file and should be added to .gitignore

  • In case NODE_ENV environment variable is set botium.<node-env>.local.json in the current directory

  • In case the environment variable “BOTIUM_CONFIG” points to a file, same files read as above (botium.json, botium.<node-env>.json, botium.local.json, botium.<node-env>.local.json)

  • Environment variables “BOTIUM_capability name” are read and considered

Every step overwrites the configuration capabilities from the previous step.

When using Botium in continuous build / testing / deployment environment, it is generally advised to not include passwords or other secrets in the configuration files, but handing it over with environment variables

The configuration files are JSON files with this anatomy:

  "botium": {
    "Capabilities": {
      "PROJECTNAME": "My Botium Project",
      "CONTAINERMODE": "echo",

Connector/Chatbot Technology Selection

The capability CONTAINERMODE is one of the most important settings, as it defines the Botium Connector to use. Botium will try to load the connector by several means, in this order:

  • If it refers to a relative filename of a custom connector (see Howto develop your own Botium Connector) in the current working directory, this file is loaded and used as connector

  • Botium tries to load an NPM module with this name

  • Botium tries to load an NPM module named with a “botium-connector-” prefix

A list of well-known Botium Connectors is available here: Botium Connectors


"CONTAINERMODE": "src/myconnector.js"


"CONTAINERMODE": "botium-connector-echo"

Generic Capabilities

Those capabilities apply to all Chatbot types for all Botium Connectors.


Default: “defaultproject”

The name of the chatbot project. This will be shown in logfiles and reports.


Default: “botiumwork”

The working directory for Botium (relative or absolute). For each session there will be a separate unique working directory created in this directory. It will be created if it doesn’t exist.


Default: “true”

Botium will remove the unique working directory after each session, including all created logfiles and docker containers. In case Botium or a Chatbot doesn’t work as expected, this value should be changed to “false” to keep the logfiles, making it able to investigate.


Default: “10000” (10 seconds)

When waiting for a Chatbot response, this is the default timeout (in milliseconds). An error will be thrown if Chatbot doesn’t answer in time. This can be overruled when using the Botium API (in case long waiting period is expected).


Default: “false”

Simulates human typing speed. Falsy, or ms/keystroke. (Average typing speed is about 290 ms/keystroke)


Default: “true”

If turned off then some features which may damage the run environment, are deactivated.

So is not possible

  • to execute own JavaScript code

  • change environment variable

Specific Capabilities

Capabilities which are specific to a Botium Connector are documented for each Botium Connector separately.

Scripting Capabilities

These capabilities are for fine-tuning the Botium Scripting behaviour.


Default: “wildcardIgnoreCase”

Logic to use for comparing the bot response to the utterances:

  • wildcard to use the asterisk * as wildcard (case sensitive)

  • wildcardIgnoreCase to use the asterisk * as wildcard (case insensitive)

  • regexp to use regular expressions (case sensitive)

  • regexpIgnoreCase to use regular expressions (case insensitive)

  • include to do a substring matching (case sensitive)

  • includeIgnoreCase (or includeLowerCase - legacy value) to do a substring matching (case insensitive)

  • equals to do exact string matching (case sensitive)

  • equalsIgnoreCase to do a exact string matching (case insensitive)


Default: false

Enable the scripting memory.


Default: true

All texts can be “normalized” (cleaned by HTML tags, multiple spaces, line breaks etc)


Default for Botium Core: false Default for Botium Box: true

Collect all asserter errors for a conversation step and return all with one test failure (instead of failing on first failure)


Default: \n

Line ending character for text files.


Default: all

Logic to use for utterances expansion:

  • all: using all utterances (number of scripts grows exponential)

  • first: only take first utterance

  • random: select random utterances (count: see below)


Default: 1

Number of utterances to select by random


Default: empty

When expanding utterances, Botium can be instructed to add an INCOMPREHENSION asserter to make sure the chatbot answers with something meaningful. One of the utterances is noted as INCOMPREHENSION.

For example, the INCOMPREHENSION utterance looks like this:

sorry i don't understand
i didn't get that
can you please repeat

Expanded convos will look like this:

test case 1

sending some text



Default: false

In many data collections, the utterance name is the same as the intent the NLU engine should predict. For these cases, this flag can be used to add an INTENT asserter when expanding the utterances to convos.

For example, an utterance looks like this:

user example 1
user example 2
user example 3

Expanded convos will look like this:





Default: “false”

Used while reading scripting memory from file. If it is set to true then the original convo will be kept


Determines how the variables are extracted from text.

Default: “non_whitespace”

non_whitespace: captures every non whitespace characters:


capturing text


Your name is Joe.

Your name is $name


Your name is John Doe.

Your name is $name


Today is 02/15/2019

Today is $today


word: only take captures word characters:


capturing text


Your name is Joe.

Your name is $name


Your name is John Doe.

Your name is $name


Today is 02/15/2019

Today is $today


joker: capture everything (result is not trimmed!)


capturing text


Your name is Joe.

Your name is $name


Your name is John Doe.

Your name is $name

John Doe.

Today is 02/15/2019

Today is $today


Excel Parsing Capabilities

See Composing in Excel files

CSV Parsing Capabilities

See Composing in CSV files

Rate Limiting

Some cloud-based APIs are subject to rate limiting and only allow a fixed number of requests in a defined time period. Botium Core can limit the number of requests sent to the Botium connector.

When running in Botium Box on multiple agents in parallel, these settings are applied to each agent separately.

See Bottleneck project page for details.


The minimum number of milliseconds between two UserSays calls.

Example: use 333 to limit rate to at most 3 calls per second.


The maximum number of concurrent calls.

Configuring Generic Retry Behaviour

Botium can be configured to retry test cases on certain error conditions. This is an optional behaviour, but it can help you to avoid flaky tests. Some examples where it makes sense:

  • Connection to the chatbot engine is somehow unstable, leading to

    failing test cases, where not the chatbot engine itself is the source of the problem, but the infrastructure

  • Maybe chatbot engine itself occasionally fails on high load, but only

    in test environment. Using the retry mechanism it can be avoided to fail in these cases.

The following capabilities are available for various connector operations:




  • STOP






Default: nothing

Configure a regular expression or a list of regular expressions (JSON array) to trigger the retry behaviour. Often, a simple substring matching is enough.


Default: 1

Number of retries in case a retry-able error has been identified


Default: 1

If more than one retry, you can decide to increase wait times between retries by applying a factor higher than 1 for calculating the time to wait for the next retry


Default: 1000 (1 sec)

Given in milliseconds. The minimum timeout to wait for the next retry.

Botium CLI

Botium CLI is the command line tool to access Botium Core functionality (and more).


Botium CLI is available as Node.js module and as Docker image.

Installing as global Node.js module:

npm install -g botium-cli botium-core

Installing into an existing NPM project (package.json exists):

npm install --save-dev botium-cli

Using the Botium CLI docker image

Instead of installing the NPM package, you can use the Botium CLI docker image instead:

docker run --rm -v $(pwd):/app/workdir botium/botium-cli

Special considerations:

  • You cannot use absolute pathes, but all pathes should be given relative to the current working directory. The current working directory is mapped to the docker container with the -v switch (above this is mapped to the current working directory)

  • For running the console emulator, you will have to add the -it flag to the docker command to enable terminal interactions

docker run --rm -v \$(pwd):/app/workdir -it botium/botium-cli emulator console

Docker Usage under Windows

When using the above command under Windows, especially with git bash, you may receive an error like this:

C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error response from daemon: Mount denied:
The source path "C:/dev/xxxxx;C"
doesn't exist and is not known to Docker.

In this case you have to disable the bash path conversion:



Prepare and run a simple Botium test case:

botium-cli init
botium-cli run

Get help on the command line options:

botium-cli help

Get help on an invididual command:

botium-cli run help

Botium Capabilities configuration

The chatbot capabilities are described in a configuration file. By default, the file named “botium.json” in the current directory is used, but it can be specified with the “–config” command line parameter. The configuration file holds capabilities, envs and sources. Configuration via environment variables is supported as well.

  "botium": {
    "Capabilities": {
      "PROJECTNAME": "botium-sample1",
    "Sources: {
    "Envs": {


botium-cli init

Prepare a directory for Botium usage:

  • Adds a simple botium.json

  • Adds a sample convo file

botium-cli init-dev [connector|asserter|logichook]

Setup a boilerplate development project for Botium connectors, asserters or logic hooks in the current directory: * Adds a Javascript source file with the skeleton code * Adds a botium.json with connector/asserter/logic hook registration * Adds a sample convo file

botium-cli run

Automatically run all your scripted conversations against your chatbot and output a test report

botium-cli hello

Runs a connectivity check against your chatbot by sending a message (by default ‘hello’) and waiting for an answer from bot.

botium-cli nlpanalytics <algorithm>

Runs NLP analytics with the selected algorithm.

  • validate - run one-shot training and testing of NLP engine

  • k-fold - run k-fold training and testing of NLP engine

See this article for further information.

botium-cli nlpextract

Extract utterances from selected Botium connector and write to Botium Utterances files. Supported not by all connectors, please check connector documentation. Supported at least by:

  • Dialogflow

  • IBM Watson

  • Amazon Lex


  • NLP.js

and more to come.

botium-cli *import

Import conversation scripts or utterances from some source (for example, from IBM Watson workspace)

botium-cli inbound-proxy

Launch an HTTP/JSON endpoint for inbound messages, forwarding them to Redis to make them consumable by Botium Core.

See here how to use.

botium-cli emulator

The Botium Console Emulator is a basic command line interface to your chatbot running within Botium. You can record and save your conversation files.:

botium-cli emulator console

botium-cli crawler-run / botium-cli crawler-feedbacks

The Botium Crawler is command line interface to generate conversations along buttons.

The simplest way you can use it from the same folder where you a botium.json file placed. In this case the crawler is going to start with hello and help entry points, and by default try to make the all possible conversation 5 depth along buttons. By default the result is stored in the ./crawler-result folder:

botium-cli crawler-run

The Botium Crawler is able to ask user for feedbacks in case of there are no buttons in the bot answer, so the conversation is stucked before the depth is reached. By default the user feedbacks are stored in ./crawler-result/userFeedback.json file, and these feedbacks are reused in the next runs. With the following command you can edit (add, remove, overwrite) your stored feedbacks:

botium-cli crawler-feedbacks

There are many other configuration parameters. For more information see Botium Crawler.

Botium Bindings


Botium Bindings is available as Node.js module:

npm install botium-cli


You should already have a Node.js project set up with the test runner of your choice (Mocha, Jasmine, Jest supported out of the box). For mocha, you can do it like this:

cd my-project-dir
npm init -y
npm install --save-dev mocha

The following commands will install Botium Bindings, extend your Mocha specs with the Botium test case runner and run a sample Botium test:

cd my-project-dir
npm install --save-dev botium-bindings
npx botium-bindings init mocha
npm install && npm run mocha

Here is what’s happening:

  • Your package.json file is extended with a “botium”-Section and some devDependencies

  • A botium.json file is created in the root directory of your project

  • A botium.spec.js file is created in the “spec” folder to dynamically create test cases out of your Botium scripts

  • A sample convo file is created in the “spec/convo” folder

Place your own Botium scripts in the “spec/convo” folder and Mocha will find them on the next run.

Botium Capabilities configuration

The Botium Capabilities are read from botium.json, see Botium Capabilities.

Botium Bindings configuration

Configuration settings for Botium Bindings are read from the botium section of the package.json file. A typical package.json looks like this:

  "name": "custom",
  "version": "1.0.0",
  "scripts": {
    "test": "mocha spec"
  "devDependencies": {
    "botium-bindings": "latest",
    "botium-connector-echo": "latest",
    "mocha": "latest"
  "botium": {
    "convodirs": [
    "expandConvos": true,
    "expandUtterancesToConvos": false


The folders to look for the BotiumScript test cases.


If you are using BotiumScript with utterances files, enable this to expand the convo files with all possible user examples.


If you are using BotiumScript with utterance files only, enable this to expand all user examples to simple question/response test cases.


If you are using BotiumScript with Scripting Memory Files, enable this to expand the scripting memory.

See this as an example.

Test Runner Configuration

Configuration of the test runners (Mocha, Jasmine or Jest) are done in the package.json command line calls to the test runner CLI. For example, to choose another Mocha reporter than the default one, change it in package.json:

  "scripts": {
    "test": "mocha --reporter json spec"

Test Runner Timeouts

Botium tests can take a rather long time, whereas test runners like Mocha and Jasmine expect the tests to complete within a short period of time. It is possible to extend this period of default 60000ms (60 seconds) by setting the environment variables BOTIUM_MOCHA_TIMEOUT / BOTIUM_JASMINE_TIMEOUT (milliseconds).

Running Tests in Parallel

This is possible when using Jest as test runner.

First, split your convos into multiple directories - for each directory, Jest will run a separate job.

In each directory, place a file botium.spec.js:

const BotiumBindings = require('botium-bindings')
BotiumBindings.helper.jest().setupJestTestSuite({ bb: new BotiumBindings({ convodirs: [__dirname] }) })

Jest will now run test cases in parallel:

  "scripts": {
    "test": "jest spec"

See this as an example.

Botium Crawler

The Botium Crawler is doing the work of detecting the conversation flows supported by your chatbot by itself. It does so by analyzing the quick responses offered by your chatbot and simulating clicks on all of the options in parallel, following all pathes down until it reaches the end of the conversation.

All detected conversation flows along all pathes are saved as Botium test cases and utterance lists and can be used as base for a regression test set.


Install as CLI tool

Install Botium CLI (botium crawler is included):

npm install -g botium-cli

Or you can install directly Botium Crawler:

npm install -g botium-crawler

Install as Node.js module

You can install botium crawler as library in your own project:

npm install botium-crawler

Using as CLI tool with Botium CLI

Botium CLI using Botium Crawler is able crawl your chatbot various way according to the parameters, and it is able to generate and store all possible conversations.

Basically there are two command in Botium CLI to use Botium Crawler crawler-run and crawler-feedbacks.

crawler-run command

Get help on the available parameters:

botium-cli crawler-run --help

The parameters can be applied in two different ways:

  • One is the classic way to add these as command line parameters after the crawler-run command.

  • The other way is to store these parameters into botium-crawler.json file into the root of you working directory and reuse them for the next run. In this case the parameters are read from the botium-crawler.json as default.

You are able to generate `botium-crawler.json` file with `–storeParams` flag.


You can set the path of a json configuration file (e.g.: botium.json):

botium-cli crawler-run --config ./custom-path/botium.json


You can set the output folder of the crawler result. By default the path is ./crawler-result. A scripts folder is going to be created under the output path, and the generated convos and utterances are going to be stored here:

botium-cli crawler-run --config ./botium.json --output ../custom-output


In the entry points array you can define one or more ‘user message’ from where the crawler is going to start the conversations. * By default the crawler is going to start with [‘hello’, ‘help’] entry points, if the chatbot has no auto welcome message(s). * If the chatbot has auto welcome messages, than these welcome messages are going to be taken as entry points, if the user do not specify others in this parameter. (see –numberOfWelcomeMessages parameter)

botium-cli crawler-run --config ./botium.json --entryPoints 'Good Morning' 'Next conversation'


You have to specify the number of auto welcome messages exactly, because the crawler has to wait for these welcome messages before each conversation. By default this is 0. If the bot has auto welcome messages, each generated conversation will start with the auto welcome messages.:

botium-cli crawler-run --config ./botium.json --numberOfWelcomeMessages 2


You can specify the depth of the crawling, by default it is 5.:

botium-cli crawler-run --config ./botium.json --depth 3


You can specify here the array of messages has to be ignore during the crawling process.:

botium-cli crawler-run --config ./botium.json --ignoreSteps 'this message is ignored'


You can specify here the array of messages, which has to be considered during incomprehension validation. The result of the validation is going to be stored in error.log file in the output folder.:

botium-cli crawler-run --config ./botium.json --incomprehension 'Unkown command'


Setting this flag true the same bot answers are going to be merged in one utterance file. By default the flag is true to avoid high number of utterance files.:

botium-cli crawler-run --config ./botium.json --mergeUtterances false


When the crawler stuck at a point in the conversation, before depth is reached, then the crawler is able to ask the user for answers. If this flag is true, then these feedbacks are going to be stored in userFeedback.json file in the output folder, and these answers are automatically used during the next run of the crawler. By default the flag is true.:

botium-cli crawler-run --config ./botium.json --recycleUserFeedback false


Milliseconds to wait for the bot to present the prompt ore response. Useful if the bot sends multiple responses at once.:

botium-cli crawler-run --waitForPrompt 1000


If you would like to generate/overwrite the ./botium-crawler.json file with you currect parameters, you can turn this flag on. This way the parameter are going to be read from this file for the next run. By default the flag is false.:

botium-cli crawler-run --config ./botium.json --storeParams true

Content of ./botium-crawler.json:

  "recycleUserFeedback": true,
  "output": "./crawler-result",
  "incomprehension": [],
  "config": "./botium.json",
  "entryPoints": [],
  "numberOfWelcomeMessages": 0,
  "depth": 5,
  "ignoreSteps": [],
  "mergeUtterances": true,
  "waitForPrompt": 100

Example of crawler-run usage

In this example the botium echo connector will be used, which basically just echoing back what you say. The botium.json configuration file looks like this:

  "botium": {
    "Capabilities": {
      "SCRIPTING_MATCHING_MODE": "wildcardIgnoreCase",
      "CONTAINERMODE": "echo"
    "Envs": {}

Keeping it simple I set just ‘hi’ as entry points. The commandline will look like this:

$ botium-cli crawler-run --config ./botium.json --entryPoints 'hi'
Crawler started...




You said: hi


This path is stucked before reaching depth.
Would you like to continue with your own answers?  [yes, no, no all]: yes
Enter your 1. answer: I said hi
Do you want to add additional answers? [y/n]: n


    hi_I said hi


You said: hi

I said hi

You said: I said hi


This path is stucked before reaching depth.
Would you like to continue with your own answers?  [yes, no, no all]: no
Saving testcases...
The 'crawler-result/scripts/1.1_HI_I-SAID-HI.convo.txt' file is persisted
Crawler finished successfully

The crawler-result folder will look like this:

    ├── scripts
    │   ├── 1.1_HI_I-SAID-HI.convo.txt
    │   ├── UTT_1.1_HI_I-SAID-HI_BOT_1.utterances.txt
    │   └── UTT_1.1_HI_I-SAID-HI_BOT_2.utterances.txt
    └── userFeedback.json

In the next run nothing is asked from the user, because the previous feedbacks are stored in userFeedback.json. (Before next run the crawler-result/scripts folder has to be emptied.) So now the commandline much simpler than at the previous run:

$ botium-cli crawler-run --config ./botium.json --entryPoints 'hi'
Crawler started...
Saving testcases...
The 'crawler-result/scripts/1.1_HI_I-SAID-HI.convo.txt' file is persisted
Crawler finished successfully
  • The convo file is going to be created, despite something goes wrong with any conversation, but it will be differentiated by a FAILED postfix in convo name and filename ( e.g.: 1.1_HI_I-SAID-HI_FAILED.convo.txt ).*

crawler-feedback command

With crawler-feedback command you can edit (add, remove, overwrite) your stored feedbacks in userFeedback.json:

botium-cli crawler-feedback --help


You can specify the path of the json file, where the user feedbacks are stored. By default it reads the ./crawler-result/userFeedback.json if it exits.


You can specify the output path, where the edited feedback has to be stored. By default it is the same as input, so basically the input file is going to be overwritten.

Example of crawler-feedback usage

In this example you have to edit in the previous example stored userFeedback.json file. You will overwrite the previously set I said hi answer with I said hello and then skip the rest:

$ botium-cli crawler-feedbacks



You said: hi


User answers:
1: I said hi

What would you like to do with these answers? [add, remove, overwrite, skip, skip all]: overwrite
Enter your 1. answer: I said hello
Do you want to add additional answers? [y/n]: n

hi_I said hi


You said: hi

I said hi

You said: I said hi


User answers:

What would you like to do with these answers? [add, remove, overwrite, skip, skip all]: skip
Edit finished, exiting... Do you want to save your modifications? [y/n]: y

Now if I run again the crawler from the previous crawler-run example, then the crawler-result folder will look like this:

botium-cli crawler-run --config ./botium.json --entryPoints 'hi'
    ├── scripts
    │   ├── 1.1_HI_I-SAID-HELLO.convo.txt
    │   ├── UTT_1.1_HI_I-SAID-HELLO_BOT_1.utterances.txt
    │   └── UTT_1.1_HI_I-SAID-HELLO_BOT_2.utterances.txt
    └── userFeedback.json

You can use Botium Crawler as individual CLI tool pretty similar as with Botium CLI

Using as library - API Docs

The Botium Crawler is publishing a Crawler and a ConvoHandler. See Github Repository for an example.

Crawler Object

The Crawler need an initialized BotiumDriver from Botium Core or a config parameter, which is a json object with the corresponding Capabilities. Two callback function can be passed as well. The first for ask user to give feedback for the stucked conversations. The second for validating bot answers. You can find example for these callback functions in the sample code as well.

The Crawler has a crawl function, with that the crawling process can be triggered. This function parameters are identical with the CLI parameters:

crawl ({ entryPoints = [], numberOfWelcomeMessages = 0, depth = 5, ignoreSteps = [] })

ConvoHandler Object

The ConvoHandler can decompile the result of the crawl function with decompileConvos function. The decompileConvos function result is an object with a scriptObjects array and a generalUtterances array property.

Botium Grid

The Botium Grid allows you to distribute Botium tests over several machines. Just start the Botium Agent on a remote machine and connect your Botium scripts to the remote agent.

Starting the Botium Grid Agent

… from Botium Core Source

Clone the Botium Core Github repository <> and run:

npm install
npm run agent

This is meant or developers only, as it requires you to setup npm links to botium-core as well.

Using the Botium Agent

… from Botium Bindings or Botium CLI

Set these capabilities:

And run your script as usual.

… from another client

Botium Agent exports a very simple HTTP/JSON API (see Swagger definition). This can be used from any programming language capable of doing HTTP communication, or from tools like Tricentis Tosca and Postman/Newman.


Botium Agent should be started with security enabled. The environment variable “BOTIUM_API_TOKEN” is expected to contain the API Token (“password”) the clients should send in all HTTP requests (in HTTP-Header “BOTIUM_API_TOKEN”).