The process

So, once the cell on the retina did it’s work and the electrical impulse is activated, it travels along a nerve to the brain. From there, that single impulse will most likely activate 1 or more other neurons. These in turn will activate yet another part of the brain and so on and so on. Sometimes the activity can return to the same neurons, causing loop backs. Most likely, this activity somehow finally gets less and less until it dies out (although maybe not completely).

The model

Now, from this simple  process, lets try and create a model that roughly mimics it. So lets pick up some points of interest from the previous description:

• the electrical impulse somehow divides itself over multiple path ways.
• some of these paths die out, others grow.
• there are loop backs
• an electrical signal has a direction: it goes from one point to (an)other(s).
• an electrical signal travels along a path (axons)
• paths connect neural cells

The first thing to notice: something must be happening to the signal cause it is changing. Now, here is where I put down a rule: whatever it is that’s causing these changes, it must either be another neural cell or a pathway (axon), nothing else allowed.

And this is when the idea of resonance was born: what if there are 1 ore more neural cells reacting to the very fact that there is an electrical signal travelling along a pathway, because a cell released some sort of chemicals or some sort of interaction between the 2?  Now, suppose that there are various types of neurons, some of which can change or manipulate other parts of the neural network. These cells could be grouped together into a single cluster and react in sequence to the electrical signal, thus giving the ability to manipulate and change the very signal that caused the activity in the first place.

And thus we get to a programmable network.

A second thing to notice is the ‘split’ behaviour. If there are more neural cells than there are nerves coming into the brain, than the only way that those other neural cells can be activated, is if somehow the electrical impulse divides itself. To mimic this, I introduced the concept of  the ‘split’ instruction. This is a neuron that’s able to change the electrical signal itself by duplicating it and it’s current processing state. Each duplicate however, has small, predetermined variations, causing each signal’s path to grow apart.

Thesaurus

In the chatbot designer, the thesaurus data is used to express abstract knowledge/relationships between words (the plural of a word, conjugations, the opposite, ‘is a relationships,…) . The application provides extensive support for managing this type of data. You can add, delete, move around, import and export single nodes, a section or the entire thesaurus.

Assets

Assets are used to express concrete knowledge (as in: I have 2 sisters). The chatbot designer application provides a custom editor for managing assets.

Topics

Conversations are usually scripted in topics which can also be edited using an editor which is a mix of text processing and structured editors.

More

The neural network designer version provides many more editors that can be used to manipulate various data structures that can be used for visual recognition, audio processing, pattern matching,…

Most of these editors are in constant development. New features and improvements tend to be added as needed. If you have any special requests, you can always post it in the forum section.

At the time of writing, only the windows application provides support for avatars though this feature is also planned for the mobile and web versions.

Both the windows and android versions support speech to text conversion. For the windows application, this is done through the managed SAPI libraries provided by Microsoft. The same is valid for android: speech input is provided by the operating system.

In both cases though, the network can be used to augment the input quality. Speech libraries usually provide a list of possible input statements, each with a ‘weight’ to determine the likelihood. When the neural network is properly set up (read: not to many regular variables are used), it should be able to pick out the best match.

At the time of writing, speech input is not yet directly supported for the online version, although some browsers (chrome) already support speech input and there is some effort to make it into a web standard.

Text to speech

All versions are also able to say the output statements that are generated by the neural network. In windows, this is again done through the SAPI libraries. Both the managed and non managed versions are supported so that the system can work with most voices currently available on the market.

The android app and web interface both make use of the Espeak library to generate the speech. This is a more limiting system since it only allows for synth based voices, so things sound more robotic. Still, espeak also allows you to select between different voices and both pitch and speed can be altered.

In the case of the android app, Espeak is used ‘by default’ as this is usually the speech system that ships with the device. However, android is a fairly open platform: it allows users to replace the default espeak app with a different one. This is no problem for the chatbot application: it will automatically switch to the newly installed voice system.

Future

At the time of writing, the neural network is already able to ‘improve’ input quality by checking multiple possible combinations with it’s known state. This system can still be improved by adding an extra verification layer.

For speech input, there are plans to build a custom input system that is able to more tightly integrate into the neural network. The idea being that this should provide more accuracy. However, this is still just a theory so….

The same goes for speech rendering: with a better integration into the neural network, the system should be able to better control things like pitch, accentuation, speed, volume,… which should result into a more human-like (or emotionally loaded) voice.

Chatting with a chatbot can be done through the viewer or editor application. Both can be started from the windows-start menu. The software package works like a standard windows app: before you can begin, you need to open a project. For this tutorial, lets use one of the demo projects ‘Name & age’ (located in {my documents}\NND\Demos. After loading, you should see something like the next image:

Lets focus on the chatbot window for this tutorial. It contains a toolbar at the top, to the left, an input box and a conversation history. On the right, you can see the character.
With the toolbar, you can perform the following actions

• Toggle audio output: select if the character speaks (and lip-syncs) the output (you also need to have a voice selected for speech output to work).
• Toggle audio input: turn the mic on or off for using free form speech input. Text will appear in the textbox, you will still need to press enter for sending text.
• Hide/display possible multiple variations that were detected by the speech input system so that you can select the correct one.
• Select the voice used by the character.
• cut, copy or clear the conversation history.
• Select the character
• Change the size of the character
• Make the character free floating on the desktop or dock it in the application

All windows clients have a ‘Stop’ button. This is used to terminate the pattern-matching process, in case that there is an error in the patterns (or code) and the system got stuck in an eternal loop or worse. Normally, you only use this while debugging patterns (and code).

Online

Coming soon.

The chatbot’s properties

This can be opened from the menu (view/chatbot properties) or from the toolbar on the project tool-page (first button). In the properties, you can specify some global settings, specific to the current project. This includes some preferences like your and the bot’s name, birthday and gender. These values can be accessed in the patterns.

You can also specify if the system should try to resolve synonyms automatically or not. When first starting out using simple input-patterns, it’s best to use the auto resolve feature. If you are using lots of thesaurus variables in your input patterns however, all that this flag does, is create more unnecessary work and should best be turned off.
A final flag is used to determine how output of multiple rules is combined. This flag only becomes useful if you use sub-topics/rules in your input patterns. For starting out, this can be left as is.

Useful sections for starting out:

• Say on startup: If you like your bot to start the conversation, this is where you put all the possible output statements that can be used as openings for a conversation. The bot will pick 1 at random. When this section is empty, the bot will wait until you initiate a conversation.
• Fall back: If the bot can’t match any input-patterns against the input, it will return a default text ‘No output defined’. In this section, you can overwrite this default value with your custom output. If multiple outputs are specified, one will be picked at random.
• Repetition: In this section you can specify global conditional outputs for handling repetition in the input. A system variable ($Repeatcount) provides access to the nr of repeating patterns that the bot found.

Other sections include:

• Do on startup: this contains do-patterns that are executed when the bot is first started.
• Do after output: this section also contains do-patterns, but which are executed after every input/output pair.
• Context: The final section is used for declaring asset paths to memory that can be used in operations that require context.

These properties and global patterns can be exported and imported to/from an xml file from the main menu (‘file/import’ and ‘file/export’).

Thesaurus data

A second piece of the puzzle consists out of the thesaurus. This is a collection of words that are grouped together in a tree-like structure. The default template and all the demos have an empty thesaurus. This allows you to use whichever thesaurus that you like, including one for a different language (although the current version is still mostly geared for English, particularly for verb conjugations).

To start with though, it’s probably easiest to download the English thesaurus and import it into your projects (right click on the thesaurus tool-frame, select import, thesaurus). This way, you already have a properly structured thesaurus with plenty of initial values. The file to import is in xml format, so you can edit this manually if you want, though this is only advised for building thesauri for other languages.
Once you have a basic data set, you can extend this, either by manually entering new words or by importing more data, as other sub sections in xml format, or you can also import comma separated lists of words. Simply copy the list to the clipboard or a file, right click on the thesaurus node where the items should be added, select import). All the words in comma separated lists will become children of the selected node.

Topics

The next important part to think about are the topics. This is where you define how the bot needs to respond to different input. Like the other sections, you can import already existing topics (from the menu: file/import/topic). Importing multiple topics at once is also supported, just select multiple files in the dialog box. Imported topics will appear in the project view. From there, you can rename, organize or delete all the topics in the project. Note that every topic should have a unique name, topics with duplicate names get a red icon (and can’t be used as sub-topics).

Creating new topics

Of course, at some point, you will want to create your own content. To start a new topic, use one of the toolbar buttons or the insert menu. This will show a new topic (a unique name will automatically be generated) with an empty ‘you say’ and ‘bot says’ section.

To create a new rule, simply start typing in one of these sections. This will automatically create a new rule and input or output pattern. In the simplest form, you put the exact input text that needs to be matched on the left and the bot’s response to the right. When you are ready to move on to calculations and do-pattern, you can press ctrl-D to show/hide these sections (or press on the drop-down buttons). Shft-Ctrl-D will hide/show them all at once.

Selecting and editing patterns

You can either select a section of text within a single pattern (just like in any other text editor) or you can select multiple patterns at once (for deleting, copying,…). If you want to select all the patterns of a single section (for instance, all the ‘you say’ patterns in a rule), double-click on the title of the section. To select multiple patterns in different sections, keep the ctrl key pressed while clicking inside the patterns. Every selected pattern is displayed with a darker, heavier line. Rules can also be selected by clicking on the area around the patterns but outside of the text. Selected rules will also have a darker, heavier border. Press F2 for renaming the selected rule.
Finally, you can use the arrow keys to move around anywhere on the topic, not just within a single pattern.

Delete, cut, copy and paste are done in the usual manner: with the delete button, ctrl-x, ctrl-c, ctrl-v, you know the drill. If you have a need to insert a rule or pattern somewhere, use the ctrl+I shortcut. Depending on the cursor position, a new pattern or rule will be inserted. The ‘Search and replace’ dialog can be shown with the Ctrl+F shortcut, the button on the toolbar or the menu command (Edit/view). To activate topic-search, make certain that the topic is selected. Many tool-frames also have search-capabilities, if one of them is active while pressing the search command, the search of the local tool-frame will be used.
Finally, the topic editor has a spell checker that can be turned on/off in the options dialog (Tools/options – editing tab).

Variables, questions and answers

Even the simplest of bots needs a way to pass some data from the input to the output section and handle question/answer type of situations. The first can be done by using variables in the input patterns. The second example on the right (My name is $value [.]) contains a variable ($value). As you have probably already guessed, variables start with a ‘$’. To display the value in the output, simply use the same variable name in the output pattern.

The section that handles questions and answers can be accessed by expanding the output patterns. Each output-pattern has an extra ‘Response for’ and ‘When not replied’ part. When an output contains a question, and you want to say something specific when the user didn’t respond to that question, add some outputs to ‘When not replied’. This option is most likely not used often, unless you plan to build an SS-style interrogation bot (like the example to the right).
’Response for’ is used on outputs of rules that are valid responses to previously stated questions. When a rule gets activated that has outputs with a ‘response for’ section, but no question is asked, only outputs will be used that don’t link to the pattern we got a response for.
In the example to the right, the ‘Ok, so you’re called $value’ will only be rendered if the previous output was ‘Hi there! What’s your name’ Similarly, the ‘What do you mean with $value’ statement will only be rendered if the previous output is not ‘Hi there!…’

There are other ways that question-answers can be done, but that’s outside the scope of this tutorial.

Creating characters

If you feel like creating your own characters, this is certainly possible. The current system uses a set of images that are bundled together with an xml file. The images can easily be generated with tools like DAZ3D, Poser,… Basically, you need an image for each animation frame and mouth position. Some packages will allow you to semi-automate this export process. Here’s a small tool to automatically compress the images to only the difference. This way, you can use multiple animations at once.

At the time of writing, the xml files still need to be edited manually. The structure is an extension of the CSS file format. Basically, it consists out of the following sections:

• CharacterInfo: information about the character
• Animations: a list of available animations and all the frames for each animation
• VisemeGroups: defines which image goes with which letters.
• IdleLevels: declares which animations should be played during idle time and how.
• BackgroundAnimations: optionally animations that should always play in some sort of loop (useful for things like breathing)
• Backgrounds: optionally a set of images that compose the background image. When multiple images are used, parts of the background can be hidden during animations.

To install your newly created character, create a new directory in {my documents}\nnd\characters. Give the directory the same name as your character and put the xml file in the directory. The exact location of the images is specified in the xml file, but usually you put these together in the same dir or sub.

Projects

A project contains the entire neural network (in binary form) together with a bunch of configuration files in a directory. This directory is accompanied by the main project file. Both have the same name.  When moving around a project, make certain you have both file and directory. When opening and saving projects, you only need to worry about the file, the directory will be created/managed automatically (done in such a way that extra directories like for version control systems, will remain functioning).
The project files themselves (extension ‘.dpl’), are simple xml files which contain various configuration data for the designer. In general though, you don’t need to edit this file manually, all settings can be managed from within the designer.
The most important files in the project directory are binary and should never be manually edited. Some files are xml based, but apart from the test-cases, should be left ‘as-is’ since they contain vital configurations for the internal network.

Topic files

Topics form a major component of a chatbot designer project. They define what the bot can do. Internally, topics are stored as neurons, but in order to share a topic across multiple projects and users, Topic files were introduced. This is an xml file (extension ‘topic.xml’) that contains all the rules defined in the project.

The designer provides functionality to import and export topics one by one or in bulk from the File menu or the context menu on the Project tool-window. So ideally, you should never be confronted with the internals of these files. For the die-hard text-editor fans, here’s a little more details on the structure:

A minimal topic file should contain a topic, name, rules and questions sections. Like:

<Topic>
<Name>Who has</Name>
<Rules />
<Questions />
</Topic>

And a bit more challenging, containing a single rule with 1 input and output pattern pattern:

<Topic>
<Name>Time</Name>
<Rules>
<Rule>
<Name>new rule 1</Name>
<Patterns>
<Pattern>
<Expression>What time is it [?]</Expression>
</Pattern>
</Patterns>
<Conditionals />
<Outputs>
<Output>
<Expression>It is $time:hour\:$time:minute\.</Expression>
<QuestionCanFollow>True</QuestionCanFollow>
</Output>
</Outputs>
<IsOutputSequenced />
<DoPatterns />
<Calculate />
</Rules>
<Questions />
</Topic>

For the curious, the expression: $time:hour\:$time:minute\.  is used to render something like 17:30. The expression has to be split up in 4 parts:

Global patterns

In the ‘chatbot properties’ view (From the menu: View/chatbot properties), you can specify a bunch of global patterns like the opening statements, fallback values, ‘do-patterns’ that need to be executed at startup, just after each statement,…  This also includes all the OS function-mappings that were declared in the ‘OS channel’.  All these can be exported/imported using 1 file, with the extension ‘global.xml’. Import/export can be done from the menu: ‘file/import/global patterns’ or ‘file/export/global patterns’.

As you probably already guessed, this is also in xml format, but like with the topics, the content can be fully managed from within the designer, so there is no need to edit it manually.

Thesaurus files

Thesaurus files are used to share, exchange or replace part of, or the entire thesaurus that is used in a single project (thesauri are project specific and not shared across different projects within the designer). It’s the same routine again as for the previous file types: the extension is ‘thesaurus.xml’, yes, it’s an xml file format and can be fully managed from within the designer. In fact, for the thesaurus, it is more than advised to use the designer and not edit it manually since a thesaurus node can be hard to define as there are multiple levels at which words are stored. The only scenario where you might have to do some manual editing, is when you create a new thesaurus, completely from scratch when all the functionality needs to be preserved and you don’t have access to a network-designer version of the chatbot designer. In that case, the ‘bindings’ section on certain words (numbers, pronouns like I, me, myself,.. needs to be done manually. Here’s a small example of a binding for the object ‘myself’:

  <Object Name=”myself”>
<Text Value=”myself” Index=”0″ POSGRP=”false” />
<Bindings>
<ref>I</ref>
<ref>Singular</ref>
</Bindings>
</Object>

Importing/exporting is done from the context menu on the thesaurus. If no node is selected, the entire thesaurus will be exported, or the content of the imported file will be added to the root. If a node is selected during an import operation, all items will be added as a child of the selected node. For export, only the selected item and all it’s children will be exported recursively.

It’s also possible to import or export just a single level of children of a selected node, in the form of comma-separated-value (csv) files.  For imports, you don’t even need to get the data from a file, but you can just as easily use the clipboard: just copy a csv list to the clipboard, go to the context menu of the parent node and select ‘Import\wordlist from clipboard’. This feature allows you to quickly build up trees of related words.

Asset files

Assets store concrete information like ‘you are reading this now’ and ‘I have written this text’. By now, the theme should have become clear to you: asset files are stored in the xml format, they have the extension ‘asset.xml’. But here’s where we stray a little from the path: currently asset import/export is only supported in the pro version (that is, it’s still a little bit under construction, so it will soon be available). The same for editing: the pro will have an asset editor, but not the basic version. The latter only has the chatbot window for adding input statements.

Test cases

Test case files aren’t independent files like ‘topic thesaurus, asset or global patterns’ files, but are rather part of the project directory. As such, you can’t currently export or import them automatically from within the designer (though you can fully manage them as in copy, delete, edit,.. from within the designer). That said, they do deserve a special mention since it could sometimes be useful to extract some test-cases and copy them to other projects. And, because a test-case file is actually a stand-alone xml file, this is certainly possible.

Test case files are located at {project directory}\DesignerData\  and have the extension ‘TESTCASE’. If you copy a test-case from one project to another, or you want to delete some manually, make certain that the project is closed.

CCS files (characters)

As a final file format, I’d like to talk a little about characters and their files. A character is stored as an xml file (extension ccs) together with a bunch of images. These images can be located in a subdirectory of the ccs file, as long as the path can be described, relative to the ccs file.

For the chatbot designer to be able to use a character, the files (and possibly directory structure) of the char need to be copied to the directory: {My documents}\NND\Characters\CharName,  where ‘Charname’ is the name of the character. Best to do this when the chatbot designer isn’t running.

Unfortunately, ccs files can’t yet be edited in the designer (or any other UI tool). They need to be created manually. Luckily, the structure is relatively simple and allows for lots of copy/paste. More info on the file structure will come shortly.

Images

The animations and visemes are stored as a series of images which are displayed in rapid succession, much like how old school film works. Before you can display these images though, they need to be created. Now, in the olden days, they used to have big chunky camera’s for that. Things have changed a bit since. We can use tools like DAZ3d, Poser, Blender,…

Content

What you put in the images is entirely up to you. There aren’t many limits with respect to content in the sense that there aren’t any ‘expected’ parts. That said, since we are dealing with chatbots, I’d try to put in something resembling a mouth somewhere, just to get some lip-syncing working. Most 3D tools these days, come with some way to manipulate mouth positions. The designer uses 21 different images for lip-sync (+ 1 for silence, which is actually the background image). Most 3D tools only provide morphs for 16 mouth positions. The remaining 5 visemes can be created by reusing other images or by creating your own, if the 3d app allows it. Here’s a good visual overview of all the visemes.

Animations like blinking eyes or moving hair also need to be exported from your 3D package as a series of images. Luckily most 3d software provides a feature to just that. Also, some animations can use the same image multiple times, it’s always best to reuse the same image in those cases, to save memory. For instance, an eye blink can be done with 2 or 3 images: eye fully open is provided by the background, half closed and fully closed.

File formats

Currently, the following file formats are supported:

It should also be possible to use xaml files (to create flash typed characters) , but this is not yet tested and will most likely not yet work. Give me a shout if you would like to do this.

Processing

Once you have created all the images, it’s best to process them a little further. At this stage, every image always uses up the full viewable area of the character. That’s ok, if all you want to do is lip-sync and possibly a little animation during idle times. But in this setup, your character can’t blink while speaking. To accomplish this, we need to cut out only the valid parts of each image and make the rest transparent. This way, the system can overlay multiple images and create the illusion of different parts moving at the same time.
Another added advantage of this process is that it usually (depending on the file type) shrinks the file. PNG files, when edited with the correct tools, will use less space if there is a bigger transparent area.

There are multiple ways you can crop the images. You can use the eraser tool found in most bitmap editing tools like paint.net.  Though, I’ve noticed that at least some versions of Photoshop don’t shrink png files when using the eraser, so that’s maybe not the best tool for the job. Personally, I used a little home-made app to do the job. You can download it from here.

It’s very simple to use. As you can see on the screenshot, there are 2 big buttons to the left which contain the source images. Press on each button to load the images. The top image should contain the root, in the bottom, you put the images that need to be cropped. To the right, on top, you see the result for the currently selected target image (seen in the bottom button). With the slider, you can scroll through the loaded images.
On the next line, there is a checkbox and a slider. These control the calculation process: do you want the parts of the image that are different or the same, and with the slider you select the tolerance level used to calculate the difference. When you put the slider fully to the left, there is 0 tolerance, meaning that there can be no difference between the pixel in the source image and that of the image that needs cropping. Put the slider fully to the right and the difference has to be very big.
Finally, with the button labeled ‘save’, you can save the processed images to a directory that you select (the original files are kept).

Depending on the way that the images are rendered in the 3D package, the file format and the quality of the rendering engine, there can be a bigger or smaller difference between parts of the images that should be the same. That’s why you normally have to play a little with the tolerance level. The lower, the better. Sometimes it’s better to keep a lower tolerance level and manually remove the remaining dots with an eraser in a bitmap editing package.

CCS file

When you’re ready with your images, it’s time to put them all together, a bit like a collage. Unfortunately, there currently isn’t yet a ready-made tool for this, so you are gonna have to do a little xml writing. Fortunately, there’s a lot of copy paste involved. The outer xml tag, the start of the file, is a <Character> tag, like so:

<?xml version=”1.0″ encoding=”utf-8″?>
<Character xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” xmlns:xsd=”http://www.w3.org/2001/XMLSchema”>
</Character>

Now, Let’s go over each section in order of the file content:

Character Info

The first section, and the easiest, is the character information. This is how it looks like:

  <CharacterInfo>
<Name>Mika</Name>
<Author>Ady Di Pierro</Author>
<Copyright>Copyright 2011 Ady Di Pierro</Copyright>
<License />
<AuthorWebsite>http://www.laticisimagery.com.au/</AuthorWebsite>
<CreationDate>2011-08-07T14:43:26.602+02:00</CreationDate>
<LastUpdateDate>2011-08-07T14:43:26.602+02:00</LastUpdateDate>
<Rating>
<Rating>Unknown</Rating>
<Sexual>false</Sexual>
<Violence>false</Violence>
<Other>false</Other>
<Description />
</Rating>
<OnlineOptions>
<OnlineCharacterBaseUrl />
<PreferredWidth>150</PreferredWidth>
<PreferredHeight>150</PreferredHeight>
</OnlineOptions>
</CharacterInfo>

Just copy the above and paste it into your xml file, just under <Character>. This part only contains reference information about the character: what’s it’s name, who designed it, what’s the licence, possibly a website,… All of the info in this section is optional, you can leave the tags empty, but it’s best to include them. This information, by the way, is displayed in the popup window on the ‘chatbot window’ (the button in the lower-left corner on the images window).

Online options are currently still skipped but will probably be used in future, online versions.

Background

<Background>
<ImageResource>images\Kima00.png</ImageResource>
<ImageResource>images\KimaEars.png</ImageResource>

</Background>

The background section defines the images that should be used as background (obviously). 2 things worth mentioning: a background is defined as an ImageResource. This is used often throughout the file. Whenever you want to reference an image file, you use this element. The text part of the xml element defines the relative path to the image (that is relative to the CCS file).

Also, and that’s perhaps the weirdest part, you can declare multiple backgrounds. Every image in this list will be displayed as a background, unless an animation turned one of the images off. And that’s the main usage of having multiple backgrounds: so that animations can turn part of the background off while playing. A good example are Mika’s ears.  They are drawn in a separate background image, so that the ‘flip-ears’ animation, can hide the background-ears while it’s playing. We do this cause some images in the animation sequence are smaller then the ears in rest, which would otherwise give ugly results with half an ear overlapped and the rest still visible. I’m certain there are plenty of other cool tricks to done with this feature. Hiding a background image is explained in the ‘Animations’ section.

Animations

The ‘Animations’ section is one of the bigger parts of the file. This is where you declare all the animation sequences available to the character for emotional expressions and idle times. The background animations (which run all the time, not jus at idle times), are declared somewhere else. Anyway, here’s how an animation definition looks like:

  <Animations>
<Animation>
<AnimationFrames>
<AnimationFrame>
<Duration>5</Duration>
<ImageResource>images\other\Kima Ears (01).png</ImageResource>
<VisemeGroupName/>
</AnimationFrame>
<Duration>5</Duration>
<ImageResource>images\other\Kima Ears (01).png</ImageResource>
<VisemeGroupName />
</AnimationFrame>
</AnimationFrames>
<EnableFrameSpeaking>true</EnableFrameSpeaking>
<HoldLastFrameForSpeak>false</HoldLastFrameForSpeak>
<FirstFrameUnderlay>false</FirstFrameUnderlay>
<BackgroundSuppress>KimaEars.png</BackgroundSuppress>
<Name>earsmove1</Name>
<ZIndex>0</ZIndex>
</Animation>
</Animations>

The section starts with an <Animations> element, which can contain 0, 1 or more <Animation> elements. Each animation defines a series of ‘frames’, where a frame represents a single image in the animation. A frame contains a Duration section, an ImageResource and a VisemeGroupName. The last one, you can forget about, it’s there because of backward compatibility reasons with the original CCS file format and should always be empty (for now). We’ve already been over the ImageResource element and the duration simply declares how long that the image should be displayed. This is expressed in milliseconds.

Underneath the frames, you need to declare some more info about the animation. Besides the name of the animation, which is used to start the animation in a rule’s output patterns, you have:

It might seem a tremendous task at first to declare every frame in an animation like this. But it turns out that most animations can be build using between 3 and 7 images, sometimes reusing images to build sequences of about 10-20 frames. So all in all, this is still manageable.

VisemeGroups

The next section declares the images used for lip-syncing. The basic structure looks like the following xml snippet (note that it doesn’t contain an entry for all 22 viseme images, just the first 2).

  <VisemeGroups>
<VisemeGroup>
<Name>Default</Name>
<ZIndex>1</ZIndex>
<VisemeImages>
<VisemeImage>
<VisemeIndex>0</VisemeIndex>
<ImageResource>images\Kima00.png</ImageResource>
</VisemeImage>
<VisemeImage>
<VisemeIndex>1</VisemeIndex>
<ImageResource>images\Visemes\Kima03 EH.png</ImageResource>
</VisemeImage>
</VisemeImages>
</VisemeGroup>
</VisemeGroups>

The file format already allows for multiple viseme groups to be declared, although at the time of writing, only the first one is used (and supported). Multiple viseme groups could be useful for moving heads: a viseme group for each head position.  As such, the ‘name’ element for each group would be used to reference a group. But, as already mentioned, this is something for the future.

The ZIndex element defines the Z-order that should be applied to the viseme images. This allows you to manipulate the order of the images. For instance, you could use this to move a part of the background image on top of the viseme images.

Next comes the ‘VisemeImages’ group, with a ‘VisemeImage’ for each mouth position + optionally 1 extra image for the silence position.  Each VisemeImage contains an ImageResource (as described above) and an Index (VisemeIndex), which determines the letter that the image represents (so the order in which the VisemeImages are declared, is irrelevant).

The silence is primarily for backward compatibility with verbot characters (which don’t have a separate background section). If you have a ‘Background’ section, the viseme image at index 0, will be skipped, otherwise it’s used as the background. That’s why the ‘background’ section has to be declared before the VisemeGroups.

IdleLevels

Idle time is the time when a bot doesn’t have anything to say or any emotion to show. In other words, nothing’s happening. In order to create the illusion of being alive during this period, you can use idle levels to start animations (that were declared in the ‘Animations’ section) when the bot is idle. Here’s the definition:

  <IdleLevels>
<IdleLevel>
<MinStartDelay>5</MinStartDelay>
<MaxStartDelay>15</MaxStartDelay>
<MinDuration>5</MinDuration>
<MaxDuration>15</MaxDuration>
<MinInterval>2</MinInterval>
<MaxInterval>8</MaxInterval>
<AnimationNames>
<AnimationName>earsmove1</AnimationName>
<AnimationName>eyes squint</AnimationName>
</AnimationNames>
</IdleLevel>
</IdleLevels>

Like most other sections, you can again declare multiple IdleLevels. In this example, we only have 1 though. The idea behind multiple IdleLevels is this: when the idle time starts, the first idle level is activated, but when the duration of the level has ended, the next one is activated until the last one is reached, which remains running until some activity happens. This way, you can have a bot act differently after x amount of idle time, progressively.

Each idle level contains 4 bits of information: 3 time ranges and a list of animation-names that the idle level can use. Each range has a Min and Max component, indicating the lower and upper part of the range. The different times are used for:

BackgroundAnimations

Idle levels are very useful to create a sense of liveliness, but they can be expanded upon. Sometimes, it’s also useful to have animations run all the time in the background, even while speaking. A good example of this could be breathing or eye blinking. For this purpose, there is the final section, called ‘BackgroundAnimations’, which contains a series of special animation definitions, all of which will run all the time, in a loop. Here’s the definition:

  <BackgroundAnimations>
<Animation>
<AnimationFrames>
<AnimationFrame>
<Duration>10</Duration>
<ImageResource>images\other\Kima Nose Flare (01).png</ImageResource>
<VisemeGroupName/>
</AnimationFrame>
<AnimationFrame>
<Duration>10</Duration>
<ImageResource>images\other\Kima Nose Flare (02).png</ImageResource>
<VisemeGroupName />
</AnimationFrame>
</AnimationFrames>
<Name>nosebreath</Name>
<LoopStyle>VarTimer</LoopStyle>
<MinStartDelay>1</MinStartDelay>
<MaxStartDelay>3</MaxStartDelay>
<ZIndex>2</ZIndex>
</Animation>
</BackgroundAnimations>

Unlike idle levels, the animation is declared inline this time.  That’s because there is a small difference in declaration between this type and idle/emotion animations. Another reason is to make certain that background animations can’t be used as emotions (they run all the time anyway, no need to start them separately).

The way that the individual frames are declared is the same as with regular animations: a list of ‘AnimationFrame’ objects that define the image, the duration and an optional viseme group. The difference is in the extra options: there is an extra ‘LoopStyle’ and ‘StartDelay’ range. ZIndex is also the same as in animations: It determines the Z-Order at which the animation is displayed, the higher the number, the more to the top it will be.

The ‘StartDelay’ range is used to build in a small idle time between each animation loop. Each time that the animation needs to be started, a value is selected at random from within the range. That’s the delay which will be used. Not every loop-style makes use  of this delay, some do, others skip it.

Finally, the LoopStyle element can have the following values:

And that’s it. Once you’ve got all these sections laid down, you’ve got yourself a character. All that remains, is to copy the files to the {my documents}\NND\Characters dir and restart the chatbot designer. Enjoy.

Short term memory

Or also called ‘volatile’ as it’s content is lost in time, can be accessed through the use of variables in the patterns.  Input patterns support 2 types of variables: regular variables, which can collect any type of content and thesaurus variables, which can only collect words that are equal to or are children of the thesaurus item referenced by the variable (in other words, they are filters). Asset variables, a third type of variable, is theoretically also possible, but not yet implemented. These are also filters, like thesaurus variables, except that they filter on concrete (asset) data instead of abstract (thesaurus) data.

The basic usage of this short term memory is simple: to provide a mechanism for collecting values of variable parts in the input patterns so that these values can later be used in the output and do-patterns for providing a response and feeding the long term memory.

Regular variables

As already mentioned, regular variables can’t filter on the values that they collect, but they are optionally able to limit the number of words that they collect, either as a specific number or a range. Here are some short input-pattern examples:

I’m called $var[.]

I’m called $var:1[.]

I’m called $var:1-3[.]

I’m called $var:4:CollectSpaces[.]

Copy $from:collectSpaces to $to:collectspaces

The $var constitutes the variable (‘$’ is the variable operator, followed by the name). Every word (except spaces, if ‘CollectSpaces’ is not specified) is collected by this variable until the pattern matcher finds a word in the input that follows the variable in the pattern or until the range is fully used. This variable can then be used in the output and do-patterns of the same rule (and also from other rules, if you are certain that the input-pattern is part of the result set, which can be checked upon, more on that  later).

Thesaurus variables

Thesaurus variables are a special type of input variable: they provide a mechanism for filtering possible input to a sub-branch of the thesaurus. If the input can’t be found in that branch, the pattern wont be activated. So, this is a filtering mechanism. The actual value that was found, can be accessed like any regular variable, through it’s name. There is no mechanism for providing length or range values though since they have no meaning here: it’s either an exact match with a thesaurus node or it isn’t. Here are some examples:

I’m called ^var:noun [.]

I’m called ^var:noun.name [.]

I’m called ^var:noun.(first name) [.]

I’m ^var:number years old[.]

A thesaurus variable always starts with  a ‘^’ followed by it’s name. The ‘:’ indicates the start of the thesaurus path and should always be followed with a POS (part of speech). These are the supported POS values:

noun, verb, adjective (or adj), adverb (or adv), article (or art), pronoun (or pron), conjunction (or conj), interjection (or inter), preposition (or prep), number, integer (or int), double.

You can stop there, which would indicate that you want any word of the specified part of speech. You can also continue the path with a ‘.’ followed by a text value (put into brackets if it’s multiple words).  This allows you to further refine the thesaurus path. Note that you don’t need to start at the root of the thesaurus that you are using, just as long as you are comfortable that it will point to a unique word within the tree (otherwise you can have multiple matches,… which might also be desirable). Note that the last 3 POS values (number, int and double) can’t have any further path specifiers, they have to stop at the POS value.

Collecting multiple values

It’s possible to use the same variable name multiple times in the same input pattern. This allows you to collect a list of values for the same variable. Thesaurus and regular variables can be intermixed. Here are some examples:

{$name ,} and ^name:noun.name are here  //catches something like: Tom, Flint and Warner are here

Note that there is a difference when a regular variable collects multiple words at a single location compared to when it collects single words at multiple locations in the pattern. When a single location collects multiple words, this group of words is combined into a compound word (as in ‘baby gear’), but when words are collected at multiple locations, a list is created. This list can later-on (in the long term memory) be labeled as AND, OR or LIST (unspecified).

Using short term memory

Up until now, we’ve only been talking about how to collect the values for the short term memory.  Of course, there’s no point in doing that unless you can actually do something with these values. That’s done in the output and do-patterns. As already mentioned, you access the content through the variable names. Here are some output-pattern examples:

Ok, I see, your name is $var\.

So you are $var, nice to meet you!

So, you can $verb:Infinitive, can you?

I see, $name:interleaf("\, ", " and ")

At it’s most basic form you specify the ‘$’ operator followed by the name of the variable that you want to render. Note that you should always use the ‘$’ operator while rendering, even if the value was collected using a thesaurus variable ( ^ ). This is because the ^ operator is used to access the long-term, abstract memory (the thesaurus data itself).

Rendering the value as it was collected, is useful but often we want to do a little more, sometimes we need to do some kind of change or transformation to the values, like conjugating a verb, get the plural of a noun or find the attribute for the value (see later),…  This is done through functions that you define in the path. A function starts with a ‘:’ followed by the name of the function (a list of all the available functions will come shortly) and optionally a list of arguments for the function, specified between brackets and separated by a ‘,’. Note: if you use the ‘,’ sign as an argument value, it must always be escaped with a \ Also, if you need to preserve spaces, the argument should be placed between brackets (as in the last example).

Long term memory

The second major type of memory that’s available to the bot is used to store and retrieve values so that they can cross the boundary of the single-shot input/response system, in other words: long term memory. Currently, there are 2 types: a thesaurus structure for storing abstract information and assets which maintain concrete knowledge.  Typically, you use this data to compare against short-term variables, render previously stored data or store newly acquired knowledge.

The thesaurus

As already mentioned, thesaurus variables are used in the input-patterns so that the valid content for a variable can be filtered.  When the ‘^’ operator is used in output, conditional or do patterns however, it behaves a little bit different: it becomes a value generator instead of collector.  Consider the following output patterns:

We are in ^noun.month[$time:month-1]

I like ^noun.food.(Italian food):random

I ^verb.be:conjugate(#bot) trying something complicated   //render: I am trying something complicate

As you can see, a thesaurus output-path contains a mix of statics and functions which eventually result into 0, 1 or 2 values. Because they render values and don’t collect it, no name is required. You can use the [] operator to select a child at a specific index position, like in the first example, which is used the generate the name of the month instead of a number. Note that the index is 0 based. In case that a static path item contains multiple words (like ‘Italian food’), use () brackets to group them. Also, if there are no values found for the path, any spaces that follow it in the output are stripped.

You can also store new data in the thesaurus. This is done in the calculation or do-sections. There are basically 2 operations that you can do at 2 different levels: you can add or remove values either as thesaurus children or as conjugations/references. To explain the difference between children and conjugations or references, take the following examples and how they are stored:

In the first example, we are declaring a child relationship: house is a building. If you have done any coding before, the syntax might be vaguely familiar: the left part of the statement contains the thesaurus path, the ‘+=’ operator to indicate that we want to create an ‘is child’ relationship, and on the right-side comes the value that needs to be stored. This could be a variable reference, an asset, another thesaurus path,….

The second and third examples look identical and for all intent and purpose, they are. The only difference is on the inside: in the first example ‘plural’ is a known conjugation form, ‘opposite’ is not. The statement used for storing this information, is a little bit different. First of, the thesaurus path ends with a ‘->’ followed by the name of the relationship that you would like to edit. Next, we use the ‘=’ assign operator instead of ‘+=’ to indicate that we want to change the relationship value.

The 2 last examples demonstrate what happens when you use the –> operator together with the += assignment or when you use multiple –> operators. When combining += with –>, you will first calculate the full result of the left side.  So in our example, we first take the singular value of ‘birds’, then we add a child to this result, which is ‘bird’. A similar thing happens when you use multiple –> operators: the value is calculated.

Except for the POS value at the start of the path, every other item in a thesaurus path can be a static, a variable reference, an asset path or another thesaurus path. This allows for tremendous flexibility in the way that you store data. We could generalize some of the previous statements like this:

^noun.building += $value

^noun.($singular)->plural = $value

^adj.good->($relationship) = $value

Removing values from the thesaurus is done using the ‘-=’ operator or by assigning to the ‘null’ value. Like with storing, all parts can be static or variable. This is probably best explained with some examples:

Assets

As already mentioned, assets could theoretically also be used in the input, but that’s not yet supported. If someone has a need for this, let me know, it’s not that tremendously difficult to add, it just creates a little more overhead.

Anyway, like thesaurus paths, asset paths can be used in output, do and conditional patterns. They are declared in much the same way as thesaurus paths by using the ‘.’ (dot) or ‘:’ (function) operators, except that they start with a # and ‘–>’ (links) are not supported.  For the thesaurus path, the ‘.’ (dot) operator selected a child node, for assets, this selects an attribute value. Here are a few output examples:

My name is #bot.name

your children are called #user.child.name:interleaf(“\, “, “ and “)

a book is made of $(#(^noun.book).component.name):interleaf(“\, “, “ and “)

Bot and User are hardcoded assets and refer to me and you respectively, from the bot’s point of view. In the third example, the first value in the asset path, is actually a thesaurus path. This results in concrete information about abstract data (a book is made of paper, ink, glue,…).

Also in the last example, the entire asset is the first value in a normal variable path, because an asset path will always calculate it’s result based on 1 value, if the previous path item resulted in multiple values (like ‘component), the next part of the path is calculated as if there was only 1 result (internally, a split is done), and only at the end of the path, all results are joined. This doesn’t work for ‘:interleaf’, it expects a list of values to combine. A variable path can do this, hence this construct.

To store asset data, the = (assign), += (assign add), != (assign not) and !+= (assign add not) operators are used. Removing data is done with the –= (assign remove) operator.   Take the following examples (input statement to the left, how to store/remove it to the right):

When you use the ‘=’ (assign) operator, you declare an ‘is’ relationship: ‘color’ becomes the attribute, ‘blue’ the value. Since ‘blue’ is not an asset, but just a word, we have a terminator: blue can’t have any more children.  But, there is a way to cross this border, by using the ‘:extra’ function, as in the 5th example.
If instead, you want to declare that something is not y, you can use the not-assign operator (!=). This allows you to still store the information that something is not. Be careful though, there is a thin line between being and not being, if you don’t check on this in the conditions, you might say that something is, while it isn’t (sounds familiar?).

The ‘+=’ or assign-add operator is used to create ‘has’ relationships, like in the second example. The major difference with the first one is that ‘dog’ becomes the attribute and the value becomes a new asset that will represent the dog. There is also the not version:  !+= which is used to indicate that the asset doesn’t have something.

If you want to create a list of values, you can use either the ‘;=’, ‘|=’ or ‘&=’ operators. The first one creates (or adds to) a generic list, the second is for OR lists and the last for AND lists. The generic list operator can add to any type of list without modifying it’s type. The |= and &= operators will change a generic list to OR and AND respectively. When you try to add an item to an OR list with an AND operator, you create a new list object that contains the original OR list and the newly added item. the same goes for an OR operator with an AND list.

Finally, you can also actually remove an attribute value. This is done with the ‘-=’ operator. The  right-side should be the name of the attribute that you want to remove. The value that is removed get’s cleaned up automatically, so if the value was another asset which isn’t referenced anymore after the remove, the entire asset will be destroyed. (Removing items from a list has to be done with the ‘:Remove’ function.)

As already mentioned, every asset value can always use the ‘:extra’ function to get to a sub-asset. There are a few other functions worth mentioning which allow you to expand the dataset that the asset can store. These are used to declare things like when, where, why, how, amount,… Functions are:

Mid term memory

Some functions, like the ‘:attribute’ function (which is able to extract, for instance,  ‘color’ from ‘blue’, or ‘name’ from ‘Jan), make use of context, if it is declared. This context is usually a list of asset paths that point to some memory region of the bot. The idea is that, together with a response, you also generate the meaning of what was said and store this information in the asset that you declared as context. If you refresh this context on each run, you effectively have simulated mid term memory.

The basic setup for using mid term memory consist out of:

• a global context declaration so that the system knows where to go look for contextual info.
• some global do-after-each-statement patterns. These are responsible for erasing the previously collected data and possibly creating an echo.
• some global do-on-startup patterns which will remove any data from the previous run.
• do-patterns on each rule to actually collect the knowledge about what is being said.

Context:

#bot.memory.subject

#bot.memory.attribute

#bot.memory.object

#bot.PrevMem.subject

#bot.PrevMem.attribute

#bot.PrevMem.object

 

Do after output:

#bot.prevmem = #bot.memory

#bot –= memory

 

do on startup:

#bot –= memory

#bot –= prevmem

 

on pattern (example pattern = I like $value [.])

#bot.memory.subject = #user

#bot.memory.attribute = like

#bot.memory.object = $value

 

Mid term memory also becomes very useful once you start working with recursive sub rules/topics. This technique allows you to rebuild the extracted data.

Patterns

All the different types of patterns (input, output, conditional) could also be considered as a form of long term memory. Internally, they are stored in exactly the same manner as all the other data. As such, they can also be manipulated in a similar manner as the other long term data. Although, at the time of writing, there is still limited support for this.  More on that to come.