edu / BeuthBot_WS17

BeuthBot_WS17

Ein enorm intelligenter, messenger-unabhängiger Chatbot.

1
skipped 993096b5 Add final presentation (converted from google docs) · by Benjamin Rühl

BeuthBot

BeuthBot is a master project of the Beuth University of Applied Sciences Berlin. This implementation is the work of media informatic students from semesters SS2017 and WS2017/18.

Requirements

  • Java 8 to build the application
  • Docker daemon to start application server

Technologies

  • Oracle Java 8
  • Gradle Build - build system
  • PostgreSQL - relational database
  • Hibernate - object relational mapping
  • Adminer - database administration
  • WildFly 10
  • Docker

First steps

The following five steps clarify what is necessary to set up the BeuthBot for the first time. Most of the steps are also useful during development to ultimately test the changes in the application.

All commands are listed in the following five steps and can be executed from the project path. If this is different than this is indicated accordingly. The commands may differ depending on which operating system is used.

1. Set up integrated development environment

The project should be cloned from the repository. For development it is possible to use IDE Intelij. The cloned project can then be imported into the IDE as Gradle project.

2. Execute project build

With the following command it is possible to build the whole project. The construction of the project is possible during operation and before starting the server. In the first case the running server is automatically updated during operation.

    ./gradlew war

3. Run application

The application can be started with the following command:

    # Unix
    ./gradlew chatbotRun
    # Windows
    gradlew.bat chatbotRun
    # MacOS
    cd docker
    docker-compose up --build

4. Test application

After the application has started the following two endpoints of the server are easily accessible via the browser:

It is possible to use a Postman Collection to test the configured endpoints for Telegram and Facebook. The docu/postman/ directory includes a Postman Collection containing corresponding endpoints. The JSON file beuth-bot.json can be imported into Postman. If changes are made to the Postman Collection, the service of the JSON file should be increased accordingly.

In addition it is also possible to set up your own bot for example, in Telegram and test it with the local server. For this however a local tunnel must be set up to make the application visible to the outside. For details see the section Local tunnel.

5. Stop application

The following command can be used to stop the running application:

    # Unix
    ./gradlew chatbotStop
    # Windows
    gradlew.bat chatbotStop
    # MacOS
    cd docker
    docker-compose down

Project documention

In addition to this general documentation, the directory docu/ also contains several sections about the project documentation itself.

General

Individual modules

  • api.ai - Usage of API.ai natural language processing
  • attachment store - Storing files that come with chat messages
  • bing speech - Usage of bing speech API
  • canteen parser - Extracting data from the canteen HTML web page
  • drools - Manage business rules defining the chat bot's behavior
  • facebook - Facebook adapter, utils and Facebook app
  • rasa - Bringing data from API.ai to Rasa
  • telegram - Telegram bot API and Telegram adapter

Application properties

Settings for this application can be found in the beuthbot.proterties file. This is located in the directory docker/wildfly/volumes/conf/. The file is automatically entered in the Tomcat server during deployment and is used for configuration.

Docker configuration

The Docker configurations can be found in the docker-compose.yml file. This is stored in the directory docker/. Furthermore you can find the Dockerfile file in the folder docker/wildfly/. The following port mappings are set in the Docker files:

Backend

  • Internal port of the container: 8080
  • External port from the container: 8080

WildFly server manager

  • Internal port of the container: 9990
  • External port from the container: 9990

Database

  • Internal port of the container: 5432
  • External port from the container: 5432

Database administration

  • Internal port of the container: 8080
  • External port from the container: 8081

Localtunnel

Localtunnel is a great way to test the system using your preferred messenger. Both Facebook and Telegram require secure connections via HTTPS in order to transmit messages to a registered bot server. This is done by setting up a random domain name on the localtunnel server and linking it to your internet-ip address. For this process to work, you need administrative access to your wifi-router, where you need to activate port forwarding, and have port 8080 be forwarded to your local ip-address. Once this is setup, you can install and use localtunnel. Localtunnel requires NodeJS in order to work.

MacOS:

1.Installing NodeJS MacOS requires XCode and Homebrew in order to install NodeJS. XCode can be downloaded and installed for free via the App Store.

To install Homebrew, open a terminal window and type

    ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

After Homebrew has completed installation, install NodeJS by typing:

    brew install node

Finally install localtunnel by typing:

    npm install -g localtunnel

If everything has worked correctly, you are now ready to use localtunnel. Using localtunnel is fairly easy, simply type:

    lt --port 8080

Where the number entered after --port corresponds to the port you want localtunnel to route its HTTPS-Domain to. The localtunnel server can be slow and a little unreliable at times, but if everything goes well you should see something like:

    your url is: https://uawfhausjdj.localtunnel.me

which is the url you can put in the first line of the beuthbot.properties file in the /docker/wildfly/volumes/conf` directory.

IMPORTANT: The terminal window in which you launched localtunnel needs to be kept open in order for it to work! Closing the terminal-window will kill your localtunnel-connection.

NOTE: Localtunnel tends to crash every now and then, displaying an error message that the connection was refused. In this case, run the localtunnel command again to get a new address.

Debug application

At the moment, the JBoss server is always started with open debugging ports. To attach to the running docker container process using IntelliJ, a remote debug configuration has to be created. To do so, perform the following steps:

  • Go to Run -> Edit Configurations...
  • Add a new configuration using the button in the top left corner of the dialog
  • Choose Remote
  • Assign a name, ignore the arguments section and enter port 8787 (localhost) in the settings section
  • click OK and select the new configuration at the top right of the toolbar if it has not been selected automatically
  • run the configuration using the Debug button

Deploy application

The following paragraph describes how the deplyoment is performed from a local Git repository to the production server. All development statuses have to be transferred to the master branch before performing the deployment. Just from the master branch it is possible to carry out the deployment on the productive server. To be able to perform the deplyment, it is necessary to be able to access the server via SSH. Then Git will transfer the status of the master branch to the server. This requires customizations on the server and in the local Git.

1. Setup on the server

On the server the following Git command have to be executed in the directory home/<user>/. Instead of the placeholder <projectname> the actual name of the project has to be used. The command creates a Git directory which includes all necessary files for a Git Hook.

git init --bare ~/<projectname>.git

After the Git folder has been successfully created, a new file with the name post-receive has to be created in the subfolder <projectname>.git/hooks. Afterwards the script has to be inserted into this file and the placeholders get changed by project-specific variables. The script will also start the deployment.sh script. This executes the build of the project and starts the appropriate Docker command. In addition, this script ensures that the beuthbot.properties file on the server is replaced by the corresponding productive file. In local development the post-receive and deployment.sh file can be found in the scripts/ folder.

#!/bin/bash
while read oldrev newrev ref
do
    # only checking out the master (or whatever branch you would like to deploy)
    if [[ $ref =~ .*/master$ ]];
    then
        echo "Master ref received.  Deploying master branch to production..."
        git --work-tree=/home/<user>/<projectname>/ --git-dir=/home/<user>/<projectname>.git/ checkout -f
        cd /home/<user>/<projectname>/
        ./scripts/deployment.sh
    else
        echo "Ref $ref successfully received. Doing nothing: only the master branch may be deployed on this server."
    fi
done

In addition a folder with the project name <projectname> has to be created on the server in the directory home/<user>/. This is where the code from the master branch is pushed.

mkdir ~/<projectname>

If the URL of the BeutBots API changes, the variable WEB_URL in thebeuthbot.proterties has to be adjusted. In addition the call of setWebhook methods for Facebook and Telegram is necessary. The corresponding endpoints can be found in the Postman Collection.

2. Use in a local development environment

This step sets up a reference to the server on which the application is deployed. For this the following command must be executed. The placeholder <user> has to be replaced by the matching SSH user and the project <projectname> that has been set up on the server.

git remote add production ssh://<user>@<ip>:<port>/home/<user>/<projectname>

After the Git remote settings have been made the current state of the master branch can be pushed to the server using the following command.

git push production master

Known issues

The following issues are currently known in the application:

Gradlew

  • The ./gradlew chatbotRun command does not work on Mac OS X
  • The general build command ./gradlew build causes errors

Wildfly

  • During the boot process of the Wildfly server, the error Mount point not found occurs, wich does not seem to have any impact on the running system

Used online sources