Updated: 02/06/2018 - v.0.4

Main page | Remindmebot on Facebook | Remindmebot - Github

Received a task:

"Create chatbot using Node.js, which allows a user to conduct all his reminders - create/delete and reminders alert with an ability to confirm/snooze the reminder.
Bot should use LUIS or Dialogflow API for NLU, and bot application must be the only one on a Facebook Page. You should use LUIS/Dialogflow as an API only (use for defining intents and save context).
Bot should have “Getting started" feature and use a persistent menu for a basic operation like - create a reminder, show all reminders for today etc.
Bot should use structured messages to interact with a user, for example, reminders alert message should be a ""Generic template"" with accept/snooze buttons.
You should use ESlint(Airbnb code style config), npm scripts, database (PostgreSQL or MongoDB), environment variables (Dotenv)."


Knowledge before starting the project:

  • Total coding experience: ~7.5 months / 450 hours, Python
  • MongoDB – 2/10: (using pymongo wrapped for Python; used in 3 miniprojects – 1, 2, 3)
  • Dialogflow API – 2/10 (using apiai wrapper for Python; 1 miniproject)
  • Chatbots for FB Messenger – 1/10 (1 chatbot as an integration from Dialogflow, with webhooks on Python)


Time Limits:

Not strict but desired in 1 week. If by Thu., 05/24/2018: ~40 hours (5h/d * 5 weekdays + ~7h/d * 2 days off) because of current job and family ;)


Feeling like

Summary for Day 1

Set up a general plan of work and preliminary dialog flow for the bot
Installed node.js, express (+ request, apiai), WebStorm
Got "Hello world" on JS ;)
Created a FB page and app, got FB credentials
Created Dialogflow's agent
Following manual - https://medium.com/crowdbotics/how-to-create-your-very-own-facebook-messenger-bot-with-dialogflow-and-node-js-in-just-one-day-f5f2f5792be5
Uploaded code to github - https://github.com/IuriiD/remindmebot.git
At the moment messages don't come from FB Messenger to local webhook



9:12
Planning..

1) "Hello world" on JS (Node.js)
2) Write functions:
a. create_reminder(),
b. delete_reminder(N),
c. clear_all_reminders(),
d. show_all_reminders() # for today
e. confirm_reminder(),
f. snooze_reminder() # change time +X min (for eg., 15min)
g. alert_reminder()
3) Create a bot on Dialogflow (DF), get credentials
4) Communication with DF via API, triggering functions

9:30
5) Contexts and dialog flow:
a. INPUT_CONTEXT = []
Greeting (Getting started" feature?): Good day [morning/day/evening], . I'm here to keep track of your reminders. What would you like to do next?
Buttons:
Display my reminders for today
Add a reminder
Let's talk/Tell me a joke ;) (SmallTalk/etc, optionally)
OUTPUT_CONTEXT = []

b. INPUT_CONTEXT = []
User clicked "Display my reminders for today" - trigger show_all_reminders()
Buttons:
Add a reminder
Delete reminder
Clear all reminders
OUTPUT_CONTEXT = [reminders_shown]

c. INPUT_CONTEXT = [reminders_shown]
User clicked "Delete reminder" – ask user to enter a reminder number (integer value will be expected from DF)
Buttons:
Cancel
Main menu (Display…/Add a reminder/…)
OUTPUT_CONTEXT = [which_reminder_to_delete]

d. INPUT_CONTEXT = [which_reminder_to_delete]
User entered something
Integer number is expected – if not, tell user that's not a number and to ask to repeat + "Cancel" button
In range of reminders quantity – if not, tell user that value entered is incorrect and ask to repeat + "Cancel" button
If Ok – trigger delete_reminder(N), trigger show_all_reminders() (for today)
Buttons:
Add a reminder
Delete reminder
Clear all reminders
OUTPUT_CONTEXT = []

e. INPUT_CONTEXT = []
User clicked "Add a reminder" – ask user smth like "What to remind you about?" sys.any input is expected
Buttons:
Main menu (Display…/Add a reminder/…)
OUTPUT_CONTEXT = [reminder_content_input]

f. INPUT_CONTEXT = [reminder_content_input]
User entered something (anything) – tell smth like "Got it!"
Hold reminder content
On which date/time should I set this reminder? – date + time input is expected from DF
Buttons:
Cancel (go to main menu)
OUTPUT_CONTEXT = [reminder_time_input]

g. INPUT_CONTEXT = [reminder_time_input]
If not correct – "Sorry but that date/time seems to be incorrect. Could you please enter a valid value? Thanks ;)"
Buttons:
Cancel (go to main menu)
OUTPUT_CONTEXT = [reminder_time_input]

h. INPUT_CONTEXT = [reminder_time_input]
If date/time Ok - trigger create_reminder(), save data to DB
Inform user "A new reminder 'XXXX' for was successfully created. What should I do next?"
Buttons:
Display my reminders for today
Add a reminder
Let's talk/Tell me a joke ;) (SmallTalk/etc, optionally)
OUTPUT_CONTEXT = []

10:00
Draw a paper scheme of actions

10:30
6) Create a FB page and app, get credentials
Backend – FB Messenger interaction

Reading
How To Create Your Very Own Facebook Messenger Bot with Dialogflow and Node.js In Just One Day
Building a Facebook Messenger Bot Using Node.js, Heroku & API.AI

11:00
Created a FB Page: https://www.facebook.com/Remindmebot-191533734823967
Created a FB Messenger bot – Remindmebot
Generated FB page access token: ####
FB App – Callback ULR - https://709fd61d.ngrok.io
Verify Token – remindmebot
Messages, postbacks

11:30
Trying to install express.js on Ubunty in Vagrant or on Windows
npm install express --save
Getting CERT_UNTRUSTED error on Ubuntu
https://www.digitalocean.com/community/tutorials/how-to-install-node-js-on-ubuntu-16-04
https://expressjs.com/en/starter/installing.html
github - https://github.com/IuriiD/remindmebot.git
Bypassing https helped (https://stackoverflow.com/questions/21855035/ssl-error-cert-untrusted-while-using-npm-command?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa)
npm config set strict-ssl false
npm install express --save
Reading “Hello World!” app with Node.js and Express

12:00
Failing to get 'Hello world' work
var express = require('express');
var app = express();
app.get('/', function (req, res) {
res.send('Hello World!');
});
app.listen(3000, function () {
console.log('Example app listening on port 3000!');
});

Didn't work in Ubuntu (in Vagrant on Windows) but works from Windows
Getting Cannot GET / error on localhost:3000
Installing WebStorm (worked in PyCharm ;)

12:30
So:
Hello world from manual (https://medium.com/@adnanrahic/hello-world-app-with-node-js-and-express-c1eb7cfa8a30 ) works, including via ngrok (https://709fd61d.ngrok.io/)
Code from manual (https://medium.com/crowdbotics/how-to-create-your-very-own-facebook-messenger-bot-with-dialogflow-and-node-js-in-just-one-day-f5f2f5792be5 ) gives Cannot GET / error (localhost:3000, https://709fd61d.ngrok.io/) >> failing to finish NewPageSubscription on FB

13:00
Article (https://medium.com/crowdbotics/how-to-create-your-very-own-facebook-messenger-bot-with-dialogflow-and-node-js-in-just-one-day-f5f2f5792be5) seems to be a bit incorrect(?)
Following comments, changed

const express = require('express');
const bodyParser = require('body-parser');
const app = express();
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));
app.listen(3000, () => console.log('Webhook server is listening, port 3000'));


to

const express = require('express');
const bodyParser = require('body-parser');
const app = express();
const verificationController = require('./controllers/verification');
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));
app.get('/', verificationController);
app.listen(3000, () => console.log('Webhook server is listening, port 3000'));

and NewPageSubscription on FB seems to have succeeded

Creating a bot on Dialogflow…
Remindmebot
Client access token: ###

13:43
npm install apiai —- save
npm install request —- save
Stop for today - back to main job

Codeacademy beginner's course on JS – from 14% to 56%
Spent a lot of time (2+ hours) struggling with software:
- Localhost:3000 in not accessible (failed to fix that; launching server at port 5000);
- It appeared that what I've installed using "apt-get install node" on Ubuntu is not Node.js – reinstalled that (not from the 1st try ;)
- Failed to get ngrok working on Ubuntu (but it works from Windows and that's enough for now)
- Bot recreated according to manual (https://medium.com/crowdbotics/how-to-create-your-very-own-facebook-messenger-bot-with-dialogflow-and-node-js-in-just-one-day-f5f2f5792be5) (run on port 5000, tunneled using ngrok) still doesn't respond to messages from FB Messenger = at the moment my backend doesn't respond to messages from FB
Wrote several functions on JS for working with mongoDB - createReminder(),checkForDuplicates(), deleteReminder()


9:25
Planning
Yesterday I also started an intro course on JS on Codeacademy (https://www.codecademy.com/learn/introduction-to-javascript), current progress = 14% (https://www.codecademy.com/gigaWhiz78105)

Plan for the day. 2 main directions:
A) JS – learn basics and try to write necessary functions (see "2)" from yesterday's plan) and
B) Achieve message exchange between FB Messenger, backend (on localhost using ngrok or on Heroku if local problems) and Dialogflow.

1. JS basics (https://www.codecademy.com/learn/introduction-to-javascript) for 1-1:30
2. Functions – JS/express and MongoDB
3. Back to chatbot and manual (https://medium.com/crowdbotics/how-to-create-your-very-own-facebook-messenger-bot-with-dialogflow-and-node-js-in-just-one-day-f5f2f5792be5), achieve at least successful messages exchange between FB Messenger, localhost (or maybe Heroku if unsuccessful) and Dialogflow
4. If #2 and #3 succeed – deploy functions to dialog flow between FBM and backend

10:00
https://www.codecademy.com/learn/introduction-to-javascript - progress 21%

10:30
https://www.codecademy.com/learn/introduction-to-javascript - progress 28%

11:00
https://www.codecademy.com/learn/introduction-to-javascript - progress 35%

11:30
https://www.codecademy.com/learn/introduction-to-javascript - progress 42%

12:05
https://www.codecademy.com/learn/introduction-to-javascript - progress 50%

12:39
https://www.codecademy.com/learn/introduction-to-javascript - progress 56%
Stopped at lesson #8 "Objects"

12:51
Looked through 2 articles on MongoDB and nodejs:
1) https://closebrace.com/tutorials/2017-03-02/the-dead-simple-step-by-step-guide-for-front-end-developers-to-getting-up-and-running-with-nodejs-express-and-mongodb
2) https://closebrace.com/tutorials/2017-03-02/creating-a-simple-restful-web-app-with-nodejs-express-and-mongodb

Reading:
Node.js MongoDB - https://www.w3schools.com/nodejs/nodejs_mongodb.asp
Installed MongoDB
npm install mongodb (on Ubuntu14.04 in VagrantBox on Windows)
Using Mongobooster

14:30
On Day1 I was using node on Windows but want to continue on Ubuntu (as with all my projects) (14.04; in VagrantBox on Win7).
But on Ubuntu after installing Node with "apt-get install node" command "$ node somefile.js" did nothing in shell. It appeared that node installed was wrong
Solution: https://stackoverflow.com/questions/24721182/when-i-run-node-nothing-happens-the-same-with-forever?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa
(uninstalled node, downloaded 64-bit binary for Linux from http://nodejs.org/download/, added PATH=$PATH:/vagrant/soft/node/bin)

Another local problem:
Can't reach service on 3000 port BUT can on port 5000 (for eg., in my python apps on Flask)

telnet localhost 3000
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused


lsof -i :3000 -n
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python 7236 root 4u IPv4 34677 0t0 TCP *:3000 (LISTEN)


Windows firewall turned off
Decided to launch chatbot server on port 5000

16:15
Back to MongoDB with JS (https://www.w3schools.com/nodejs/nodejs_mongodb_create_db.asp)
Wrote several functions - createReminder(),checkForDuplicates(),deleteReminder()
25% time spent but moving very slowly :(
The 3rd day was not very productive:
- finally sorted out with how to properly install nodejs/npm on Ubuntu
- JS/nodejs and MongoDB - wrote 3 functions (nextAlert(), clearAllReminders(), showAllReminders4Today())


9:02
Plans for the day:
- Deal with software problems (node/npm, node-mongodb)
- Continue with functions on js for working with mongodb
- Another try to get FB M and backend exchange messages

As I'm still having some problems with node/npm (for eg., getting "SyntaxError: Use of const in strict mode." when executing scripts that work with MongoDB; node v 0.10.25) – decided to reinstall Node.

How to Install and Use Node.js and npm (Mac, Windows, Linux)
https://www.taniarascia.com/how-to-install-and-use-node-js-and-npm-mac-and-windows/

On Win7 64bit installed node-v10.1.0-x64.msi
Node 10.1.0
Npm 5.6.0
Seems to be working Ok. But on Win I will also need to install MongoDB. On Ubuntu (in Vagrant) I already have it and also I'm doing all other projects there, so it would be better (and for perspective) to work in Ubuntu. So

Completly uninstall nodejs, npm and node in ubuntu 14.04
https://stackoverflow.com/questions/32426601/completly-uninstall-nodejs-npm-and-node-in-ubuntu-14-04?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa
Currently it shows node/nodejs v. 0.10.25, npm v.1.3.10
sudo apt-get remove nodejs

sudo apt-get remove npm
sudo apt-get update
And then back to article https://www.taniarascia.com/how-to-install-and-use-node-js-and-npm-mac-and-windows/
Installation using Node Version Manager (NVM)
Now using node v10.1.0 (npm v5.6.0)

10:11
Recreate a project
Later will try to configure node interpreter from Vagrant in WebStorm (https://www.jetbrains.com/help/webstorm/configuring-remote-node-interpreters.html) – so far will use WebStorm only for writing code, execution from shell
"dependencies": {
"apiai": "^4.0.3",
"body-parser": "^1.18.3",
"express": "^4.16.3",
"mongodb": "^3.1.0-beta4",
"request": "^2.86.0"
}
Functions for MongoDB seem to be working Ok

10:40
One more quick try to get this working https://medium.com/crowdbotics/how-to-create-your-very-own-facebook-messenger-bot-with-dialogflow-and-node-js-in-just-one-day-f5f2f5792be5
One more try to get ngrok working on Ubuntu in Vagrant
https://www.npmjs.com/package/ngrok
npm install ngrok and some other variants fail
only this one worked:
https://github.com/inconshreveable/ngrok/issues/429
sudo npm i -g ngrok --unsafe-perm=true --allow-root
https://1627ad0b.ngrok.io

So,
1) I'm following the manual https://medium.com/crowdbotics/how-to-create-your-very-own-facebook-messenger-bot-with-dialogflow-and-node-js-in-just-one-day-f5f2f5792be5
2) My webhook is listening on port 5000, localhost:5000 tunnelled with ngrok to https://1627ad0b.ngrok.io (can be tested in browser)
3) Messages from FB M still don't come :(

11:40
Back to js and mongodb

15:47
Off-topic: https://da-14.com/blog/python-vs-nodejs-which-better-your-project

19:27
Wrote functions:
nextAlert()
clearAllReminders()
showAllReminders4Today()
Continued with codecademy's cource on javascript (57%>>78%)
Finally managed to get the bot built according to manual https://goo.gl/Fi7Zwo work (get messages from FB Messenger, pass them to Dialogflow and return the response)


9:20
Plan for the day:
- Continue with functions to deal with mongoDB
- Test another example of chatbot for FBM on Node.js

9:34
Following article https://www.sitepoint.com/building-facebook-chat-bot-node-heroku/
Ok, this thing at least responds (though with ' Sorry, I don't understand your request.') though some moments didn't work (like adding a Get Started button – gives response
Sorry, something went wrong. We're working on it and we'll get it fixed as soon as we can.
Next try in this direction will be to combine this manual with that previous one (https://medium.com/crowdbotics/how-to-create-your-very-own-facebook-messenger-bot-with-dialogflow-and-node-js-in-just-one-day-f5f2f5792be5)

12:08
https://www.codecademy.com/learn/introduction-to-javascript - progress 64%

13:24
https://www.codecademy.com/learn/introduction-to-javascript - progress 71%

15:38
https://www.codecademy.com/learn/introduction-to-javascript - progress 78%

Bot built according to manual https://medium.com/crowdbotics/how-to-create-your-very-own-facebook-messenger-bot-with-dialogflow-and-node-js-in-just-one-day-f5f2f5792be5) is finally working! My guess is that (what is NOT said in the article) in "Webhooks" section app should be subscribed to the page.
Spent the day (~7:00) rewriting [createReminder()] and writing functions to work with DB [ifReminderIsToday(), showAllReminders4Today(), snoozeReminder()]. Spent >1.5h trying to determine strange behavior of one function working with DB before came to know that it's because of asynchronous mechanism of JS (as distinct from Python).
Played a bit with code for the bot analyzing messages coming from FB Messenger and Dialogflow.


Tasks for the day:
- Finish with functions
- Structured messages on FB, dialog flow

9:01-17:00
Coding functions - new ifReminderIsToday(), showAllReminders4Today(), snoozeReminder()
Resources used:
https://www.w3schools.com/nodejs/nodejs_mongodb.asp
https://www.w3schools.com/jsref/jsref_obj_date.asp
etc.
Finished writing functions. Learned to use promises to make functions execute in the proper order. Bot can display reminders for today, add reminder, small-talk, clear all reminders (almost finished).

Plan for the day:
- Finish with functions
- FB Messenger – structured messages, connect to backend

9:20
Finished function snoozeReminder()

12:44
Spent ~2h reading about ways to deal with asynchronous way JS works (query to DB finishes after functions that should present results of this query).
Reading:
https://developers.google.com/web/fundamentals/primers/promises

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise

https://stackoverflow.com/questions/5010288/how-to-make-a-function-wait-until-a-callback-has-been-called-using-node-js?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa

https://github.com/luciotato/waitfor-ES6

https://github.com/kriskowal/q
Sleepy…

13:20
Finally managed using this example https://pastebin.com/s96L3kPp and one my brother's script.

16:40
Bot can display reminders for today, add reminder, small-talk, clear all reminders (almost finished).
Using text commands bot can:
- Show reminders
- Add reminders
- Clear all reminders
- Delete a specific reminder
Added "Get started" button (without handler so far) and Greeting text. Adding Persistent Menu using CURL request and API Explorer gives "result: success" but persistent menu doesn't show up. Read about generic messages.


Plan for the day:
- Continue 'connecting' backend to FB Messenger (text commands, structured messages, persistent menu, getting started feature)

8:50
Morning reading (unread yesterday/earlier):
https://learn.javascript.ru/promise
btw nice resource https://learn.javascript.ru/

16:46
Using text commands bot can:
- Show reminders
- Add reminders
- Clear all reminders
- Delete a specific reminder
Problems with adding persisntent menu:
https://stackoverflow.com/questions/44338774/facebook-new-nested-persistence-menu-not-working?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa
>>
https://i.stack.imgur.com/WPHEY.png
>>
https://developers.facebook.com/tools/explorer/
I get "result: success" but don't see persistent menu in my chatbot
At the moment the bot looks like that:
Most features done. "Hello world" on JS was ~50h ago. What's left: alarming/confirming/snoozing reminders. Also so far I failed to add button or generic templates (don't know why – tried several variants) so for now I'm using quick replies. Hope to finish in 1d.

Tasks for the day:
- Persistent menu +
- Handling callbacks from "Get started"(+) and other buttons(+)
- Function for alerting reminders(-)
- Present [preliminary] results

9:28
Trying to handle button clicks by myself
Hey, persistent menu is working! Looks like it may need some time to appear(?)

14:30 Back to main work (5:30)
Plan for the day:
- Reminders alerting/confirming/snoozing;
- Bug fixing and improving;
- Presentation

09:06
Button template is working – found a silly syntax mistake.

18:12 (~9:00)
I did my best to finish a bot with main features requested today but unfortunately failed. Up to 3:00 were spent trying to "promisify" a rather complicated function with nested loops and DB requests. Finally I did that. At this moment RemindMeBot is able to:
- Add reminders;
- Remove a chosen reminder;
- Clear all reminders;
- Alert reminders (checks are done every 1 minute);
- Has "Getting started" feature, persistent menu, uses button template.
Confirming/snoozing alerts is left – (buttons are done, their handling is done but I can't get updates to DB. Also text and button responses from the bot are sometimes mixed up though they are presented using promises (need to find out why).
Finished alerts snoozing/confirming. Main features are done.
Some additional features are needed (like checking the number of reminder to be deleted if it's in range of existing reminders etc) and checking/polishing and code cleaning.
Deployment to Heroku or my server.
Think to present results on Mon. (weekend might be not the best time ;)