Recording Brain Waves -- Mongo Database with a NodeJS API

Saving Brain Waves to Remote MongoDB by way of Node REST API

In this section I’m going to focus getting a remote Linux server setup with MongoDB and NodeJS. This will allow us to make POST requests to our Linux server, saving the EEG data.

I’m going to assume you are able to SSH into your Ubuntu 16 LTS server for this guide. You don’t have a server? No sweat. I wrote a guide on setting up a blog post which explains how to get a cheap Linux server setup.

1. Install MongoDB

SSH into your server. I’m assume this is a fresh new Linux install. Let’s start with upgrading the packages.

sudo apt-get update -y

I’ll be following the Mongo website for instructions on installing MonogoDB Community version on Ubuntu.

Let’s get started. Add the Debian package key.

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4

We need to create a list file.

echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list

Now reload the database

sudo apt-get update

If you try to update and run into this error

E: The method driver /usr/lib/apt/methods/https could not be found.
N: Is the package apt-transport-https installed?
E: Failed to fetch https://repo.mongodb.org/apt/ubuntu/dists/xenial/mongodb-org/4.0/InRelease  
E: Some index files failed to download. They have been ignored, or old ones used instead.

Then install apt-transport-https

sudo apt-get install apt-transport-https

Now, let’s install MongoDB.

sudo apt-get install -y mongodb-org

Voila!

2. Setup MongoDB

We still need to do a bit of setup. First, let’s check and make sure Mongo is fully installed.

sudo service mongod start

This starts the MongoDB daemon, the program which runs in the background and waits for someone to make connection with the database.

Speaking of which, let’s try to connect to the database

mongo

You should get the following:

root@localhost:~# mongo
MongoDB shell version v4.0.2
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 4.0.2
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
	http://docs.mongodb.org/
Questions? Try the support group
	http://groups.google.com/group/mongodb-user
Server has startup warnings:
2018-09-02T03:52:18.996+0000 I STORAGE  [initandlisten]
2018-09-02T03:52:18.996+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-09-02T03:52:18.996+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-09-02T03:52:19.820+0000 I CONTROL  [initandlisten]
2018-09-02T03:52:19.820+0000 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-09-02T03:52:19.820+0000 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-09-02T03:52:19.820+0000 I CONTROL  [initandlisten]
---
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).

The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.

To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
>

This is good. It means Mongo is up and running. Notice, it is listening on 127.0.0.1:27017. If you try to access the database from any network, other than locally, it will refuse. The plan, to have NodeJS connect to the MongoDB database locally. Then, will send all of our data to Node and let it handle security.

In the Mongo command line type:

quit()

And hit enter. This should bring you back to the Linux command prompt.

A few notes on MongoDB on Ubuntu.

  • The congfiguration file is located at /etc/mongod.conf
  • Log file is at /var/log/mongodb/mongod.log
  • The database is stored at /var/lib/mongodb, but this can be changed in the config file.

Oh, and one last bit. Still at the Linux command prompt type:

sudo systemctl enable mongod

You should get back

Created symlink from /etc/systemd/system/multi-user.target.wants/mongod.service to /lib/systemd/system/mongod.service.

This setup a symlink which will cause Linux to load mongod every time it boots–you won’t need to manually start it.

Next, NodeJS.

3. Install NodeJS and npm

Type

sudo apt-get install nodejs -y

This should install NodeJS, but we also need the Node Package Managers npm.

sudo apt-get install npm -y

Let’s upgrade npm. This is important, as the mind-wave-journal-server depends on recent versions of several packages that are not accessible to earlier versions of npm.

The following commands should prepare npm for upgrading, then upgrade.

sudo npm cache clean -f
sudo npm install -g n
sudo n stable
sudo n latest

Let’s reboot the server to make sure all of the upgrades are in place.

sudo reboot now

When the server boots back up, ssh back in.

Check and make sure your mongod is still running

mongo

If mongo doesn’t start, then revisit step 2.

Let’s check our node and npm versions.

node -v

I’m running node v10.9.0

npm -v

I’m running npm v6.2.0

4. Clone, Install, and Run the mind-wave-journal-server

I’ve already created a basic Node project, which we’ll be able to grab from my Github account.

If you don’t already have git installed, let’s do it now.

sudo apt-get install git -y

Now, grab the Noder server I built.

git clone https://github.com/Ladvien/mind-wave-journal-server.git
cd mind-wave-journal-server/

Install all the needed Node packages.

npm install

This should download all the packages needed to run the little server program I wrote to store the EEG data into the Mongo database.

Let’s run the mind-wave-journal-server.

node server/server.js

This should be followed with:

root@localhost:~/mind-wave-journal-server# node server/server.js
(node:1443) DeprecationWarning: current URL string parser is deprecated, and will be removed in a future version. To use the new parser, pass option { useNewUrlParser: true } to MongoClient.connect.
Started on port 8080

5. Testing mind-wave-journal-server with Postman

Now, we are going to use Postman to test our new API.

For this next part you’ll need either a Mac or Chrome, as Postman has a native Mac app or a Chrome app.

I’m going to show the Chrome application.

Head over to the Chrome app store:

add-postman-chrome-app

After you add the Postman app it should redirect you to your Chrome applications. Click on the Postman icon.

run-postman-chrome-app

Your choice, but I skipped the sign-up option for now.

skipped-signup-postman-chrome-app

Select Create a Request skipped-signup-postman-chrome-app

The purpose of Postman, in a nutshell, we are going to use it to create POST requests and send them to the mind-wave-journal-server to make sure it’s ready for the iOS app to start making POST requests, saving the EEG data to our Mongo server.

Let’s create our first test POST request. Start by naming the request Test eegsamples. Create a folder to put the new request in, I named it mind-wave-journal-server. Then click

create-request-postman-chrome-app

You will need to set the type as POST. The url will be

http://your_ip_address:8080/eegsamples

create-request-postman-chrome-app

No select the Headers section and add the Content Type: application/json

create-request-postman-chrome-app

Lastly, select Body, then raw and enter the following JSON into the text area:

{  
   "highBeta":5,
   "lowGamma":6,
   "theta":55,
   "lowAlpha":2,
   "highAlpha":3,
   "lowBeta":4,
   "highGamma":7,
   "blink":55,
   "attention":8,
   "meditation":9,
   "time":4
}

And then! Hit Send

create-request-postman-chrome-app

If all goes well, then you should get a similar response in the Postman response section

create-request-postman-chrome-app

Notice, the response is similar to what we sent. However, there is the additional _id. This is great. It is the id assigned to the by MongoDB when the data is entered. In short, it means it successfully saved to the database.

6. Now What?

Several caveats.

First, each time you restart your server you will manually need to start your mind-waver-journal-server. You can turn it into a Linux service and enable it. If this interests anyone, let me know in the comments and I’ll add it.

Second, notice I don’t currently have a way to retrieve data from the MongDB. The easiest way will probably be using Robot 3T. Like the first caveat, if anyone is interested let me know and I’ll add instructions. Otherwise, this series will stay on track to setup a Mongo BI connection to the database for viewing in Tableau (eh, gross).

Your Node server is ready to be called by the iOS app. In the next article I’ll return to building the MindWaveJournal app in iOS.

Recording Brain Waves -- iOS SDK Setup

Step 1: iOS App

I’m going to assume you have Xcode installed.

Step 1.1: Install CocoaPods

CocoaPods is a package handler for Xcode. We will be using it to install Alamofire, which a Swift library for making HTTP requests. We will need HTTP call support as we will call our server to store the EEG samples.

sudo gem install cocoapods

After you hit Return it will prompt for your password

cocoapods-installation

Step 1.2: Setup Xcode Project

Now, let’s setup a project folder. This is main folder where all the iOS app code will live. It’s a bad habit, but I usually put mine on the Desktop.

Open Xcode and select “Create a new Xcode proejct”

xcode-project-start

Then select “Single View App” and click “Next”

xcode-project-start

Let’s call the project MindWaveJournaler and click “Next” xcode-project-start

Choose your Desktop as location for the project and click “Create” xcode-project-start

Step 1.3: Development Environment Setup

You’ve created a Project Folder, but we have to setup the project folder to be used with CocoaPods. After, we will use CocoaPods to install Alamofire.

Back in the terminal, type:

cd ~/Desktop/MindWaveJournaler
pod init

This creates a Podfile in the root folder of our project. We can list CocoaPod packages in the Podfile and run pod install in the same directory, this will cause CocoaPods to install all the packages we listed.

Sadly, we are really only doing this for Alamofire right now. But, later, when we start building on to this app it will allow us to quickly access third-party frameworks.

Ok, back to typing:

open -a Xcode Podfile

This will open the Podfile for editing in Xcode. Now let’s insert the our desired pod information.

Copy information below and paste it into your file:

# Uncomment the next line to define a global platform for your project
platform :ios, '11.4'

target 'MindWaveJournaler' do
  # Comment the next line if you're not using Swift and don't want to use dynamic frameworks
  use_frameworks!

  # Pods for MindWaveJournaler
  pod 'Alamofire', '~> 4.7'

  target 'MindWaveJournalerTests' do
    inherit! :search_paths
    # Pods for testing
  end

  target 'MindWaveJournalerUITests' do
    inherit! :search_paths
    # Pods for testing
  end

end

You may notice the only changes we made were

platform :ios, '11.4'
...
pod 'Alamofire', '~> 4.7'

These lines tell CocoaPods which version of iOS we are targetting with our app (this will silence a warning, but shouldn’t be required). The other, is telling CocoaPods which version of Alamofire we’d like to use on this project.

Ok, now let’s run this Podfile.

Back in the same directory as the Podfile type:

pod install

You should see CocoaPods do its thing with output much like below.

cocoapods-installed-alamofire

Step 1.4: Install NeuroSky iOS SDK

NeuroSky has a “Swift SDK.” Really, it’s an Objective-C SDK which is “bridged” into Swift. Essentialy, this means we won’t be able to see what’s going on the SDK, but we can use functions from the pre-compiled binaries.

I’ve not been impressed with NeuroSky’s website. Or the SDK. It does the job, but not much more.

Anyway, the SDK download is annoyingly behind a sign-up wall.

Visit the link above and click on “Add to Cart”

neurosky-sdk-sign-up

Then “Proceed to Checkout”

neurosky-sdk-sign-up

Lastly, you have to enter your “Billing Information.” Really, this is only your email address, last name, street address, city, and zip.

(Really NeuroSky? This is very 1990.)

Eh, I made mine up.

Anyway, after your enter information click, then click “Continue to PayPal” (What? I just provided my information…) You should be rewarded with a download link. Click it and download the files.

neurosky-sdk-sign-up

Unzip the files and navigate lib folder

iOS Developer Tools 4.8 -> MWM_Comm_SDK_for_iOS_V0.2.9 -> lib

Copy all files from the lib folder into the main directory of the MindWaveJournaler project folders.

neurosky-sdk-lib

Step 1.5: Workspace Setup

CocoaPods works by creating a .xcworkspace file. It contains all the information needed to compile your project with all of the CocoaPod packages installed. In our case the file will be called MindWaveJournaler.xcworkspace. And every time you want to work on your project, you must open it with this specific file.

It can be a bit confusing because Xcode created a .xcodeproj file which is tempting to click on. xcworkspace

Go ahead and open the MindWaveJournaler.xcworkspace file. The workspace should open with one warning, which we will resolve shortly.

But first, another caveat. CoreBluetooth, Apple’s Bluetooth LE Framework, only works when compiled for and run on an actual device. It does *not work in the iOS Simulator.* Once upon a time it did, if your Mac had the hardware, however, my version of the story is Apple didn’t like having to support the confusion and dropped it.

eeg-apple-workspace

Moving on. Click on the yellow warning. Then click on the warning in the sidebar. This should create a prompt asking if you’d like to make some changes. This should automatically make some tweaks to the build settings which should make our project mo’ betta.

Click Perform Changes. eeg-apple-workspace-resolve-warning

This should silence the warning and make your project error free. Go ahead and hit Play button and let it compile to the simulator (we aren’t testing the Bluetooth, so it’s ok). Everything should compile correctly, if not, just let me know the specifics of your problems in the comments.

Step 1.5: Enable Secure HTTP Request

There are still a few tweaks we need to make to the Xcode workspace to get everything working.

First, open the ViewController.swift file and add import Alamofire right below import UIKit. If auto-complete lists Alamofire as an option you know the workspace is detecting its presence. Good deal.

Now, for Alamofire to be able to securely make HTTP request an option needs to be added to the Info.plist file. I scratched my head as to why the HTTP calls were not being made successfully until Manab Kumar Mal’s StackOverflow post:

Thanks, buddy.

Ok, following his instructions open up the Info.plist file in your MindWaveJournaler folder. Now add an entry by right-clicking and selecting Add Row. Change the Application Category to NSAppTransportSecurity and make sure it’s set as dictionary. Now, click the plus sign by the new dictionary and set this attribute as NSAllowsArbitraryLoads, setting the type bool, and the value as YES.

eeg-apple-workspace-add-secure-layer

Step 1.5: Setup Objective-C Bridge Header for MindWave SDK

There’s a few other bits of housekeeping, though. As I mentioned earlier, the MindwAve SDK is in an Objective-C precompiled binary. It is usable in a Swift project, but requires setting up a “bridge header” file.

Start by creating the bridge header file. Go to File -> New -> File...

bridge-header-file

Then select Header and click Next.

bridge-header-file

Name the file YourProjectName-Bridging-Header and make sure the file is saved to the same folder which contains the .xcworkspace file, then click Create.

bridge-header-file

The header file should automatically open. Copy and paste the following to the bottom of the header file.

#import "MWMDevice.h"
#import "MWMDelegate.h"
#import "MWMEnum.h"

My entire file looked like this once done.

MindWaveJournaler-Bridging-Header.h

//
//  MindWaveJournaler-Bridging-Header.h
//  MindWaveJournaler
//
//  Created by Casey Brittain on 8/3/18.
//  Copyright © 2018 Honeysuckle Hardware. All rights reserved.
//

#ifndef MindWaveJournaler_Bridging_Header_h
#define MindWaveJournaler_Bridging_Header_h


#endif /* MindWaveJournaler_Bridging_Header_h */

#import "MWMDevice.h"
#import "MWMDelegate.h"
#import "MWMEnum.h"

Let’s tell the Swift compile we have a header file. In Xcode go to Project File -> Build Settings -> All then in the search box type Swift Compiler - General (if you don’t include the hyphen and spaces it wont find it).

bridge-header-file

Double-click on the line Objective-C Bridging Header directly underneath the name of your project (see red box in image). Copy and paste the following into the box and click off to save the change.

$(PROJECT_DIR)/$(PROJECT_NAME)-Bridging-Header.h

This creates a relative path to your Bridging-Header file. In a little bit we are going to try to compile, if you get errors around this file not being found, then it’s probably not named per our naming scheme (YourProjectName-Bridging-Header) or it wasn’t saved in the same folder as the .xworkspace file. No worries, if you have troubles just leave me a comment below.

bridge-header-file

One last thing to do before we’re ready to code. We still need to import the MindWave SDK into our project.

bridge-header-file

Right click on your project file and select New Group. Name the group MindWave SDK. Now right click on the folder you created and select Add Files to "MindWave SDK".... Navigate to the lib folder containing the MindWave SDK and select all files inside it.

mindwave-sdk

When you add the SDK, Xcode should automatically detect the binary file (libMWMSDK.a) and create a link to it. But, let’s make sure, just in case. Click on your project file, then go to the General tab.

mindwave-sdk

It needs to be linked under the Build Phases tab as well, under Linked Frameworks and Libraries.

mindwave-sdk

That’s it. Let’s test and make sure your app is finding the SDK appropriately.

Open the ViewController file and under viewDidLoad() after the existing code, type:

let mwDevice = MWMDevice()
mwDevice.scanDevice()

Watch for autocomplete detecting the existince of the MindWave SDK

mindwave-sdk

Now for the true test, Compile and Run. But, before we do, please be aware–this will only work on an actual iOS device. If you try to run it in the iOS simulator it will fail. It actually fails on two accounts, first, CoreBluetooth will not work in the iOS simulator, second, the MindWave SDK binaries were compiled specifically ARM architecture.

Ok! Enough preamble. Connect and select your iOS device and hit Run.

mindwave-app-run

If all goes well you should see two things. A blank white screen appear on your phone and concerning message in the Xcode console.

corebluetooth-error-api-misuse

The CoreBluetooth error has to do with firing up the iOS Bluetooth services without checking to make sure the iOS BLE is turned on and ready to go. This is a good thing, it probably means the MindWave SDK has been foudn and is functioning properly.

If you get any other errors, let’s chat. I’ll help if I can.

This is part of a series, which I’m writing with care as I’ve time. I’ll get the next part out ASAP.

Recording Brain Waves to MongoDB

Description

This project takes brain wave readings from a MindWave Mobile 2+, transmits them to an iOS app via Bluetooth LE. The iOS app makes calls to a remote Node server, which is a minimal REST API, passing off the brain wave sample. The Node server stores the data on a MongoDB server. The MongoDB server is then exposed to business intelligence applications use with MongoDB BI Connector. Lastly, using Tableau Professional Desktop, the data is accessed and visualizations created.

Whew.

To recap:

The end result is a system which could allow a remote EEG analyst to examine samples nearly in real time.

eeg-visualization

Below, I’m going to show how I was able to setup the system. But, before that a few words of warning.

Gotchas

Hacker Haters

This isn’t a hacker friendly project. It relies on several paid licenses, an Apple Developer License ($99) and Tableau Desktop Professional ($10,000,000,000 or something). Of course, the central piece of hardware, the MindWave Mobile, is also $99, but I think that one is fair. Oh! Let’s not forget, even though you bought an Apple Developer license, you still need a Mac (or Hackintosh) to compile the app.

However, as a proof-of-concept, I think it’s solid. Hopefully a good hacker will be able to see how several tweaks in the system could make it dirt cheap to deploy.

Mimimum Viable Hack..er, Product

The source code provided here is a minimally viable. Fancy words meaning, only base functionality was implemented. There many other things which could be done to improve each piece of the system.

Not to be a douche, but please don’t point them out. That’s the only thing I ask for providing this free information.

There are many improvements I know can be made. The reason they were not made had nothing to do with my ignorance (well, at least a majority of them), but rather my time constraints.

I Hate Tableau

That’s it. I hate Tableau.

Getting Started

Let’s make a list of what’s needed before beginning this project.

Regarding the business intelligence platform–if anyone has a free suggestions, please leave them in the comments below. The first improvement I’d like to the entire system is to get away from Tableau. Have I mentioned I hate it?

Ok, let’s get started!

Setting up Nginx on Linode

I’ve used Jekyll to create my website. A lot of the heavy lifting was done by Michael Rose in the form of a Jekyll theme he created called Hpstr.

Much respect.

But, setup was pretty painful for me. I knew nothing about websites, let alone creating a static page website. I’ve decided to set my hand to journal a lot of the nuances I ran into. Try to save someone some time. Or, save myself some time when something goes wrong.

These articles will not be on CSS, JavaScript, or HTML. After tinkering with computers for 20 years, I still suck at CSS and HTML–no, there are much better resources on the matter.

I actually recommend spending $30 on the following Udemy courses. They are great courses and will get you everything you need to be competitive.

(Note, make sure to get them on sale. Second note, they go on sale a lot.)

I’m not getting a kick back from Udemy, I list these courses because they are the ones I’ve taken and will vouch they are great courses to with this guide series.

1. Orientation

A lot of other articles will recommend setting up Jekyll locally, building your site to perfection, then get a rent a server when you have the time. I don’t recommend going this route.

In one way it makes sense to get a feel for Jekyll before deploying. You aren’t paying money while you learn. But, building a Jekyll site out locally, with all the bells and whistles, may cause a lot of problems deploying it. Was it the 5th gem or the 12th gem which is causing problems? No, I found it’s better to go for broke and start building the site on the web.

To compare the work steps

Common Workflow My Workflow
Setup Jekyll Locally Get Server
Deploy Site Locally Setup Server
Refine Setup Jekyll on Server
Deploy Site Locally Setup Jekyll Locally
Refine Deploy Site to Server
Deploy Site Locally Refine
Get Server Deploy Site to Server
Setup Server Refine
Setup Jekyll on Server Deploy Site to Server
Deploy Site to Server Beer
Beer Second Beer

A couple of reasons I prefer my workflow.

First, the psychological payoff doesn’t happen until the gross stuff is out of the way. Setting up the server side is tedious and can be boring. But, it is necessary for your site to be up and running on your own server. The payoff being when your site is available to your buddy in Maine who can see the friggin awesome site you’ve built.

If you put the kudos and warm fuzzies at the beginning, meaning, you deploy your site locally and tell yourself how great it looks, it robs you have the drive needed to trudge through the server side setup. Science!

Second, there are many different variables to account for between your local machine and the server. For example, if you are building Jekyll from a Windows machine and serving it on Ubuntu there can often be dependency differences which you must troubleshoot. Best to start doing it right away (see first point).

Ok, have I persuaded you? No? Then why are you still reading? Ha!

Also, the one thing you’ll have setting up the server side I did not is this guide. I plan to setup a new site walking while writing these articles to assure this guide is relevant. But if I miss anything, I’m available to help in the comments. It makes my day to save someone some development time.

2. Choose a Server Provided

Ever rented a server before? I hadn’t either.

Here is my tip sheet laden with my opinion.

a. Don’t Go Flashy

I don’t recommend going with a flashy name. E.g, GoDaddy, HostGator, etc. The general rule is, if they are pushy with their marketing they probably aren’t a solid choice.

The two solid choices right now are * Digital Ocean * Linode

b. Go with Linux

Oh! And go with Linux!

I had a CEO one time who forced me to use Windows on our server. Man, it was a flop.

First, Windows back-ends aren’t well documented on the web. They cost more. There are fewer free tools. You know what, let me just refer you to others’ rants.

There is a reason 80% (circa 2014) of servers are deployed using Linux, jus’ sayin’.

c. Go Small and Scale

If you go with Digital Ocean or Linode, they both have reasonable start servers, which can in turn be scaled. Meaning, you can pay more later for additional server resources without having to completely rebuild your server.

Ok! For this article I’m going to use Linode. I like them. They’ve who I started with and was extremely happy with their quality and reliability.

3. Get a Server

Head over to

Linode

And Sign Up

Login, then go to Add Linode. Here select the smallest sized Linode as possible. When I started, the small servers were $5 a month–but it looks like they’ve gone up. My guess is, you can find them on sale occasionally.

You don’t have to select the smallest–but I think it’s plenty for a Jekyll blog.

Once you’ve selected the size of server, scroll to the bottom and select a location central to your audience. If there isn’t one, then simply select the location closet to you.

Then select Add this Linode!

Once you’ve added your Linode you will be re-directed to your Linodes dashboard

Notice, the IP Address is the IP address of your very first server! Waahoo!

It’ll take it a second, but the status of your linode should change from Being created to Brand new, when it does, you will be ready for the fun!

6. Setup Linux

Let’s get Linux setup on your machine. Click on the name of your Linode.

This should load the server dashboard for your server. Looking something like this.

Don’t be alarmed. There is a lot going on here, but we are going to taker it one step at a time. Don’t worry, I got you.

First, let’s tell the computer which manages your server to install Linux on it. You can do this by going to Deploy an Image

Beware ye Stackscripts!

A stackscript is a Linux script meant for a machine with newly installed Linux. The script tells the machine to do a bunch of automated setup work to prepare the machine for a particular task. In our case prepare our machine to be a server. I’m not going to show how to use them in this walkthrough. For a few reasons. We will learn more setting things up ourselves, and therefore, will be able to maintain it. Also, I’ve not found a stack which is specifically for Jekyll. Most of them have a lot of extra stuff we don’t need.

Ok, back to work. Let’s fill out our setup request

Be sure to save your password somewhere! Not a lot of ways to recover it. Once everything is selected hit Deploy

Your server will quickly be formatted and a fresh copy of Ubuntu 16.04 LTS installed. Oh, and I’ve not mentioned

5. SSH

SSH stands for secure shell access. Shell being the command prompt environment which Linux is based. This is going to be our main way of interacting with the server. It may feel terse and inhumane, but I strongly encourage you to embrace the command line. If you do, the powers of Linux will be yours for free.

And besides, I’m writing this tutorial around it, so you kinda must to keep following along.

Ok, let’s fire up your machine. Open up the Linode dashboard and click on your linode’s name. At the top right there be a box called Server Status and it is probably Powered Off. Let’s turn it on by hitting the Boot button.

Wait until the status below shows your linode has fully booted.

Now, I’m assuming you are using Linux or Mac as your local operating system. On either, open a terminal and type

ssh root@your.ip.number.here

And press enter.

You should see something along the lines

[ladvien@ladvien ladvien.github.io]$ ssh root@your.ip.number.here
The authenticity of host 'your.ip.number.here (your.ip.number.here)' can't be established.
ECDSA key fingerprint is SHA256:ee2BPBSeaZAFbVdpWFj1oHLxdPdGoxCaSRl3lu6u2Fc.
Are you sure you want to continue connecting (yes/no)?

Type yes and hit enter.

You will then be prompted to enter the password entered as the root password during the setup phase in the Linode Manager.

6. Nginx Setup

You are now on your server. Do you feel a bit like Mr. Robot? Live the feeling. And don’t let anyone give you a hard time for being a shell noob. Embrace the shell.

I’m not going to go Linux stuff in detail. Please refer to more in depth tutorial. They are all over the Internet. But, I will point out, the Tab key works as an auto-complete. This is the single most important tidbit of working in shell. Instead of having to type out a long file name, type the first two letters and hit tab. It’ll try to fill it in for you.

Let’s start our server setup.

Your server is simply a computer. But, we are going to install a program your computer which will cause anyone visiting your IP address in the browser to see parts of your file system. The visitor’s browser loads information from your file system and, if the files are in a language the browser understands, renders it for the visitor. These files will be in HTML and CSS produced by Jekyll.

Ok. The server program we will be using is called nginx. It is not the oldest or the most common. But I find its use straightforward and it seems pretty darn fast too.

But first, let’s update Linux system. At your server’s command line type.

sudo apt-get update

And hit enter. This causes all the repository links to be updated. The repository links are libraries of Internet addresses telling your computer when it can find free stuff! Everything is swag on Linux.

Let’s take a second to check something before we start install nginx. Open any browser and type your linode’s ip address in the browser address bar and hit enter. Most likely, nothing will happen. The browser is trying to make contact with your server, but there is no program installed on your server to serve the website to a browser. That’s what nginx will do.

Let’s download nginx now

sudo apt-get install nginx

It will ask if you want to install nginx say yes.

Once it’s installed, let’s test and make sure it works.

Type

nginx

It should respond with

nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] still could not bind()

Great! This means it is installed and working. We just need to setup nginx to serve our files on our server address instead of 0.0.0.0:80.

Also, open a browser and type your sever’s IP address again. Hit enter. This time you should see:

Wow, your are now serving an html to the world, for anyone who visits your website. Pretty cool, eh? I think so.

Want to see something pretty cool?

Type (note, do not include sudo here)

nano /var/www/html/index.nginx-debian.html

You should see the content of the html file being served by nginx.

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Change

<h1>Welcome to nginx!</h1>

To

<h1>Welcome to the Jungle, baby!</h1>

Then hit CTRL + O, which should save the file. Then hit CTRL + X to exit the nano editor.

Now, switch back to your browser, go back to your website’s IP address, and hit refresh. You should see.

Not seeing it? You didn’t change the <title> instead of the <h1>, right? Ask me how I know that…

Friggin awesome! Let’s move on to setting up Nginx, so you can serve your own website.

Linode actually has a great walkthrough on setting up Nginx.

But, for now, are going to stick with the basic nginx setup. There will other articles in this series where I show how to edit nginx to make the website better.

7. Jekyll

Let’s setup Jekyll locally. To follow utilize Jekyll we are going to need to download and install the following programs.

Ruby

Ruby is programing environment which contains a package manager which we will use a lot called [gem](https://en.wikipedia.org/wiki/RubyGems). For example, when we type gem install cool-program it is the ruby environment pulling the cool-program from the Internet and installing it on your machine.

Bundler

Bundler is a program which helps pull all the dependencies needed to run a program together. As they say in the README, “Bundler makes sure Ruby applications run the same code on every machine.”

Git

Git is version control program. It also has the ability to pull source code off line. We are going to use it at first to pull a theme off line, but eventually, we will manage your website Jekyll source code with it.

Homebrew (Mac Only)

Homebrew, often referred to sa Brew, is a program which is like apt for Linux. It is a command line tool which lets you pull programs from the Internet and installs them locally.

Ok, let’s get going

At your local computer’s terminal type:

Linux

sudo apt-get install ruby
gem install jekyll

Mac

To setup Ruby correctly on Mac we are going to install a command line package manager for Mac called brewed. This is the equivalent of apt in Linux.

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew install ruby
gem install jekyll
gem install bundler

8. Get a Jekyll Starter

Jekyll is great for creating websites, but there is a lot of boilerplate. I found it much easier to clone someone else’s Jekyll starter site than make my own from scratch.

For this series we are going to use the Neo-HPSTR theme.

Open the terminal and pick a directory where you would like to put a copy of your website. For me, I’m Linux and will use the home directory.

Now, let’s download our theme.

git clone https://github.com/aron-bordin/neo-hpstr-jekyll-theme

Git clones the neo-HPSTR theme from the Internet and puts it in a directory called /neo-hpstr-jekyll-theme Feel free to rename the directory the name of your website. For example, my directory is called ladvien.com We are getting to putting this website on-line, just a few more steps.

9. Build the Jekyll Theme

Open your website’s directory

cd neo-hpstr-jekyll-theme

And enter

bundler install

This will pull all the need programs to make this theme build on your computer. Note, you may be required to enter your password for file access.

Ok, moment of truth. Type

bundle exec jekyll build

You should see a response similar to

Configuration file: /home/ladvien/neo-hpstr-jekyll-theme/_config.yml
       Deprecation: The 'gems' configuration option has been renamed to 'plugins'. Please update your config file accordingly.
            Source: /home/ladvien/neo-hpstr-jekyll-theme
       Destination: /home/ladvien/neo-hpstr-jekyll-theme/_site
 Incremental build: disabled. Enable with --incremental
      Generating...
                    done in 1.103 seconds.
 Auto-regeneration: disabled. Use --watch to enable.

But, if you didn’t get any errors, you should be good.

Breaking this down, we used the bundler program to execute the jekyll program. We passed the build command to the jekyll program, which tells jekyll to take all your jekyll files and compile them into your website. The bundler program made sure jekyll had everything it needed to compile correctly.

In your file explorer, navigate to your website directory and enter the _site directory. This directory contains your entire website after compilation.

jekyll_site_folder

Open this folder and then double click on the file index.html. This should open your website locally in the browser.

jekyll_site_locally

But this isn’t what we want. Let’s get it on the webserver we setup.

Open the command prompt and switch directories to your website’s main directory. Then, type

scp -r _site/*  root@your.website.ip.address:/var/www/html/

This should copy all of your compiled website files to your website. Go to your website address and you should see the website on-line! Booyah!

10. That It?

Noooooo, this was the bare minimum setup. Here’s a list of what I plan to tackle in this series.

  • Editing the _config.yml file to customize your theme
  • Setup your code on Github
  • Adding SSL encryption
  • Tweaking the server to zip assets before sending them to your viewers
  • Make the server more secure – this is called hardening
  • Create a script which will automatically compile Jekyll, send it to Github, and then copy the compiled files to your website.
Creating a GPU Accelerated Deep-Learning Environment on Arch Linux

This article logs a weekend of efforts to create a deep-learning environment which meets the following criteria

  • GPU Enabled
  • On Arch Linux
  • Uses Keras with Tensorflow as a backend
  • Main IDE being RStudio

It was a tough one.

TL;DR

There was error I had a hell of a time debugging. Installing the toolchain is fairly straightforward, except CUDA. At the time of writing this article (2018-04-29), there is a version mismatch between CUDA and CUDNN in the Arch Linux repositories.

This results in an the following error every time I tried to import tensorflow in Python.

ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory

The Arch Linux package CUDA was pulling the latest version 9.1.1 (at writing) and the Arch Linux package CUDNN was looking for version 9.0. That little mismatch cost me 10 hours.

0. Other Arch Linux Deep-Learning Articles

There are a couple other Arch Linux deep-learning setup walkthroughs. Definitely need to give these guys credit, they are smarter than me. However, neither walkthrough had everything I was looking for.

This article was alright. But it focused a lot on preparing Arch Linux from the bare metal, which is usually the right idea with Arch, if you are on a resource budget. For example, running on a server or Raspberry Pi. But the extra few bytes of RAM saved doesn’t really justify the time spent on meticulous tunning when we will be talking in megabytes and not bytes. And let my immolation begin.

Also, this article doesn’t include information on GPU support. Whaawhaa.

This one was a bit closer to what I need. In fact, I did use the middle part. However, the mismatch was not mentioned. Of course, it’s not the author’s fault. At the time he wrote it I’m guessing the repositories matched.

Alright, on to my attempt.

1. Install Antergos (Arch Linux)

I love me some Arch Linux. It’s lightweight and avoids the long-term issues of other flavors. Plus, it is meant to be headless, so it’s great for embedded projects. Given how many embedded projects I take on it made me accustomed to using daily, eventually, I made it my main desktop flavor. It should sound too Linux-snobby, though, I dual-boot it on my Mac Book Pro. The one issue with Arch Linux is it can be a little unfriendly to new users–or those with limited time and cannot be bothered with the nuances of setup. Enter Antergos.

Antergos is essentially Arch Linux with a desktop environment and a GUI installer. A perfect choice for my deep-learning endeavors. Really, you should check it out. Go now.

We’re going to use it for this project.

Download the iso file

You’ll need a little jumpdrive, 4gb should work.

I use Etcher as it makes it painless to create boot media.

After Etcher does its thing, insert the jumpdrive, open Etcher, and then select the Antergos iso file. Here’s the usual warning, if you have anything on your jumpdrive it’s about to get deleted forever.

Insert the media into the machine you want to install Arch on and boot from the jumpdrive.

Windows

You will need to hit a special key during the boot sequence to enter the BIOS’ boot menu

Mac

While booting hold down the Option key.

If all goes well you should see a menu which says

Welcome to GRUB!

And then shows an Antergos boot menu. Select boot Antergos Live.

Once the boot sequence is finished you should see the Antergos desktop environment start and shortly after cnchi, which is Antergos’ GUI installer

Select Install It. The installer is fairly self explantory. But, if you run in to any issues, please feel free to ask me questions in the comments. I’m glad to help.

Once the installer is complete you will be prompted to restart the computer. It’s go time.

2. Install NVIDIA

When you boot up the installed Antergos open the terminal.

We will start with installing the base NVIDIA packages. As part of it, we are going to get the wrong version of CUDA. But, I found downloading the NVIDIA as whole packages and then replacing CUDA with an earlier version, much eaiser than trying to pull everything together myself.

Ok, here we go.

sudo pacman -S nvidia nvidia-utils cuda cdnn

That might take awhile.

So, how you been? Oh–wait, it’s done.

Ok, to initialize the changes reboot.

sudo reboot now

3. Downgrade CUDA to match CDNN

That should have gotten everything at once. Now, let’s downgrade CUDA from 9.1 to 9.0.

wget https://archive.archlinux.org/packages/c/cuda/cuda-9.0.176-4-x86_64.pkg.tar.xz

This downloads a pkg file for CUDA 9.0, which is what the most recent version of Tensorflow is expecting (at this time, 1.8). I found the easiest way to replace CUDA 9.1 with 9.0 to simply double click on the file we downloaded from the GUI file browser. This opens it in Antergos’ answer to a GUI based package manager. It will warn you this package will downgrade your CUDA version and ask you to Commit to the changes. Hit the commit button.

Wait for the file to be replaced before moving on.

4. Anaconda (Optional)

Anaconda is a great package manager for data (mad) scientist tools. It is Python centric, but also supports R and other stuff I don’t know how to use yet.

We will be using it to prepare our system to support deep-learning projects.

Download the Linux version suited for your computer.

Once the file is downloaded right click on the file and select Show In Folder. Once there, right-click in the open space and select Open in Terminal.

Make Anaconda executable and then run it.

chmod +x Anaconda3-5.1.0-Linux-x86_64.sh
./Anaconda3-5.1.0-Linux-x86_64.sh

The Anaconda installtion is off and running. It will ask you to agree to a form. After, it will ask whether you want to install Anaconda in its default directory. We do.

Now, it will install every data scientist package known to existance. Mwhahaa. Erm.

When it asks

Do you wish the installer to prepend the Anaconda3 install location
to PATH in your /home/ladvien/.bashrc ? [yes|no]

Type yes. This will make Anaconda accessible throughout your system.

Of course, this new path variable will not be loaded until you start your user session again (log off and back on). But we can force it to load by typing.

cd ~
source ./bash_profile

Double check we are using the Anaconda version of Python.

[ladvien@ladvien ~]$ which python
/home/ladvien/anaconda3/bin/python

If it doesn’t refer to anaconda somewhere in this path, then we need to fix that. Let me know in the comments below and I’ll walk you through correcting it.

If it does, then let’s move forward!

6. Tensorflow and Keras

Alright, almost done.

Let’s go back to the command prompt and type:

sudo pacman -S python-pip

This will download Python’s module download manager pip. This is usually packaged with Python, but isn’t included on Arch.

How’d we get Python? Anaconda installed it.

Let’s download Tensorflow with GPU support.

sudo pip install tensorflow-gpu --upgrade --ignore-installed

Let’s test and see if it’s worked. At command prompt type

python

And in Python

import tensorflow as tf
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))

You should a response similar to

2018-05-01 05:25:25.929575: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties:
name: GeForce GTX 1060 6GB major: 6 minor: 1 memoryClockRate(GHz): 1.7715
pciBusID: 0000:01:00.0
totalMemory: 5.93GiB freeMemory: 5.66GiB
2018-05-01 05:25:25.929619: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-05-01 05:25:26.333292: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-05-01 05:25:26.333346: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-05-01 05:25:26.333356: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-05-01 05:25:26.333580: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5442 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0, compute capability: 6.1)
Device mapping:
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0, compute capability: 6.1
2018-05-01 05:25:26.455082: I tensorflow/core/common_runtime/direct_session.cc:284] Device mapping:
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0, compute capability: 6.1

Which means you are good to go! At this point, Python is setup to do accelerated deep-learning. Most deep-learning peeps stop here, as Python is the deep-learning language. However, like a pirate I’m an R sort of guy.

7. Installing R and RStudio

To setup a GPU accelerated deep-learning environment in R there isn’t a lot of additional setup. There are keras and tensorflow R packages, which connect the R code to a Python backend.

To get R in Arch Linux open the terminal and type:

sudo pacman -S r

And what’s R without RStudio? Actually, it’s still R, which is bad-ass unto itself–but anyway, let’s not argue. Time to download RStudio…because you insist.

In terminal

cd ~
git clone https://aur.archlinux.org/rstudio-desktop-bin.git
cd rstudio-desktop-bin
makepkg -i

After, you should find RStudio in the Antergos Menu.

You can right click on the icon and click Add to Panel to make a shortcut.

Open up RStudio and lets finish this up.

8. R Packages for Deep Learning

Inside RStudio’s code console type

install.packages("tensorflow")

This will install the package which will help the R environment find the Tensorflow Python modules.

Then,

install.packages("keras")

Keras is the boss package, it’s going to connect all the Python modules needed to Tensorflow for us to focus on just the high-level deep-learning tuning. It’s awesome.

Once the keras package is installed, we need to load it and connect it to the unerlying infrastructure we setup.

library(keras)
install_keras(method = "conda", tensorflow = "gpu")

This will install the underlying Keras packages using the Anaconda ecosystem and Tensorflow Python modules using CUDA and CUDDN. Note, a lot of this we setup manually, so it should report the needed modules are already there. However, this step is still needed to awaken R to the fact those modules exist.

Alright, moment of truth. Let’s run this code in R.

library(tensorflow)

with(tf$device("/gpu:0"), {
  const <- tf$constant(42)
})

sess <- tf$Session()
sess$run(const)

If all went well, it should provide you with a familiar output

> library(tensorflow)
>
> with(tf$device("/gpu:0"), {
+   const <- tf$constant(42)
+ })
/home/dl/.virtualenvs/r-tensorflow/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
>
> sess <- tf$Session()
2018-05-01 05:55:07.412011: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties:
name: GeForce GTX 1060 6GB major: 6 minor: 1 memoryClockRate(GHz): 1.7715
pciBusID: 0000:01:00.0
totalMemory: 5.93GiB freeMemory: 5.38GiB
2018-05-01 05:55:07.412057: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-05-01 05:55:07.805042: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-05-01 05:55:07.805090: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-05-01 05:55:07.805115: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-05-01 05:55:07.805348: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5150 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0, compute capability: 6.1)
> sess$run(const)
[1] 42

9. Scream Hello World

And the payoff?

Using the prepared Deep Dream script from the Keras documentation

Voila!